[Systems] Bandwidth usage on bender

[Bernie Innocenti]

This Sunday I got a phone call from Develer about excessive bandwidth
usage from bender that was resulting in a near-DoS for the rest of their
systems.

Using some more bandwidth overnight and over the week-end would
generally be acceptable, because it doesn't interfere with work
activity.  Of course Develer uses a traffic shaper to ensure fair
bandwidth sharing and higher QoS for VOIP traffic.

However, our IPv6 traffic is being encapsulated inside an AYIYA tunnel,
which looks like UDP packates to the Develer gateway.  This creates two
difficulties: first, the TC rules are unable to distinguish between RTP
and other UDP traffic, so they queue our outbound traffic before
anything else.  Second, unlike TCP, UDP has no explicit ACKs that can be
used to rate limit incoming traffic in a straightforward way.

The net result is that when we're rsyncing a file from Bender to
Sunjammer, ping times from Develer's LAN clients rise to 3000ms.  I
think what we're seeing is the result of the remote HDSL endpoint of our
ISP filling up a rather large queue, so there's little we can do about
it.

Things might improve slightly over the next few weeks, as Develer
migrates to a new 10Mbit FC, hopefully with better network equipment.
Because the new uplink comes with a /4 network (16 IPs), bender will get
a public static IP, which means we'll be able to switch to 6in4.
However, I strongly suspect 6in4 will be equally hard to limit.

For the time being, please ensure every large file-transfer from or to
the VMs is done with rsync rather than a dumb protocol, and always add a
"--bwlimit=100".  Moreover, please desynchronize cron jobs to avoid
simultaneous transfers from multiple VMs, and try to use night hours
when possible.

The overall Develer bandwidth is ~400KB/s in both directions, and will
soon become 1MB/s with a much better latency.

-- 
   // Bernie Innocenti - http://codewiz.org/
 \X/  Sugar Labs       - http://sugarlabs.org/