[Systems] Disk nearly full on treehouse (was: Re: [Systems-logs] treehouse.sugarlabs.org: Munin alert)

[Bernie Innocenti]

On Wed, 2010-11-17 at 12:57 +0100, Sascha Silbe wrote:
> Excerpts from munin's message of Wed Nov 17 11:55:17 +0100 2010:
> 
> > sugarlabs.org :: treehouse.sugarlabs.org :: Disk usage (in percent)
> >     WARNINGs: /backup/sunjammer/srv is 93.00 (outside range [:92]), /backup/sunjammer/root is 95.92 (outside range [:92]).
> 
> While I could enlarge those two LVs by a few GBs, I don't think it'll
> help for long and we don't have much capacity left. Do we have loads of
> unused HD space (> .5T) somewhere else? If not, we should request money
> for new HDs from SLOB or try to find a sponsor.

We're running out of space in the vg.

The only feasible place for rsync backups is treehouse because it's
local. I will add two more hard drives as soon as I visit the co-lo.
Alternatively, we could start using the FSF backup server which has an
impressive amount of storage:

 Filesystem            Size  Used Avail Use% Mounted on
 /dev/md4              2.7T  2.3T  334G  88% /srv/md4
 /dev/md5              2.7T  2.3T  292G  89% /srv/md5
 /dev/md6              2.7T  1.6T  961G  64% /srv/md6
 /dev/md7              2.7T  2.5T   60G  98% /srv/md7


Meanwhile, we could exclude /var/lib/mysql from the backup: the ibdata
file is a gigantic blob of 4GB that changes a little bit every day.

We're already taking a daily dump, although there's no history.


> What got of the Wikimedia servers, BTW? Can we turn one of those into a
> backup server (after adding the new HDs)?

No, because they're not being hosted locally. We don't need more
servers, we just need more disk space :-)

-- 
   // Bernie Innocenti - http://codewiz.org/
 \X/  Sugar Labs       - http://sugarlabs.org/