[Systems] Docker backups

[Sam P.]

Hi Bernie,

I don't think you need to back up /var/lib/docker/.  I will look more into
this.

>From memory, that just stores the docker images; all user data is stores in
/srv/socialhelp.  All the images are either build from the code in
/srv/socialhelp (~10min build time - less if cache survives) or downloaded
from the docker hub.

As for why it is so big, I don't know.  I don't think docker is using aufs,
instead it has fallen back to btrfs (for reasons unknown).  This is
probably sucking more space.  There are also lots of unused images floating
around, so I am removing them by:

    docker ps -a -q | xargs docker rm
    docker images -q | xargs docker rmi

Both commands should return errors about not being able to delete running
images (good thing).  They freed about 1GB of space :)

I am happy to work on this tomorrow (or now).  I'm in Australia (google
"time in Canberra"), so your afternoon is my morning and a good time to
work on this.

Sam

On Sat, Oct 25, 2014 at 8:58 AM, Bernie Innocenti <bernie at sugarlabs.org>
wrote:

> On 23/10/14 23:28, Munin wrote:
> > sugarlabs.org :: justice.sugarlabs.org :: Disk usage in percent
> >       WARNINGs: /backup is 93.37 (outside range [:92]).
> >       OKs: / is 61.40, /run/shm is 0.00, /run is 0.08, /dev is 0.00,
> /sys/fs/cgroup is 0.00, /run/lock is 0.00.
>
> Sam, yesterday dogi and I figured out that /backup on justice was full
> due to the docker backups from freedom taking up unreasonable amounts of
> disk.
>
> We "fixed" it by excluding /var/tmp and /var/lib/docker from the backup,
> but this is clearly not what we want.
>
> Besides, a simple 'du -sx /var/lib/docker/*' takes forever to complete
> due to... not sure what. Maybe btrfs or aufs being incredibly slow?
>
> Could you explain the directory structure on freedom, and what should be
> backed up? The goals are:
>
>  - Not loosing any user-generated data
>  - Quick reconstruction of the images in case of disaster recovery
>
> Our support for Docker needs to mature before we can start moving
> services to it. Working backups are (of course) a blocker, but we also
> need to think about monitoring and management. Would you have time to
> chat about it this Saturday? I'll be doing some infrastructure work from
> the west coast (PDT).
>
> --
> Bernie Innocenti
> Sugar Labs Infrastructure Team
> http://wiki.sugarlabs.org/go/Infrastructure_Team
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sugarlabs.org/private/systems/attachments/20141025/fcdb4c46/attachment.html>