[Systems] download.sugarlabs.org

[David Farning]

Over the last couple of months the load, importance, and available
hardware of the Sugar Labs Infrastructure has been increasing quite
rapidly.  Just like with a development team, this growth requires
increased communication.  Unlike the development team which is release
focused, the activities of an Infrastructure team are task focused.
Putting mirrorbrain into production was similar to planning a military
operation.

I am not suggesting that we follow this process every time we do
something.... But, it was enlightening.
1. Situation - What resource are available?  In this instance we had
three software packages available: cacheboy, bouncer, mirror manager,
and mirrorrain.  We have three types of test environments:  VMs on my
desktop machine, VMs on the SL infrastructure, and sunjammer.
2. Mission - What do we want to accomplish?  Increase the available
download capacity of SL with the following constraints: Use proven
technology, scalable, distribute the load across multiple redundant
external machines,....
3. Execution - How do we accomplish the mission? In this instance I
tried to create a check list of how to do the final installation on
Sunjammer.
4. Support - Who do I need available to help?  In this case, Bernie and Peter.
5. Communication - How are we going to communicate?  Mailing lists,
email, and exchange phone numbers.

The majority of the work happened in step 3 - Execution.  I tried the
various options and determined that mirrorbrain was the best fit for
Sugar Labs.  The main problem was that this was the first time
mirrorbrain was being put into production on ubuntu servers.

So, for the next couple of week I kept doing test installs on a vm on
my desktop.  At the same time Peter created ubuntu packages. Every
time something would break, I talked to Peter and he made a some
fixes.  After several iterations, we had a working packages and
processes.

After it seemed everything was going correctly, I started smoke
testing.  I this case, I connected a couple of laptop to my home
router and wrote a shell script to download from the mirrorbrain
vhost.  I let three laptop run overnight.  The limiting factor in the
test was the laptops hard drives.

Next, we moved to mirrorbrain.sugarlabs.org, a VM bernie set up on the
SL infrastructure.  We did a test install found a couple of more
issues.  Bernie created  fresh VM. We ran the tests again.  Everything
worked.

Time to go live. bernie created download-testing.sugarlabs.org at
sunjammer.  I installed mirrorbrain and a dl-testing.sl.o vhost.
Everything seemed to work.  I copied the contents of the
dl-testing.sl.o to dl.sl.o changed a couple of log files, and the
doument root. And hit reload.

After Action Review.  The process basically worked.  The two weak
links were improve communication and create a some form of test
sunjammer.  Everything that went wrong was due to sunjammer and a.sl.o
specific issues which I did not test.

1. mirrorbrain could not handle symlinks in the vhost document root.
During testing I just used /srv/www-sugarlabs/download.  I didn't
realize that having /srv/www-sugarlabs/download symlink to /srv/upload
would be problem.  This caused a couple of changes to the a.sl.o
config to get rid of the symlink.

2.  Permissions. It took a while to get the permission correct so that
mirrorbrain and a.sl.o could both read and write to /srv/upload.

Lessons Learned.
1. We need to create a test VM that matches sunjammers so we can see
how changes affect the entire system.... before they go into
production.

2. Service coordinator.  For this job I used Bernie as the contact
person for all of sunjammer.  It would have made sense to communicate
more clearly to Alsroot as the a.sl.o coordinator what changes were
being made to a.sl.o.

Moving forward.
As Sugar Labs grows, we are going to need to plan and clearly
communicate what we are doing.  The development team has a very good
idea with the new feature requests.  Major changes are describe on the
request.  A developer implements the feature.  The feature is reviewed
by module owners who this affects.  If approved, the modue maintain
pull the feature into the main tree.

I would like to propose something similar for the Infrastructure team.
 Major changes are documented and reviewed through a process similar
to the new feature request.

In order to make this happen, I would like to spend the next couple of
weeks using the new feature process to migrate services to beamrider.
I am willing to:
1. Create the new feature template.
2. Create new features describing the procees of migrating services to
beamrider.
3. Have those new features reviewed.
4. Work with bernie to put those new features into production on beamrider.

david