[Systems] scaling activities.sugarlabs.org

[David Farning]

--Downloads--
I have the mirror system set up as three tiers:
Tier I. Primary activities server.
Tier II. Primary download node.
Tier III. Secondary download nodes.

Sunjammer, the Tier I server, is set with MIRROR_DELAY = 30.  All new
activities are served from this server for the first 30 minutes.

Solarsail, the Tier II server, is set with a priority 1. (very low)
Solarsail handles new activities older than the 30 minute MIRROR_DELAY
but before they are mirrored elsewhere.  The low priority means that
if an activity is available on a tier II mirror, it will be grabbed
form there.

General mirrors, the Tier III servers, are the general availability mirrors.

It looks like we will be able to tune and scale up the download
infrastructure by adding machines to one of the three tiers and tuning
the MIRROR_DELAY and sync frequency of specific mirrors.  It looks
like we can use this system to mange our activity downloads for at
least another 12 months.

--Memcache--
The next major tuning issue will be memcache.  Memcache sits between
a.sl.o and the database backend. Memcache caches frequently used
lookups to prevent comparatively expensive database hits.

We last tuned memchache while sunjammer was on a very memory
constrained machine.  We have been getting a rather low 60% hit rate
over the last week.  Last night I bumped the cache size from 256K to
512K to see how it would affect performance.

I am working with AMO developer determine target hit rates given our
specific constraints.  One idea is to set up a VM on treehouse to
provide additional memcache.  I think Bernie mentioned that treehouse
was now hosted at gnaps.

--Future--
We will see how effectively splitting the workload between sunjammer
and beam rider works.  But, it looks like we have 6-9 months before we
need to worry about scaling a.sl.o. again.  Sunjammer is happily
peaking at about 45% of max cpuload

david