[Systems] Performance tuning for activities.sugarlabs.org

[Bernie Innocenti]

Over the past months, CPU usage on Sunjammer has been increasing
gradually to the point that it made the system unusable during prime
time (east coast daytime hours):

  http://sunjammer.sugarlabs.org/munin/sugarlabs.org/sunjammer.sugarlabs.org-cpu.html


Apache is getting impressive peaks of 500 requests per minute every day:

  http://sunjammer.sugarlabs.org/munin/sugarlabs.org/sunjammer.sugarlabs.org-apache_accesses.html


The logs oddly did not match these stats.  Then I realized that ASLO
wasn't actually logging into the access-all.log.  Aha!

Ok, then ASLO is actually responsible for all this activity.  What next?
We already have memcached and APC hard at work:

  http://sunjammer.sugarlabs.org/munin/sugarlabs.org/sunjammer.sugarlabs.org-memcached_bytes.html
  http://sunjammer.sugarlabs.org/munin/sugarlabs.org/sunjammer.sugarlabs.org-memcached_rates.html
  https://sunjammer.sugarlabs.org/apc.php

Caches appear to be large enough, so there's not much we can do to
improve.  I tweaked a few things of minor relevance in php.ini, just in
case.

Looking around I noticed that the production aslo had debug and
development mode enabled, and the query cache disabled.  Tomorrow we
shall see how much changing these helped.  There may also be some MySQL
tuning to be investigated.

Anyway, the FSF is soon going to relocate us to a higher profile box
with 4 cores (we currently have only 1 assigned to our VM) and more
memory.  This should really breath some fresh new air into Sunjammer.

If Sugar adoption keeps improving at this rate, at some point we'll have
to split ASLO to an independent box.  Or two :-)  It's good to know
we're using the same software that Mozilla has already scaled up to
several millions users!

-- 
   // Bernie Innocenti - http://codewiz.org/
 \X/  Sugar Labs       - http://sugarlabs.org/