[Systems] Benchmarking aslo-proxy

[Bernie Innocenti]

El lun, 16-11-2009 a las 00:09 -0600, David Farning escribió:
> I am using a tool called siege[1] to benchmark aslo.  If I understand
> correctly, ab just hits one url over and over again.
> 
> Siege allows us to work with a more realistic working set of URL
> requests.  I am current testing by.
> 1.  Grabbing the all-access.log from sunjammer for Friday the 13th of
> November  This gives us a good representation of the pages and order
> in which users accessed the aslo pages.
> 2. Parse the all-access.log into a format which siege can use as input.
> 3.  I am running time based benchmarks.
>   a.  After changing a configuration variable or restarting a service
> - run benchmark for 15 minutes to warm up the cache
>   b.  Run benchmark.
> 4.  I have been running all benchmarks from beamrider for consistency.

Can you publish the results?


> I need to look at that today.  aslo does not use APC.  APC is limited
> to a single machine.  Aslo uses memcached which allow distributed
> caching.  At our current scale APC would be better.... but memchached
> scales better across multiple php and memcached servers.

First of all APC is not just an object store like memcached. Its main
function is transparent PHP bytecode caching, which provided
unbelievable speed boosts. This function does not exclude memcached.

For temp object storage, if we are to believe the anecdotal evidence
that APC would be 5 times faster than memcached [1] we'd start
benefiting from memcached distributed nature only when we'll have at
least 5 aslo-web nodes.

[1] http://www.mysqlperformanceblog.com/2006/09/27/apc-or-memcached/


But, IIRC, for temp object storage Remora only supports memcached. I'd
still make it run locally, though.


> This is something that we can workout with Mozilla.  They have the
> same problems.  They use proprietary traffic manager called zeus to
> handle their reverse proxy/load balancing/ and log consolidation.
> Having had some experience setting up proxies and load balancers I at
> least have some idea of the right questions to ask....

Cool. For Mediawiki, they had a neat diagram illustrating the network
topology.


> > Now, the only purpose of going through Squid would be to balance traffic
> > to multiple applications servers. Of which, we only have one at this
> > time.
> 
> I would like to watch the load averages for the week on aslo-proxy and
> sunjammer.  Squid is catching 80-90% of the hits before they get to
> the php instance.  I want to see what effect that has on the load.

No it's not! It's letting 100% of them through, because php by default
send anti-cache headers to the browser, which are intercepted and
honored by the proxies.

Subverting this simple scheme without breaking everything is a black
art. There are subtle ways in which cached pages can break navigation,
and testing is not easy because browsers also do their own caching.


> > Do we want to keep the aslo instance on sunjammer in production
> > alongside the new one until we get a new machine?
> 
> I would like to run
> aslo-proxy treehouse -> php on sunjammer -> mysql on sunjammer
> for at least the next week.
> 
> During the week I would like to:
> 1. Test and tune aslo-web and aslo-db
> 2. Set up and test HAproxy and aslo-web2.

If, after reading the above considerations, you still want to go on,
I'll point activities.sugarlabs.org at aslo-proxy.

Although I disagree on some implementation details, but I don't want to
become a brake. Consider all my comments as mere suggestions. You're the
one championing this work, therefore you get to say the final word on
how it's done.

-- 
   // Bernie Innocenti - http://codewiz.org/
 \X/  Sugar Labs       - http://sugarlabs.org/