[Systems] Benchmarking aslo-proxy

[David Farning]

2009/11/16 Bernie Innocenti <bernie at codewiz.org>:
> El lun, 16-11-2009 a las 00:09 -0600, David Farning escribió:
>> I am using a tool called siege[1] to benchmark aslo.  If I understand
>> correctly, ab just hits one url over and over again.
>>
>> Siege allows us to work with a more realistic working set of URL
>> requests.  I am current testing by.
>> 1.  Grabbing the all-access.log from sunjammer for Friday the 13th of
>> November  This gives us a good representation of the pages and order
>> in which users accessed the aslo pages.
>> 2. Parse the all-access.log into a format which siege can use as input.
>> 3.  I am running time based benchmarks.
>>   a.  After changing a configuration variable or restarting a service
>> - run benchmark for 15 minutes to warm up the cache
>>   b.  Run benchmark.
>> 4.  I have been running all benchmarks from beamrider for consistency.
>
> Can you publish the results?

I'll publishing the results on http://people.sugarlabs.org/dfarning/
But, I don't understand them enough to really trust them yet.

>> I need to look at that today.  aslo does not use APC.  APC is limited
>> to a single machine.  Aslo uses memcached which allow distributed
>> caching.  At our current scale APC would be better.... but memchached
>> scales better across multiple php and memcached servers.
>
> First of all APC is not just an object store like memcached. Its main
> function is transparent PHP bytecode caching, which provided
> unbelievable speed boosts. This function does not exclude memcached.
>
> For temp object storage, if we are to believe the anecdotal evidence
> that APC would be 5 times faster than memcached [1] we'd start
> benefiting from memcached distributed nature only when we'll have at
> least 5 aslo-web nodes.
>
> [1] http://www.mysqlperformanceblog.com/2006/09/27/apc-or-memcached/
>
>
> But, IIRC, for temp object storage Remora only supports memcached. I'd
> still make it run locally, though.

Yes, this is on my list of things to figure out.  I don't understand
why Remora does not use APC.  My naive guess is that they feel it is
cheaper just to add another physical machine then optimise code.

>> This is something that we can workout with Mozilla.  They have the
>> same problems.  They use proprietary traffic manager called zeus to
>> handle their reverse proxy/load balancing/ and log consolidation.
>> Having had some experience setting up proxies and load balancers I at
>> least have some idea of the right questions to ask....
>
> Cool. For Mediawiki, they had a neat diagram illustrating the network
> topology.
>
>
>> > Now, the only purpose of going through Squid would be to balance traffic
>> > to multiple applications servers. Of which, we only have one at this
>> > time.
>>
>> I would like to watch the load averages for the week on aslo-proxy and
>> sunjammer.  Squid is catching 80-90% of the hits before they get to
>> the php instance.  I want to see what effect that has on the load.
>
> No it's not! It's letting 100% of them through, because php by default
> send anti-cache headers to the browser, which are intercepted and
> honored by the proxies.

Let me look at this again.  The squid cachemgr3 is reporting that it
is handling 80% of hits out of cache.....

> Subverting this simple scheme without breaking everything is a black
> art. There are subtle ways in which cached pages can break navigation,
> and testing is not easy because browsers also do their own caching.

hmmm. Maybe siege is grabbing _everything_ fresh every time whereas a
normal browser is cache stuff locally.  That would explain why squid
is serving so many copies of *.js

>> > Do we want to keep the aslo instance on sunjammer in production
>> > alongside the new one until we get a new machine?
>>
>> I would like to run
>> aslo-proxy treehouse -> php on sunjammer -> mysql on sunjammer
>> for at least the next week.
>>
>> During the week I would like to:
>> 1. Test and tune aslo-web and aslo-db
>> 2. Set up and test HAproxy and aslo-web2.
>
> If, after reading the above considerations, you still want to go on,
> I'll point activities.sugarlabs.org at aslo-proxy.
>
> Although I disagree on some implementation details, but I don't want to
> become a brake. Consider all my comments as mere suggestions. You're the
> one championing this work, therefore you get to say the final word on
> how it's done.

I consider you comments rather stronger than suggestions.  I consider
them the most cost effective way to meet our infrastructure needs at
our current scale... in terms of load, machines, and admins.

The challenge is that we are scaling rapidly.  In the long term we
need to be able to be able to geometrically increase our load
capacity, while keeping our machine growth linear and our admins
constant.  This sudden increase in machines is my not so subtle way of
pushing you (kicking and screaming) from being the Sugar Labs
infrastructure wizard into the infrastructure team coordinator.

david

>   // Bernie Innocenti - http://codewiz.org/
>  \X/  Sugar Labs       - http://sugarlabs.org/
>
>