[Systems] Benchmarking aslo-proxy

[David Farning]

I am using a tool called siege[1] to benchmark aslo.  If I understand
correctly, ab just hits one url over and over again.

Siege allows us to work with a more realistic working set of URL
requests.  I am current testing by.
1.  Grabbing the all-access.log from sunjammer for Friday the 13th of
November  This gives us a good representation of the pages and order
in which users accessed the aslo pages.
2. Parse the all-access.log into a format which siege can use as input.
3.  I am running time based benchmarks.
  a.  After changing a configuration variable or restarting a service
- run benchmark for 15 minutes to warm up the cache
  b.  Run benchmark.
4.  I have been running all benchmarks from beamrider for consistency.



1. http://www.joedog.org/index/siege-home


On Sun, Nov 15, 2009 at 6:17 PM, Bernie Innocenti <bernie at codewiz.org> wrote:
> I tried to measure the overhead of these 3 scenarios:
>
> 1) aslo-proxy.sugarlabs.org: 9.94 reqs/sec
> 2) activities.sugarlabs.org: 9.83 reqs/sec
> 3) aslo-web.sugarlabs.org:   1.90 reqs/sec
>
> I've performed all three tests from the machine hosting the service to
> ensure no network overhead. The overhead of ab itself is negligible with
> with so few connections per second.
>
> There's definitely something wrong with aslo-web's configuration. I'd
> expect it to perform approximately as well as sunjammer, because the CPU
> speeds are equivalent. Is APC installed?

I need to look at that today.  aslo does not use APC.  APC is limited
to a single machine.  Aslo uses memcached which allow distributed
caching.  At our current scale APC would be better.... but memchached
scales better across multiple php and memcached servers.

> Note that proxying a PHP application with Squid generally does not buy
> anything, because all static files are already being served by Apache
> directly, which in turn uses mod_mem_cache.
>
> We may also add kludgy rules to cache the most frequently accessed pages
> for a few minutes (Wikipedia style), but this would also introduce
> complexity and fragility so I'd rather delay such optimizations until
> we'll run out of better options.

This is something that we can workout with Mozilla.  They have the
same problems.  They use proprietary traffic manager called zeus to
handle their reverse proxy/load balancing/ and log consolidation.
Having had some experience setting up proxies and load balancers I at
least have some idea of the right questions to ask....

> Now, the only purpose of going through Squid would be to balance traffic
> to multiple applications servers. Of which, we only have one at this
> time.

I would like to watch the load averages for the week on aslo-proxy and
sunjammer.  Squid is catching 80-90% of the hits before they get to
the php instance.  I want to see what effect that has on the load.

> Do we want to keep the aslo instance on sunjammer in production
> alongside the new one until we get a new machine?

I would like to run
aslo-proxy treehouse -> php on sunjammer -> mysql on sunjammer
for at least the next week.

During the week I would like to:
1. Test and tune aslo-web and aslo-db
2. Set up and test HAproxy and aslo-web2.


> ----------8<----------8<----------8<----------8<----------8<----------
>
> 130!aslo-proxy:~# ab  -n 1000 -c 4 http://aslo-proxy.sugarlabs.org/en-US/sugar/
> This is ApacheBench, Version 2.3 <$Revision: 655654 $>
> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
> Licensed to The Apache Software Foundation, http://www.apache.org/
>
> Benchmarking aslo-proxy.sugarlabs.org (be patient)
> Completed 100 requests
> Completed 200 requests
> Completed 300 requests
> Completed 400 requests
> Completed 500 requests
> Completed 600 requests
> Completed 700 requests
> Completed 800 requests
> Completed 900 requests
> Completed 1000 requests
> Finished 1000 requests
>
>
> Server Software:        Apache/2.2.11
> Server Hostname:        aslo-proxy.sugarlabs.org
> Server Port:            80
>
> Document Path:          /en-US/sugar/
> Document Length:        38132 bytes
>
> Concurrency Level:      4
> Time taken for tests:   100.580 seconds
> Complete requests:      1000
> Failed requests:        762
>   (Connect: 0, Receive: 0, Length: 762, Exceptions: 0)
> Write errors:           0
> Total transferred:      38528786 bytes
> HTML transferred:       38062786 bytes
> Requests per second:    9.94 [#/sec] (mean)
> Time per request:       402.322 [ms] (mean)
> Time per request:       100.580 [ms] (mean, across all concurrent requests)
> Transfer rate:          374.09 [Kbytes/sec] received
>
> Connection Times (ms)
>              min  mean[+/-sd] median   max
> Connect:        0    0   1.5      0      46
> Processing:   180  402 183.0    380    2782
> Waiting:      160  372 175.5    349    2668
> Total:        180  402 183.0    380    2782
>
> Percentage of the requests served within a certain time (ms)
>  50%    380
>  66%    417
>  75%    442
>  80%    464
>  90%    520
>  95%    585
>  98%    670
>  99%    929
>  100%   2782 (longest request)
>
>
> ----------8<----------8<----------8<----------8<----------8<----------
>
> 130!sunjammer:~# ab  -n 1000 -c 4 http://activities.sugarlabs.org/en-US/sugar/
> This is ApacheBench, Version 2.3 <$Revision: 655654 $>
> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
> Licensed to The Apache Software Foundation, http://www.apache.org/
>
> Benchmarking activities.sugarlabs.org (be patient)
> Completed 100 requests
> Completed 200 requests
> Completed 300 requests
> Completed 400 requests
> Completed 500 requests
> Completed 600 requests
> Completed 700 requests
> Completed 800 requests
> Completed 900 requests
> Completed 1000 requests
> Finished 1000 requests
>
>
> Server Software:        Apache/2.2.11
> Server Hostname:        activities.sugarlabs.org
> Server Port:            80
>
> Document Path:          /en-US/sugar/
> Document Length:        37935 bytes
>
> Concurrency Level:      4
> Time taken for tests:   101.718 seconds
> Complete requests:      1000
> Failed requests:        996
>   (Connect: 0, Receive: 0, Length: 996, Exceptions: 0)
> Write errors:           0
> Total transferred:      38487710 bytes
> HTML transferred:       38132710 bytes
> Requests per second:    9.83 [#/sec] (mean)
> Time per request:       406.873 [ms] (mean)
> Time per request:       101.718 [ms] (mean, across all concurrent requests)
> Transfer rate:          369.51 [Kbytes/sec] received
>
> Connection Times (ms)
>              min  mean[+/-sd] median   max
> Connect:        0    0   0.6      0      13
> Processing:   172  406 134.7    383    1642
> Waiting:      150  385 126.2    366    1496
> Total:        172  407 134.7    383    1642
>
> Percentage of the requests served within a certain time (ms)
>  50%    383
>  66%    423
>  75%    457
>  80%    473
>  90%    531
>  95%    585
>  98%    691
>  99%    838
>  100%   1642 (longest request)
> sunjammer:~#
>
>
> ----------8<----------8<----------8<----------8<----------8<----------
>
> bernie at aslo-web:~$ ab  -n 1000 -c 4 http://aslo-web.sugarlabs.org/en-US/sugar/This is ApacheBench, Version 2.3 <$Revision: 655654 $>
> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
> Licensed to The Apache Software Foundation, http://www.apache.org/
>
> Benchmarking aslo-web.sugarlabs.org (be patient)
> Completed 100 requests
> Completed 200 requests
> Completed 300 requests
> Completed 400 requests
> Completed 500 requests
> Completed 600 requests
> Completed 700 requests
> Completed 800 requests
> Completed 900 requests
> Completed 1000 requests
> Finished 1000 requests
>
>
> Server Software:        Apache/2.2.12
> Server Hostname:        aslo-web.sugarlabs.org
> Server Port:            80
>
> Document Path:          /en-US/sugar/
> Document Length:        38421 bytes
>
> Concurrency Level:      4
> Time taken for tests:   527.102 seconds
> Complete requests:      1000
> Failed requests:        999
>   (Connect: 0, Receive: 0, Length: 999, Exceptions: 0)
> Write errors:           0
> Total transferred:      38382416 bytes
> HTML transferred:       38150416 bytes
> Requests per second:    1.90 [#/sec] (mean)
> Time per request:       2108.409 [ms] (mean)
> Time per request:       527.102 [ms] (mean, across all concurrent requests)
> Transfer rate:          71.11 [Kbytes/sec] received
>
> Connection Times (ms)
>              min  mean[+/-sd] median   max
> Connect:        0    0   0.6      0      19
> Processing:  1264 2105 320.1   2074    3818
> Waiting:     1239 2077 316.8   2051    3784
> Total:       1264 2105 320.1   2074    3818
>
> Percentage of the requests served within a certain time (ms)
>  50%   2074
>  66%   2202
>  75%   2300
>  80%   2350
>  90%   2491
>  95%   2641
>  98%   2885
>  99%   3126
>  100%   3818 (longest request)
>
> --
>   // Bernie Innocenti - http://codewiz.org/
>  \X/  Sugar Labs       - http://sugarlabs.org/
>
>