[Systems] pointing activities.sugarlabs.org at the proxy

[David Farning]

On Mon, Nov 16, 2009 at 10:05 AM, Bernie Innocenti <bernie at codewiz.org> wrote:
> El lun, 16-11-2009 a las 02:27 -0600, David Farning escribió:
>
>> Let's agree to disagree on this until we get the benchmarks in place.
>> The knowledge I am gaining running reliable benchmarks is probab
>> worth the effort.  If the the experiment fails, I will gladly throw it
>> out.
>
> Ok. This effort is certainly justified for the sake of science and
> good-looking charts :)
>
>
>> Ideally this will be the case.  Have you talked to Steve at RIT?
>> Apparently they have an entire rack of blade servers sitting idle in
>> the inovation center.
>
> That would be amazing. I'll write him again.

Walter is going up to give a distinguished lecturer talk in a few
weeks.  It might be good for us to coordinate the ask.

>> > Actually, I'd propose an opposite strategy: create MySQL replica slaves
>> > on each aslo-web node to maximize throughput.
>>
>> Eventually we are going to need both.  The downsides of depending on
>> master slave databases for scalability is that:
>> a. each db server caches that same hot data.  -  If we set up a
>> master-slave combo each with 8GB of cache, the same 8GB of objects
>> with be cached on each machine.  Memcached, while slower, shares the
>> same cache across multiple php servers and db servers.
>
> Hmmm.... it's "more data sharing" VS "more network traffic". Hard to
> tell which one would perform better. I guess we could measure in both
> configurations to find out what the impact really is.
>
>
>> Currently,
>> aslo is getting a ratio of about 90% reads to 10% writes.
>> aslo-db is getting a cache hit rate of just below 90% with 4GB of
>> memory and 3GB of dedicated innodb_cache.
>> The entire database is about 70GB
>
> Does it make sense to have the table cache bigger than the actual data
> set? (the query cache may, indeed, need to be much bigger than the
> dataset).

Yes, these are the pieces that I am trying to working out:
-The entire data base was about 70 GB when I took the snapshot last week.

-The database uses both innobd and myisam tables.  All of the
literature suggests that you don't mix table types unless you know
what you are doing.... because, while it can be more effective, it
make things hard to tune.   aslo-db currently has 4GB assigned, I will
need to figure out how to best allocate that.

-Memcache is living on aslo-web. I have it set for .5GB  with 2.5GB
available for the the apache and php processes.  I have bumped the
time to store page in memcache from 60 seconds to 300 seconds.  This
increased the memcache hit rate from mid 70% to mid 90%.

-Bernie was correct in his assumption of Squid.  The static files that
I have been worried about are generally cached locally in the browser.
 On the other hand, with running squid in front of the php server for
siege benchmarks there seems to be about a 10% to 15% increase in
transactions per second.

-My best result so far is 24 transactions per second.  I had to let it
run for 6 hours undisturbed to completely warm the caches (I haven't
been able to reproduce it)  Sunjammer maxed out at 13 transactions per
second.

>> Yes, I agree VMs incur an unavoidable cost in complexity, fragility,
>> and overhead.  At our currently scale they are nothing more than a
>> pain in the ass:(  Note--dfarning does not like using vms
>> unnecessarily in production.
>>
>> My goal is to learn to set up, learn to test, learn to benchmark, and
>> learn to maintain aslo when it is spread across multiple physical
>> machines.
>
> I agree, this is a valuable goal even if it makes performance suffer
> initially.
>
>
>> > If you agree, I'll migrate also-db, aslo-web and aslo-proxy to their
>> > partitions.
>>
>> Yes, I think that would help significantly.
>
> Ok then, I'll see if I could do it on the fly.

Berne, can you try to move the machine to partitions on lvm?

I think the setup is getting close to useable.  But I am still
concerned that the load average goes up before we max out the cpus.
Is this a vm related issuse?  The interrupts at
http://sunjammer.sugarlabs.org/munin/sugarlabs.org/aslo-db.sugarlabs.org-irqstats.html
look pretty high.

david

>> yummmm.  I have been going to a different pizzeria every evening for a
>> pizza and beer.  I'll take you up on that bet.
>
> GRRR!!! You guys had way too much fun in Bolzano!
>
> Make sure you write a great report to make the rest us even more
> envious.
>
> --
>   // Bernie Innocenti - http://codewiz.org/
>  \X/  Sugar Labs       - http://sugarlabs.org/
>
>