[Systems] pointing activities.sugarlabs.org at the proxy

[David Farning]

Sorry for going AWOL while travelling home from Bolzano.  Italy seemed
pretty unfriendly to anonymous internet connections from random coffee
shops.

On Wed, Nov 18, 2009 at 11:40 AM, Bernie Innocenti <bernie at codewiz.org> wrote:
> El lun, 16-11-2009 a las 23:32 -0600, David Farning escribió:
>
>> Yes, these are the pieces that I am trying to working out:
>> -The entire data base was about 70 GB when I took the snapshot last week.
>
> Err.. 70MB, not 70GB.

Opps

> The entire mysql tablespace on sunjammer, comprising all databases, is
> 788MB.
>
>> -The database uses both innobd and myisam tables.  All of the
>> literature suggests that you don't mix table types unless you know
>> what you are doing.... because, while it can be more effective, it
>> make things hard to tune.
>
> Yeah, and now I also know the specific details: over the week-end I've
> been digging through Drizzle's storage back-end API :-)

I spend the time on the air plane reading 'high performance mysql'  so
I have some idea what to look for now.

>>   aslo-db currently has 4GB assigned, I will
>> need to figure out how to best allocate that.
>
> I think it's still an order of magnitude too much. In order to tune
> these parameters, the best approach is to let the application run for a
> while with small (default) caches and then collect runtime stats about
> what caches were churning too much.

Based on the the fact the the the entire db is less than 1G we should
be able to store everything in memory.  I am not sure how to tell what
specific cache is churning.....

> phpMyAdmin offers a *great* statistics dashboard with plenty of good
> advice. It's the only feature I'd use it for.

Is phpmyadmin safe on a production machine?  I have been looking for
substitutes when ever possiable....

>
>> -Memcache is living on aslo-web. I have it set for .5GB  with 2.5GB
>> available for the the apache and php processes.  I have bumped the
>> time to store page in memcache from 60 seconds to 300 seconds.  This
>> increased the memcache hit rate from mid 70% to mid 90%.
>
> Ok.
>
>
>> -Bernie was correct in his assumption of Squid.  The static files that
>> I have been worried about are generally cached locally in the browser.
>>  On the other hand, with running squid in front of the php server for
>> siege benchmarks there seems to be about a 10% to 15% increase in
>> transactions per second.
>
> Perhaps the Mozilla people could give us an indication of how they
> configured Squid.

I'll ping them.

>> -My best result so far is 24 transactions per second.  I had to let it
>> run for 6 hours undisturbed to completely warm the caches (I haven't
>> been able to reproduce it)  Sunjammer maxed out at 13 transactions per
>> second.
>
> Do you know what's causing this 2x increase in throughput? On what
> page(s) was it?

The pages were randomly pinging pages on the url list
http://people.sugarlabs.org/dfarning/working_subset.10000.aslo-proxy.
I think it has to do with the all of the automatic caches being warm.
We can configure specific caches for the various tools, but the kernel
does alot of automatic LRU caching behind the scenes.  I think that
all of the other process on sunjammer were dirtying the caches.

>> > Ok then, I'll see if I could do it on the fly.
>>
>> Berne, can you try to move the machine to partitions on lvm?
>
> I tried, it's not as easy as I thought. I'd need a package called
> guestfs-tools, which is not yet available for Ubuntu and has some weird
> dependencies such as ocaml.
>
> I'll do it this week-end from within the virtual machines, but it will
> require restarting them twice.

It is fine to kill them.

> We don't have to wait until I'm done before switching
> activities.sugarlabs.org to production. If we're ready, I'll switch the
> DNS tonight.

We have a couple of things to up date first.  I'll need to go through
the rest of this thread in detail.  (it looks like a lot was happening
when I was enjoying myself)


>> I think the setup is getting close to useable.  But I am still
>> concerned that the load average goes up before we max out the cpus.
>> Is this a vm related issuse?
>
> Yeah, this is really weird, I can't make sense of it. It may be
> interference from the other XEN domains sharing the physical machine
> (cloud9).
>
> The load on that box should go down soon, when 2 new machines I'm
> setting up for the FSF will transition into production.

Very nice.

>>   The interrupts at
>> http://sunjammer.sugarlabs.org/munin/sugarlabs.org/aslo-db.sugarlabs.org-irqstats.html
>> look pretty high.
>
> Hmm... I can't check now because I'm riding the T. If it's less than
> 1000 IRQs per second, then it's fine.
>
> I'll investigate with powertop, which has the nice side-effect of
> telling you what process is causing more interrupts.

Thanks for taking the time to look into this....  I understand that it
would have been easier to do yourself:)

> --
>   // Bernie Innocenti - http://codewiz.org/
>  \X/  Sugar Labs       - http://sugarlabs.org/
>
>
>
>