[Systems] aslo cluster progress

[David Farning]

On Sat, Mar 13, 2010 at 12:31 PM, Bernie Innocenti <bernie at codewiz.org> wrote:
> On Sat, 2010-03-13 at 10:58 -0600, David Farning wrote:
>
>> Yes, we require master-master.  The goal here is if one machine to go
>> down, the other takes over.  Master-Master is the only arrangement in
>> which the second machine can take over the writes.
>
> Multi-master is really hard, IMHO. It even requires a special table
> format which probably performs worse and doesn't support some of the
> features of InnoDB.
>
> I was told that MySQL's master-slave configuration now supports
> automatic turn-around in case the master disappears. I can imagine
> scenarios in which such as setup like this might loose a transaction or
> two, but I guess we don't care since we're not running a bank.

I'll do some more reading about this.

>> As an FYI, as you predicted, database load is not going to be a
>> problem for a _long_.  I think we can handle about 10 php servers with


>> each database server.  Cache 1, memchached, has a hit rate in the mid
>> 80%. Cache 2, the mysql cache, has a hit rate in the low 90%.  I just
>> did a mysql dump and the size of the data base was just under 100MB
>>
> Good. I've seen the workload on memcached increase a lot over the past 3
> days. What happened? Did we turn on more aggressive caching?

I haven't turned the web node on treehouse back on since it crashed last week.  All of the traffic is still going through sunjammer again.  

> I'm very glad the load on Sunjammer is lower than before. Do you think
> we may need to setup one more web node on housetree?

We never exceeded 1/4 of the available cpu on housetree while sending 99% of the php traffic to the web-node.   If we let sunjammer carry part of the load we can handle 6 times the current load with the existing setup.

My concern is the networking on treehouse.  When treehouse hung last week and when it kept crashing last fall it always seemed to happen while under heavy network usage.

> For NFS, I might configure a VPN or something like that.

If it is not to hard, it is worth trying....

>
>> Now that the basic plumbing is in place it will be pretty time
>> effective for experienced admins like you and hhardy to review and
>> improve the manifests.
>
> I've just had a look at the proxy configuration. Is it using only one
> web node on treehouse, or is there also an implicit pass-through to
> sunjammer?
>
>
Proxy snippet<<<

<Proxy balancer://cluster>
       Order deny,allow
       Allow from 192.186.70.53

       BalancerMember http://192.168.90.2 loadfactor=1
#       BalancerMember http://192.168.90.4 loadfactor=100
</Proxy>

<VirtualHost *:80>
        ServerName activities.sugarlabs.org
        ServerAlias activities.sugarlabs.*
        ServerAdmin webmaster at sugarlabs.org

        <Location /balancer-manager>
                SetHandler balancer-manager

                Order Deny,Allow
                Allow from all
        </Location>

        LogLevel error

        ProxyPass /balancer-manager !
        ProxyPass / balancer://cluster/ lbmethod=byrequests
        ProxyPassReverse / balancer://cluster/

        LogFormat "%h %V %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" \"%{HTTP_COOKIE}i\"" addons
        CustomLog /srv/www-sugarlabs/activities/data/access_temp addons
        CustomLog /var/log/apache2/all-access.log vhost_combined
</VirtualHost>

<<<

The proxy listens for requests for activities.sl.org on eth0 (the public network).  It passes those requests to activities.sl.org on eth1. (the private network)

The individual web nodes are called 'BalancerMembers.'   192.168.90.2 is eth1 on sunjammer and 192.168.90.4 is eth1 on the treehouse webnode.  The confusing part is that I had to call both the proxy and the individual apaches vhosts activities.sl.o.  This was required by the session authorization in aslo.

david
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 272 bytes
Desc: OpenPGP digital signature
Url : http://lists.sugarlabs.org/private/systems/attachments/20100313/0a88f4e6/attachment.pgp