[Systems] a.sl.o progress

[David Farning]

On Wed, Nov 11, 2009 at 11:51 AM, Bernie Innocenti <bernie at codewiz.org> wrote:
> El Wed, 11-11-2009 a las 01:58 -0600, David Farning escribió:
>> 1. Individually tune the components.  Because of the numerous systems
>> interacting on sunjammer I have been having trouble determining the
>> meaning of benchmarks.
>
> There will still be the same amount of interaction if we split the
> components to many virtual machines sharing resources on the same host.
> It may actually become a little harder to analyze.
>
> On top of this, components that are tightly coupled, such as the
> database, the application server and the proxy will suffer a big
> performance hit if the communication has to happen over TCP/IP sockets.
> Virtual machines are known to pay a high overhead for networking.
>
> Let's measure, but I'm worried that the overhead we're adding might
> surpass the benefits of doubling the amount of resources.

+1 This is going to be a iterative process of learning what and how to
measure.

>> 2. We will be able to move the vms onto seperate physical machines as necessary.
>> 3. There is no direct interaction between the aslo cluster and
>> sunjammer.  Rather than setting up nfs between download.sl.o and
>> sunjammer, I am running rsync every ten minutes.  If sunjammer is
>> compromised it will not affect aslo.
>
> Good idea, but how do we get it to work when we have, say, 3 aslo
> frontends running concurrently?

Sorry for the lack of clarity.  The various aslo frontends (we have
been calling them also-web) will depend on shared nfs mount point.

The separation I was writing about was between the a.sl.o cluster and
the rest of the infrastructure.  If either a.sl.o or sunjammer is
compromised, the intruder can't get into the other system.

> If files are never removed or renamed, it may be possible to setup rsync
> in both directions.

Files are never removed from /activities/files (The internal file tree).

Files can be removed for /downloads/activities . (The public file
tree) If a file is found to be buggy or have questionable licensing
issues  we need to be able to remove it from the download tree.

>
>> So far we have a test instance running at
>> http://140.186.70.144/en-US/sugar/ .  Below is to do list for the rest
>> of the week:)   It looks like it will continue to be a rather
>> iterative process.  I hope we can finish setting it up this week.
>> Benchmark and test it next week.  And go live the following week.
>
> Good work both of you, and thanks for this nice summary!

aslo-proxy  -- http://140.186.70.121 is a working squid reverse proxy.
 It is caching all of the static content (css, js, gif, png...) before
calling aslo-web.

aslo-web -- http://140.186.70.123 is a working aslo php server.  The
data is a snapshot from activites.sl.o from two days ago.

aslo-db -- http://140.186.70. is a working master database server.

It looks like today will:
1. Work on testing the backup and restore on each machine.
2. Add Munin Monitoring to each machine.
3. Create database slave which replicates from aslo-db.  This requires
configuring aslo-web is write to the master and read from the slave.

4.  If their is time available I would like to clone aslo-web and set
up aslo-web-loadbalancer.

Just to repeat.... This is architectural design is huge overkill for
our current load levels.  But, it will allow us to scale gracefully by
2 to 3 orders of magnitude.

> For benchmarking, I suggest a simple apachebench (ab) on the front page.

For benchmarking I have been using siege.  (
http://www.joedog.org/index/siege-home )  I have parsed a subset of
access-all.log from sunjammer and feed them into siege.  This gives us
a reasonable representation of the pages users actually access.

david