[Systems] Load on treehouse

[Bernie Innocenti]

On 09/29/10 22:04, Sascha Silbe wrote:
>> Adding some swap would help this situation by paging out
>> rarely used data and thus lowering the pressure on the VM system.
> 
> There was almost no disk IO when this happened, only CPU churn.

Then perhaps it's something really idiotic: ping-pong between the
buffercache (aka regular vm) and the I/O buffers.


>> If you have twice the physical memory for your working set, this problem
>> should never happen.
> 
> Iff, yes. But unless you apply carefully tuned memory limits to all
> processes there's no guarantee your physical memory will suffice.

Theoretically, there's no guarantee that *any* limited amount of memory
will suffice, with or without swap. In practice, server memory tuning
can be done by measuring actual usage in production plus good safety
margins.


> I definitely consider it a bug. Unfortunately I always have other,
> subjectively more important things to do. Some day I'll try to design
> a test case, but for now the workaround is easy enough.
> Maybe I should add it as dev-love on bugs.sl.o. ;)

Ok, let's wait a few more days to see if treehouse behaves better with
swap disabled. So far, it really seems better, but only God and Munin's
weekly chart can tell us for sure :-)


> It doesn't matter as much on the current / previous generation of
> embedded devices as they run only a very limited set of software.

Treehouse is even more limited: its job is to schedule a fixed number of
kvm processes with a fixed memory size, plus a bunch of comparatively
tiny system processes.

It's quite a stable memory regime compared to the typical NAT router.


> It doesn't matter whether the process gets killed due to OOM or the kernel
> goes into a tight loop. To the user, the box just doesn't work anymore
> and gets rebooted (or the 24h on-site service called, depending on
> whether the box is operated by a home luser or controls the coffee
> supply of $BIG_BOSS).

Hypothetically, yes. In the specific case of treehouse, the OOM should
never kick in with our current workload. Since we had 5GB of swap used,
it might be that it does, in fact, kick in some time in the future,
perhaps unveiling the bug that we were looking for.

-- 
   // Bernie Innocenti - http://codewiz.org/
 \X/  Sugar Labs       - http://sugarlabs.org/