[Systems] Wiki speed

[Samuel Cantero]

On Wed, Jun 24, 2015 at 12:29 PM, Bernie Innocenti <bernie at codewiz.org>
wrote:

> On 24/06/15 04:02, Sam P. wrote:
>
>> Hi All,
>>
>> I just saw that the docker container on freedom that does the
>> magical-wiki-visual-editor-node-js-service was not running, so that was
>> why only the edit source thing was working :)
>>
>> IDK what happened there - did somebody restart docker on freedom?  or
>> restart freedom?  all of the containers were down :(
>>
>> So about the visual editor, here were my tests on wiki-devel (running
>> "time curl
>> http://wiki-devel.sugarlabs.org/go/Welcome_to_the_Sugar_Labs_wiki >
>> /dev/null" on sunjammer):
>>
>> * ~2sec average time with visual editor and all the others
>> * ~1.5sec without visual editor => probs a bit of network cost, but not
>> the whole thing
>> * ~1.7sec without mobile front end (but with VE and all the others)
>> * ~2sec without the media viewer lightbox (but with VE and all the others)
>>
>> Don't trust my test results, but  it would probably help a little to
>> move VE together with the rest of the wiki.
>>
>> It would be pretty cool to containerize or chuck the wiki in a new VM.
>> That would make it eaiser to move the VE thing onto the same machine.
>> Moving it onto a newer OS could also let us play with HHVM, which seems
>> to work nicely with media wiki [1][2].
>>
>
> LGTM. Can you and the other Sam (scg) work together on this?
>

Of course. I couldn't get in contact with Sam yet but I have been working
on this and I have some interesting things to tell you.

>
> I think the current docker setup is still too sketchy to run on justice
> alongside other production VMs. We need to ensure that each container has
> hard limits on all resources: cpu, memory, disk space and possibly even I/O
> bandwidth.
>

Sure. I want to tell you about my research. I was reading about docker
runtime constraints on resources. By default, the memory and swap
accounting is disable on ubuntu 14.04. We need to enable it on GRUB setting
the following line:

GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"

Then we have to update the grub (sudo update-grub) and pitifully reboot the
system. I have checked it on freedom and it is not activated.

Currently, we can limit the memory for a container in docker. However, by
default, the container can use all of the memory on the host (as much
memory as needed). Another interesting fact: the swap memory will be
*unlimited* too.

We have to change this in our actual configuration because having no limit
on memory can lead to issues where one container can easily make the whole
system unstable and as a result unusable.

To check how this works in Docker, I used the *stress* tool (inside a
container) that helped me out to generate some load in the containers so I
could actually check for the resource limits being applied. The results are:

*MEMORY*

We can limit the amount of memory for a docker container (tested). By
default, the amount of the swap memory will be the same. We can set
different values for the swap memory but we can not disable it entirely.

A container with a memory limit of 256 MB will die If we stress out the
container with the twice of the assigned memory. The container uses all of
the memory and all of the swap and then die.

*CPU*

When I run the stress tool (inside the container) with 2 workers to impose
load on to the CPU, I can see that each process consume 50% of CPU. Yes, by
default, a container can use a 100% of the CPU. And If we have many cores,
it can use all the cores available.

Docker lets you specify a CPU share, but this is not so useful to actually
limit the physical cores (I guess). The CPU share is just a proportion of
CPU cycles. This is a relative weight (between containers) and has nothing
to do with the actual processor speed. Besides, the proportion will only
apply when CPU-intensive processes are running. When tasks in one container
are idle, other containers can use the left-over CPU time. As you might
guess, on an idle host, a container with low shares will still be able to
use 100% of the CPU. On a multi-core system, the shares of CPU time are
distributed over all CPU cores. Even if a container is limited to less than
100% of CPU time, it can use 100% of each individual CPU core.

As you can see, there is no way to say that a container should have access
only to 1 GHz of the CPU. Therefore, what can we do then?

1- Limit the container's CPU usage in percentage. For example, we can limit
the container to 50% of a CPU resource using CPU quota in docker.

When I ran the stress tool (inside the container) with 2 workers to impose
load on to the CPU, I saw that each process consumed around 25% of CPU
(values: 25.2 and 24.9, 25.2 and 25.2, 25.6 y 24.9 and so on). However, I
have to test it in a multi-core VM and perform other testing.

This is just a first test. I want to continue reading about CPU quota and
CPU period constraints in Docker.

2- Pin a container to a specific core, i.e, set cpu or cpus in which to
allow execution for containers.

*DISK and I/O Bandwidth*

I couldn't work on this yet.


> Without resource isolation, it's only a matter of time until a bug in one
> container takes down everything, including our primary nameserver on
> lightwave, the VMs for Paraguay and so on.
>
> Secondarily, we need some sort of monitoring to see things like memory
> leaks and overloaded services. Without good graphs, setting resource limits
> becomes a game of guessing.
>

I am agree. I will search for some monitoring tools.

Can we solve both these problems before moving the wiki? I'm sure docker
> deployments around the world have similar requirements and developed
> similar solutions.
>
> --
>  _ // Bernie Innocenti
>  \X/  http://codewiz.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sugarlabs.org/private/systems/attachments/20150626/8c4b0f74/attachment.html>