[Systems] Docker monitoring

[Samuel Cantero]

Hi all,

I've been investigating how to setup a docker monitoring system for our
docker infra. Basically the solutions consists of three components:

   - cAdvisor to monitor the resource (CPU and memory) usage in containers.
   By default, cAdvisor stores the historical data for 1 minute in memory.
   - In order to keep data over time we need a time series database. We
   have two main options: influxDB and prometheus.
   - In order to have some nice charts we can use grafana (a graph editor
   for influxdb) or promdash (dashboard builder for Prometheus).

The preferred stack is cadvisor+prometheus+promdash (or grafana). Why?
Because cAdvisor exposes container statistics as Prometheus metrics out of
the box.

As you can see, we need at least three components in order to achieve
docker monitoring. Furthermore, cAvisor consumes a lot of CPU (~20%) and
prometheus a lot of RAM (~3GB). You can check this on cAdvisor.

For this reason, I've written a plugin for munin to monitor docker
container memory usage. This way we can leverage our munin installation
(and its RRDtool) and avoid the cadvisor+prometheus+promdash stack :)

You can check the munin graph in the following URL:

http://munin.sugarlabs.org/sugarlabs.org/freedom.sugarlabs.org/docker_memory.html

The next step is to write another plugin to monitor docker container CPU
usage. This is a little more complicated. The jiffy value is a whole new
world for me. I have to investigate how to get the CPU usage in % using the
cumulative jiffie value exposed by the control groups on a
pseudo-filesystem (similar to /proc/stat).  Any help is welcome.

*Off-topic: *In order to improve our wiki response time I've tried to
deploy mediawiki with HHVM but it was not successful. I am not sure if they
are working on it. The mediawiki last version has experimental support for
HHVM 3.3.0 [1]. The last available HHVM version is 3.9.1.

Greetings,

[1] https://www.mediawiki.org/wiki/Release_notes/1.25
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sugarlabs.org/private/systems/attachments/20150908/7356cf94/attachment.html>