[Systems] CPU usage spikes caused by aggressive buildbot

[Bernie Innocenti]

Since a couple of weeks ago, we've started seeing daily spikes of abnormal
"system time" usage on sunjammer:

  http://munin.sugarlabs.org/sugarlabs.org/sunjammer.sugarlabs.org/cpu.html

Today I finally figured out that buildbot pulls from a dozen git repos *in parallel*,
probably causing disk trashing:

27135 ?        Sl   138:55 /srv/www-sugarlabs/buildbot/sandbox/bin/python -c from twisted.scripts import twistd; twistd.run() --no_save --logfile=twistd.log --python=buildbot.tac
 3559 ?        S      0:00  \_ git fetch https://src.sugarlabs.org/jukebox/mainline.git +master:refs/buildbot/https%3A%2F%2Fsrc.sugarlabs.org%2Fjukebox%2Fmainline.git/master
 3573 ?        R      1:53  |   \_ git-remote-https https://src.sugarlabs.org/jukebox/mainline.git https://src.sugarlabs.org/jukebox/mainline.git
 3571 ?        S      0:00  \_ git fetch https://src.sugarlabs.org/pippy/mainline.git +gtk3:refs/buildbot/https%3A%2F%2Fsrc.sugarlabs.org%2Fpippy%2Fmainline.git/gtk3
 3589 ?        R      1:53  |   \_ git-remote-https https://src.sugarlabs.org/pippy/mainline.git https://src.sugarlabs.org/pippy/mainline.git
 3657 ?        S      0:00  \_ git fetch https://src.sugarlabs.org/terminal/mainline.git +gtk3:refs/buildbot/https%3A%2F%2Fsrc.sugarlabs.org%2Fterminal%2Fmainline.git/gtk3
 3670 ?        R      1:54  |   \_ git-remote-https https://src.sugarlabs.org/terminal/mainline.git https://src.sugarlabs.org/terminal/mainline.git
 3664 ?        S      0:00  \_ git fetch https://src.sugarlabs.org/log/mainline.git +master:refs/buildbot/https%3A%2F%2Fsrc.sugarlabs.org%2Flog%2Fmainline.git/master
 3683 ?        R      1:54  |   \_ git-remote-https https://src.sugarlabs.org/log/mainline.git https://src.sugarlabs.org/log/mainline.git
 3710 ?        S      0:00  \_ git fetch https://src.sugarlabs.org/sugar-base/mainline.git +master:refs/buildbot/https%3A%2F%2Fsrc.sugarlabs.org%2Fsugar-base%2Fmainline.git/master
 3724 ?        R      1:54  |   \_ git-remote-https https://src.sugarlabs.org/sugar-base/mainline.git https://src.sugarlabs.org/sugar-base/mainline.git
 3728 ?        S      0:00  \_ git fetch https://src.sugarlabs.org/chat/mainline.git +gtk3:refs/buildbot/https%3A%2F%2Fsrc.sugarlabs.org%2Fchat%2Fmainline.git/gtk3
 [...and many more...]


Is there a way to run these sequentially? Additionally, since this is a batch process,
can we run the whole thing with lower priority ("nice ionice -c2 -n6 COMMAND").

-- 
 _ // Bernie Innocenti
 \X/  http://codewiz.org