<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Fri, Jan 29, 2016 at 12:33 AM, Bernie Innocenti <span dir="ltr"><<a href="mailto:bernie@sugarlabs.org" target="_blank">bernie@sugarlabs.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span>On 01/28/2016 08:32 PM, Samuel Cantero wrote:<br>
> Apache wasn't the only process calling oom-killer. I found also<br>
> opendkim, spamc, mb, uwsgi, and smtpd.<br>
><br>
> The first incident was at Jan 28 03:07:25. Usually we have a lot of<br>
> memory available in sunjammer. Munin stopped plotting at 02:40 and the<br>
> memory was low as expected. I just can only imagine some kind of<br>
> unmanaged over-commitment (over-provisioning) in the Xen Dom0.<br>
<br>
</span>I don't think the Dom0 can steal ram from the domU. That's fixed, unless<br>
you use those weird virtio balloon devices.<br></blockquote><div><br></div><div>As far as I know, Xen have the memory over-commit feature. With this feature, memory is taken from one domain and given to another using the xen "ballooning" mechanism. My simple theory is that this feature is enabled in the Xen instance running at the FSF and suddenly the host got overloaded without the capacity to provide the memory needed for our VM.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
I'm not really sure what allocated all the memory, but it had to be<br>
something internal. The kernel dumped a list of processes and their<br>
memory usage at each oom iteration, but none of them is particularly big.<br>
<br>
The real memory usage of apache is very hard to estimate, because it<br>
forks plenty of children, each with big VSS and RSS figures. However,<br>
most of the pages should be shared, so they don't add up. </blockquote><div><br></div><div>Currently, we are running apache in prefork mode (multiple child processes with one thread each). I have done a little test in order to sizing it now. We can check again this value when apache is crawling.</div><div><br></div><div>Total number of apache2 processes: <b>53</b></div><div><br></div><div><font face="monospace, monospace">ps aux | grep 'apache2' | grep -v grep | wc -l</font></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">Our server limit value is 512.</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="arial, helvetica, sans-serif">Total RSS (resident set size) value for the total number of apache2 processes: </font><span style="color:rgb(0,0,0)"><font face="arial, helvetica, sans-serif"><b>3553.78 MB</b></font></span></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">ps aux | grep 'apache2' | grep -v grep | awk '{sum += $6;} END {print sum/1024 " MB";}'</span><br></span></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)"><br></span></span></div><div><font color="#000000" face="arial, helvetica, sans-serif">According to ps man page, the RSS is </font><span style="color:rgb(0,0,0)"><font face="arial, helvetica, sans-serif">the non-swapped physical memory that a task has used. However, this sum is not related with the free cmd output.</font></span></div><div><span style="color:rgb(0,0,0)"><font face="arial, helvetica, sans-serif"><br></font></span></div><div><span style="font-family:monospace"><span style="color:rgb(178,24,24)">scg</span><span style="color:rgb(24,178,178)">@</span><span style="font-weight:bold;color:rgb(255,255,84);background-color:rgb(24,24,178)">sunjammer</span><span style="color:rgb(24,178,178)">:</span><span style="color:rgb(24,178,24)">~</span><span style="color:rgb(24,178,178)">$</span><span style="color:rgb(0,0,0)"> free -m
</span><br> total used free shared buffers cached
<br>Mem: 11992 10848 1144 0 682 8036
<br>-/+ buffers/cache: 2128 9864
<br>Swap: 8191 0 8191<br></span></div><div><font face="monospace, monospace"><br></font></div><div><font face="arial, helvetica, sans-serif">Where the used memory is only 2 G. The rest is cached memory. Do am I missing something?</font></div><div><font face="monospace, monospace"><br></font></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span>
<br>
> Regarding to disk I/O:<br>
><br>
> iostat shows:<br>
><br>
</span>> * An average of 32 tps (IOPS) in the first partition (/root). iostat<br>
<span>> -x shows an average latency (await) of 126 ms. The 25% are read<br>
> operations and the 75% are write operations. Munin shows an average<br>
> latency of 145 ms since we're running diskstats plugin.<br>
</span>> * An average of 26 tps in the third partition (/srv). iostat -x shows<br>
<span>> an average latency of 16.5 ms. The 81% are read operations and the<br>
> 19% are write operations. Munin shows an average latency of 14.5 ms.<br>
><br>
> sar -dp -f /var/log/sysstat/sa[day] shows (for some days):<br>
><br>
</span>> * Jan 27:<br>
> o An avg of 26 tps (IOPS) in the first partition (/root). An avg<br>
> latency of 126 ms.<br>
> o An avg of 11 tps in the third partition (/srv). An avg latency<br>
> of 29 ms.<br>
> o<br>
><br>
> * Jan 26:<br>
> o An avg of 27 tps (IOPS) in the first partition (/root). An avg<br>
> latency of 126 ms.<br>
> o An avg of 11 tps in the third partition (/srv). An avg latency<br>
<span>> of 29 ms.<br>
><br>
> I can check this avg in the other days.<br>
><br>
> As we can see, we have a high latency on the first partition (where<br>
> databases reside) and taking into account that our VM is struggling for<br>
> disk I/O in an old disk subsystem, it is likely that 37 IOPS would be a<br>
> big part of the total maximum IOPS value.<br>
<br>
</span>Great analysis. Ruben and I upgraded the kernel to 3.0.0, which is still<br>
ancient, but at least better than what we had before. We also disabled<br>
barriers, which might not play well with the dom0 which is also running<br>
a very old kernel.<br>
<br>
Let's see if this brings down the damn latency.<br>
<div><div><br>
--<br>
Bernie Innocenti<br>
Sugar Labs Infrastructure Team<br>
<a href="http://wiki.sugarlabs.org/go/Infrastructure_Team" rel="noreferrer" target="_blank">http://wiki.sugarlabs.org/go/Infrastructure_Team</a><br>
</div></div></blockquote></div><br></div></div>