<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 18, 2015 at 3:14 PM, Samuel Greenfeld <span dir="ltr"><<a href="mailto:samuel@greenfeld.org" target="_blank">samuel@greenfeld.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Unless the hardware is newer than I think it is, it likely is quite old.<br><br></div></div></div></div></blockquote><div>< 3 years</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div></div>OLPC's hardware in the Media Lab kept flaking out to the point most (all?) of it was eventually virtualized.<br><br></div>How much would it cost to look into getting new hardware and/or using someone's virtualization platform?<br><br></div>Sugar seems to change their setup a bit more than OLPC, so it may be worth investigating a scenario where resources could be spun up on demand.<br><br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Thu, Jun 18, 2015 at 3:09 PM, Bernie Innocenti <span dir="ltr"><<a href="mailto:bernie@sugarlabs.org" target="_blank">bernie@sugarlabs.org</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><span>On 06/18/2015 03:01 PM, Gonzalo Odiard wrote:<br> > Any chance to check if disks are dying or there other reason for these<br> > instabilities?<br> <br> </span>Nothing odd from smartctl, and anyway the server would keep responding<br> to pings even if both disks in the raid array were dead.<br> <br></div></div></blockquote></div></div></blockquote><div>+1 +2 +3 +...</div><div><br></div><div>on the console the system completely freezes </div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div> So I'm thinking it's either a kernel bug, or unstable hardware.<br> <span><br> <br> > Gonzalo<br> ><br> > On Thu, Jun 18, 2015 at 3:56 PM, Bernie Innocenti <<a href="mailto:bernie@sugarlabs.org" target="_blank">bernie@sugarlabs.org</a><br> </span><div><div>> <mailto:<a href="mailto:bernie@sugarlabs.org" target="_blank">bernie@sugarlabs.org</a>>> wrote:<br> ><br> > +systems@<br></div></div></div></div></blockquote></div></div></blockquote><div><br></div><div>thanks bernie</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div><div> ><br> > I rebooted justice from the management console and it's now responding<br> > to pings.<br> ><br> > I couldn't view the screen capture and I had no time to go to the Media<br> > Lab to physically inspect the machine, so I don't understand the<br> > root cause.<br> ><br> > As reported by Dogi, Justice seems to crash every 1-2 months.</div></div></div></div></blockquote></div></div></blockquote><div> </div><div>more ~3 months </div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div><div>> I suggest we try the following steps:<br> ><br> > 1. upgrade justice to Ubuntu 14.04 (like we did with freedom 1yr ago)<br> ><br></div></div></div></div></blockquote></div></div></blockquote><div>+1 specially since justice compared to freedom has a long history of being upgraded (it has its roots on housetree server build which means it lives already since 2009 ... freedom got fresh installed ~2012/13)</div><div><br></div><div>this is why after that we should just consider a total new install of justice, since my guess is that it is a software issue (justice always lasts 2+ months)</div><div><br></div><div>why I think it is not a hardware issue is that this crashing is the case already for the last 2 years (I did 95% of all reboots) and started with our last system upgrade (something got upgraded to unstable)</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div><div> > 2. if crashes continue, go to the server room and swap the drives with<br> > freedom (which is our hot-swap server and doesn't currently run anything<br> > critical)<br> ><br> > 3. Ask again the ML to give one of us physical access to the server<br> > room. I work nearby, but I have trouble leaving during office hours on a<br> > personal errand and if anything happens over a week-end we're in<br> > trouble.<br> ><br> > Sebastian: you should at least get access to the management console.<br> > Ping me on IRC and I'll send you the credentials on a secure channel.<br> ><br> ><br> > On 06/18/2015 10:40 AM, Sebastian Silva wrote:<br> > > Hello Sugar Oversight Board, Sugar Labs Members,<br> > ><br> > > Our main production server virtual machine host is down and I can't<br> > > reach it.<br> > > We have several systems that depend on this infrastructure, including<br> > > pootle server which was actively being used by translators of<br> > Aymara and<br> > > Awajun native languages.<br> > ><br> > > I respectfully request that you call on the phone whoever has physical<br> > > access to this machine and we try to bring it back online. I think<br> > this<br> > > should be either Bernie Inocenti or Stefan Unterhauser.<br></div></div></div></div></blockquote></div></div></blockquote><div><br></div><div>my phonenumber is <a href="tel:617%20767%202668" value="+16177672668" target="_blank">617 767 2668</a> ... just call</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div><div> > ><br> > > Also, I would like to request for more volunteers from infrasctucure<br> > > team to have virtual terminal access to these machines (not just ssh),<br> > > or to put them in a proper collocation service where we can get some<br> > > support.<br> > ><br> > > Thanks in advance for your help.<br> > > Sebastian<br> > ><br> > > On 17/06/15 20:55, Sebastian Silva wrote:<br> > >> Affected services:<br> </div></div>> >> <a href="http://translate.sugarlabs.org" rel="noreferrer" target="_blank">translate.sugarlabs.org</a> <<a href="http://translate.sugarlabs.org" rel="noreferrer" target="_blank">http://translate.sugarlabs.org</a>><br> > >> <a href="http://git.sugarlabs.org" rel="noreferrer" target="_blank">git.sugarlabs.org</a> <<a href="http://git.sugarlabs.org" rel="noreferrer" target="_blank">http://git.sugarlabs.org</a>><br> > >> <a href="http://packages.sugarlabs.org" rel="noreferrer" target="_blank">packages.sugarlabs.org</a> <<a href="http://packages.sugarlabs.org" rel="noreferrer" target="_blank">http://packages.sugarlabs.org</a>><br> <span>> >><br> > >><br> > >><br> > >> On 17/06/15 20:48, Sebastian Silva wrote:<br> > >>> We can't reach it.<br> > >>><br> > >>> Anybody with physical access to the machine please respond.<br> > >>><br> > >>><br> > >>> Regards,<br> > >>> Sebastian<br> ><br> > --<br> > Bernie Innocenti<br> > Sugar Labs Infrastructure Team<br> > <a href="http://wiki.sugarlabs.org/go/Infrastructure_Team" rel="noreferrer" target="_blank">http://wiki.sugarlabs.org/go/Infrastructure_Team</a><br> > _______________________________________________<br> > IAEP -- It's An Education Project (not a laptop project!)<br> </span>> <a href="mailto:IAEP@lists.sugarlabs.org" target="_blank">IAEP@lists.sugarlabs.org</a> <mailto:<a href="mailto:IAEP@lists.sugarlabs.org" target="_blank">IAEP@lists.sugarlabs.org</a>><br> </div></div><div><div><div><div>> <a href="http://lists.sugarlabs.org/listinfo/iaep" rel="noreferrer" target="_blank">http://lists.sugarlabs.org/listinfo/iaep</a><br> ><br> ><br> ><br> ><br> > --<br> > Gonzalo Odiard<br> ><br> > SugarLabs - Software for children learning<br> <br> <br> --<br> Bernie Innocenti<br> Sugar Labs Infrastructure Team<br> <a href="http://wiki.sugarlabs.org/go/Infrastructure_Team" rel="noreferrer" target="_blank">http://wiki.sugarlabs.org/go/Infrastructure_Team</a><br> _______________________________________________<br> IAEP -- It's An Education Project (not a laptop project!)<br> <a href="mailto:IAEP@lists.sugarlabs.org" target="_blank">IAEP@lists.sugarlabs.org</a><br> </div></div><a href="http://lists.sugarlabs.org/listinfo/iaep" rel="noreferrer" target="_blank">http://lists.sugarlabs.org/listinfo/iaep</a><br> </div></div></blockquote></div><br></div> <br>_______________________________________________<br> Systems mailing list<br> <a href="mailto:Systems@lists.sugarlabs.org" target="_blank">Systems@lists.sugarlabs.org</a><br> <a href="http://lists.sugarlabs.org/listinfo/systems" rel="noreferrer" target="_blank">http://lists.sugarlabs.org/listinfo/systems</a><br> <br></blockquote></div><br></div></div>