[Systems] Migration to SN testing node urgent

[Sebastian Silva]

Hola Aleksey,
Bernie and I were poking at jita + apache + sugar network node last 
night.
After analyzing the issue, Bernie concludes it's actually the Sugar 
Network Node leaking file descriptors.
I'll try to setup a test script and setup monitoring this weekend.
Hopefully with your help we can find this leak too!
Regards,
--
Sebastian Silva
"icarito" #sugar #somosazucar (freenode IRC)
Somos Azúcar - Fuente Libre - Sugar Labs

"Las maestras y los maestros democráticos intervenimos en el mundo a 
través del cultivo de la curiosidad" - P.Freire

El mié, 15 de oct 2014 a las 11:26 PM, Aleksey Lim 
<alsroot at sugarlabs.org> escribió:
> On Wed, Oct 15, 2014 at 11:34:01AM -0500, Sebastian Silva wrote:
>>  Alsroot,
>>  Greetings,
>>  We're observing downtime about twice a day now in production 
>> instance
>>  of Sugar Network central node.
>> 
>>  Every time I have to log into jita and issue:
>> 
>>   sudo /etc/init.d/sugar-network stop
>>   ps -o pid,comm,user,thcount -u www-data | wc -l
>>   # ^^ is useful to give an idea of traffic
>>   # goes down to ~12 when all SN threads die after a few seconds
>>   sudo /etc/init.d/sugar-network start
>> 
>>  It's probably all the traffic, but also it seems to have gotten 
>> worse
>>  after the downtime jita had some time ago.
>> 
>>  It's stressful for editors/admins and annoying for users.
>> 
>>  As I understand it, new node implementation does not have this 
>> problem.
>> 
>>  I can do the migration myself, if you provide me with some details:
>> 
>>  * procedure for migrating database
>>  * current release tarballs/sources for putting in production
>> 
>>  I think it's even better if I do it, then I will have a better 
>> sense of
>>  how the clockwork ticks. I'll attempt to document as I go. Maybe we 
>> can
>>  setup some uptime monitoring this time around (cc: systems@ for this
>>  purpose).
>> 
>>  It would be helpful to coincide when you are online on this task. 
>> For
>>  me a good time would be starting Friday 17th at 21:00 (UTC -5 / 
>> Bogota)
>>  - but I'm open to accomodate to your timezone/schedule/convenience.
>>  This way we can test over the weekend and have a working service by
>>  Monday 20th.
>> 
>>  Let me know so we can announce the planned maintenance downtime.
>> 
>>  We've gotten this far and have engaged some active users. I think 
>> there
>>  is a bright future for Sugar Network. We just need to keep rowing.
>>  Thanks for your commitment.
> 
> The issue is not w/ SN node in particular but with Apache connections
> pool, next time restart Apache. Last time we decided to not migrate
> node.sl.o to intermediate code base release. So, if the only issue is
> unavailable node, it could be started on separate IP out of Apache.
> 
> If I got it right, it is possible to grant jita new external IP,
> so, we need to ask Bernie. Then,
> 
> 1.  node.sugarlabs.org DNS should be re-pointed to the new IP
> 
> 2.  /srv/sugar-network/.config/sugar-network/config should be tuned:
> 
>         [node]
>         host = <NEWIP>
>         port = 80
> 
> 3.  /etc/apache2/sites-enabled/node.sugarlabs.org should be tuned:
> 
>     <VirtualHost *:80>
>             ServerName node.sugarlabs.org
>             ProxyPass /  http://<NEWIP>:80/
>             ProxyPassReverse /  http://<NEWIP>:80/
>     </VirtualHost>
> 
> 4.  sugar-network-node restarted and Apache reloaded.
> 
> --
> Aleksey
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sugarlabs.org/private/systems/attachments/20141018/3d8e080d/attachment.html>