[Systems] Trac maintenance

[Bernie Innocenti]

== Hostname change ==

Our trac instance is now reachable as bugs.sugarlabs.org.  The old
hostname dev.sugarlabs.org is not going away anytime soon, but
please update your links for consistency.

== Plugin updates ==

I refreshed all plugins with the latest versions and generally cleaned
up the installation as much as I could.  This is what we're presently
running:

 IniAdmin-0.2-py2.5.egg
 TracAuthOpenId-0.2.1dev-py2.5.egg
 TracCustomFieldAdmin-0.2.2-py2.5.egg
 TracHTTPAuth-1.1-py2.5.egg
 TracSpamFilter-0.2.1dev_r8330-py2.5.egg
 TracXMLRPC-1.0.6-py2.5.egg

If nobody is relying on it, I'd like to disable the XMLRPC and its
dependency HTTPAuth to simplify the configuration.


== Spam ==

I installed the TracSpamFilter plugin in the hope to reduce spammers who
have been riddling our database with bogus tickets.

A while ago I disabled wiki editing even for normal users, but the logs
are still full of failed attempts by spammers to create new spam pages:

  75.147.59.54 - - [15/Sep/2009:00:26:45 -0400] "GET /wiki/30_buy_vitamin_b_12?action=delete&version=2&old_version=1&delete_version=Delete+version+2 HTTP/1.1" 200 4322 "http://bugs.sugarlabs.org/wiki/30_buy_vitamin_b_12?action=diff&version=2" "Mozilla/5.0 (X11; U; Linux x86_64; es-AR; rv:1.9.1.3) Gecko/20090911 Fedora/3.5.3-1.fc12 Firefox/3.5.3"
  75.147.59.54 - - [15/Sep/2009:00:26:56 -0400] "POST /wiki/30_buy_vitamin_b_12 HTTP/1.1" 303 - "http://bugs.sugarlabs.org/wiki/30_buy_vitamin_b_12?action=delete&version=1" "Mozilla/5.0 (X11; U; Linux x86_64; es-AR; rv:1.9.1.3) Gecko/20090911 Fedora/3.5.3-1.fc12 Firefox/3.5.3"

At this point, these are just a big waste of cpu time.  I'm not sure
it's worth filtering the offending IPs.

== Database cleanup ==

We should come up with an SQLite recipe to get rid of all the pages they
created when the editing policy was more permissive:

  http://bugs.sugarlabs.org/wiki/TitleIndex

And there are dozens of bogus user accounts to kill.  The older ones are
easy to spot because they're all named similarily: alexqz3518,
burnessct4097, ethanvj5914...

The newest ones are subtler:

   AmuttCaralirm AmuttCaralirm almeidababich69674 at gmail.com
   BliZSaRdChiCK BliZSaRdChiCK user1196 at dorogen.in
   ...


== Performance ==

In an attempt to solve the longstanding performance issue with our trac
instance, I've rebuilt and installed the latest versions of all the
plugins we were using.

It didn't help much:

  bernie at solarsail:~$ time wget http://bugs.sugarlabs.org/ticket/17 2>/dev/null
  real	0m12.149s
  user	0m0.004s
  sys	0m0.008s
  bernie at solarsail:~$ time wget http://bugs.sugarlabs.org/ticket/17 2>/dev/null
  real	0m4.107s
  user	0m0.008s
  sys	0m0.008s
  bernie at solarsail:~$ time wget http://bugs.sugarlabs.org/ticket/17 2>/dev/null
  real	0m6.097s
  user	0m0.012s
  sys	0m0.004s

:-(

It seems that 4sec is the lowest time it takes to display an ordinary
ticket.  The slowest responses are most probably caused by concurrent
requests of spammers, search engines and regular users.  Solarsail has
32 processors, but our Trac setup with mod_wsgi is single threaded.

There's also a buildbot instance running on solarsail, but it's running
with nice 19 and disk I/O shouldn't affect trac too much.

If we can't figure out how to optimize Trac at least 500% by next week,
I propose moving it to a new VM on treehouse, which has "just" 8 cores,
but each one should be 10 times faster.

Another solution would be migrating our ticket datbaase to
launchpad.net.  From recent discussions with the Launchpad folks, it
seems we'll to wait for their next release cycle to get some features we
need to integrate LP's bug tracker nicely with the rest of our
infrastructure.  If we were really desperate, we could give up on some
of our requirements and rush a migration within next month.

(please, let's not discuss the LP migration in this thread.  Actually,
please let's not discuss it at all yet.  We're still talking with
Canonical and others to explore what long-term hosting possibilities
make sense for a project of our size.

-- 
   // Bernie Innocenti - http://codewiz.org/
 \X/  Sugar Labs       - http://sugarlabs.org/