[Systems] [Systems-logs] [Gitorious Processor] fail in PushEventProcessor

[Bernie Innocenti]

On Tue, 2012-06-26 at 15:05 -0400, Chris Leonard wrote:
> On Tue, Jun 26, 2012 at 2:54 PM, Sascha Silbe <silbe at activitycentral.com> wrote:
> > Chris Leonard <cjlhomeaddress at gmail.com> writes:
> >
> >> On Tue, Jun 26, 2012 at 4:11 AM, Sascha Silbe <silbe at activitycentral.com> wrote:
> >>>
> >>> These exceptions are happening rather often and always for Pootle. Can
> >>> we do anything about that?
> >>>
> >>> Pootle commits may be more work for jita than most commits. Maybe just
> >>> increasing the timeout would help? Or is the timeout already so high
> >>> that hitting it is indicative of some deeper issue?
> >>
> >> I can tell you that I have been working Pootle pretty hard on a
> >> variety of tasks recently (Honey division, eToys POT refresh, etc.).
> > [...]
> >
> > These exceptions happen on the git server side (gitorious), not on the
> > Pootle side. No matter how hard you work Pootle, it shouldn't affect
> > gitorious. So don't worry. :)
> 
> OK, although I suppose it is possible I could cause an overload with
> one of the "Push many PO files" commits that have been done in the
> past, which would push a hundred or so PO file commits to the same
> directory in one swell foop.

Don't worry, this time it's entirely Gitorious' fault, not Pootle's!

>From the traceback we received, it seems evident to me that the differ
is doing something stupid:

> /srv/gitorious/app/vendor/grit/lib/grit/git-ruby.rb:152:in `try_run'
> 	/srv/gitorious/app/vendor/grit/lib/grit/git-ruby.rb:43:in `diff'
> 	/srv/gitorious/app/vendor/grit/lib/grit/commit.rb:163:in `diff'
> 	/srv/gitorious/app/vendor/grit/lib/grit/commit.rb:181:in `diffs'

Let's see what's at line 152 of app/vendor/grit/lib/grit/git-ruby.rb:

149       rescue Timeout::Error => e
150         bytes = @bytes_read
151         @bytes_read = 0
152         raise Grit::Git::GitTimeout.new(command, bytes)
153       end

look! We're hitting a timeout!

Yes, but how long is it?  From git.rb, it seems to be set to 10 seconds:

 23     self.git_binary   = "/usr/bin/env git"
 24     self.git_timeout  = 10
 25     self.git_max_size = 5242880 # 5.megabytes

I think 10 seconds is an entirely reasonable limit. If something takes
so long to diff, we should just abort and send an empty notification
email.

The problem here seems to be that Gitorious doesn't actually catch the
GitTimeout exception at the emailer level and aborts everything in the
push.

So my recommended solution would be patching Emailer.notify_emailer()
with something like this:

--- app/lib/emailer.rb	2011-08-26 11:39:47.000000000 -0400
+++ app/lib/emailer-bernie.rb	2012-06-26 21:36:33.420250993 -0400
@@ -47,5 +47,7 @@ EOF
       user = User.find_by_login('pootle-commits')
       Mailer.deliver_commit_notification(user, subject, body)
     end
+  rescue Grit::Git::GitTimeout => e
+    # Don't panic, log timeout with committer etc.
   end
 end

I don't know Ruby and I haven't tested this code. Maybe alsroot or rgs
could roll out the complete solution in production?

-- 
Bernie Innocenti
Sugar Labs Infrastructure Team
http://wiki.sugarlabs.org/go/Infrastructure_Team