[Systems] [Systems-logs] [Gitorious Processor] fail in PushEventProcessor

[Aleksey Lim]

On Tue, Jun 26, 2012 at 09:39:04PM -0400, Bernie Innocenti wrote:
> On Tue, 2012-06-26 at 15:05 -0400, Chris Leonard wrote:
> > On Tue, Jun 26, 2012 at 2:54 PM, Sascha Silbe <silbe at activitycentral.com> wrote:
> > > Chris Leonard <cjlhomeaddress at gmail.com> writes:
> > >
> > >> On Tue, Jun 26, 2012 at 4:11 AM, Sascha Silbe <silbe at activitycentral.com> wrote:
> > >>>
> > >>> These exceptions are happening rather often and always for Pootle. Can
> > >>> we do anything about that?
> > >>>
> > >>> Pootle commits may be more work for jita than most commits. Maybe just
> > >>> increasing the timeout would help? Or is the timeout already so high
> > >>> that hitting it is indicative of some deeper issue?
> > >>
> > >> I can tell you that I have been working Pootle pretty hard on a
> > >> variety of tasks recently (Honey division, eToys POT refresh, etc.).
> > > [...]
> > >
> > > These exceptions happen on the git server side (gitorious), not on the
> > > Pootle side. No matter how hard you work Pootle, it shouldn't affect
> > > gitorious. So don't worry. :)
> > 
> > OK, although I suppose it is possible I could cause an overload with
> > one of the "Push many PO files" commits that have been done in the
> > past, which would push a hundred or so PO file commits to the same
> > directory in one swell foop.
> 
> Don't worry, this time it's entirely Gitorious' fault, not Pootle's!
> 
> From the traceback we received, it seems evident to me that the differ
> is doing something stupid:
> 
> > /srv/gitorious/app/vendor/grit/lib/grit/git-ruby.rb:152:in `try_run'
> > 	/srv/gitorious/app/vendor/grit/lib/grit/git-ruby.rb:43:in `diff'
> > 	/srv/gitorious/app/vendor/grit/lib/grit/commit.rb:163:in `diff'
> > 	/srv/gitorious/app/vendor/grit/lib/grit/commit.rb:181:in `diffs'
> 
> Let's see what's at line 152 of app/vendor/grit/lib/grit/git-ruby.rb:
> 
> 149       rescue Timeout::Error => e
> 150         bytes = @bytes_read
> 151         @bytes_read = 0
> 152         raise Grit::Git::GitTimeout.new(command, bytes)
> 153       end
> 
> look! We're hitting a timeout!
> 
> Yes, but how long is it?  From git.rb, it seems to be set to 10 seconds:
> 
>  23     self.git_binary   = "/usr/bin/env git"
>  24     self.git_timeout  = 10
>  25     self.git_max_size = 5242880 # 5.megabytes
> 
> I think 10 seconds is an entirely reasonable limit. If something takes
> so long to diff, we should just abort and send an empty notification
> email.
> 
> The problem here seems to be that Gitorious doesn't actually catch the
> GitTimeout exception at the emailer level and aborts everything in the
> push.
> 
> So my recommended solution would be patching Emailer.notify_emailer()
> with something like this:
> 
> --- app/lib/emailer.rb	2011-08-26 11:39:47.000000000 -0400
> +++ app/lib/emailer-bernie.rb	2012-06-26 21:36:33.420250993 -0400
> @@ -47,5 +47,7 @@ EOF
>        user = User.find_by_login('pootle-commits')
>        Mailer.deliver_commit_notification(user, subject, body)
>      end
> +  rescue Grit::Git::GitTimeout => e
> +    # Don't panic, log timeout with committer etc.
>    end
>  end
> 
> I don't know Ruby and I haven't tested this code. Maybe alsroot or rgs
> could roll out the complete solution in production?

Fixed,
https://gitorious.org/~alsroot/gitorious/sugarlabs-org/commit/bdd0d88c6f2747f80a572136ba131f6d66ca370e

-- 
Aleksey