[Sugar-devel] Debugging sugar-datastore (and similar dbus-connected services)

[Martin Langhoff]

On Wed, Sep 19, 2012 at 4:54 PM, Gonzalo Odiard <gonzalo at laptop.org> wrote:
> You already have the line
> export SUGAR_LOGGER_LEVEL=debug
> uncommented in .sugar/debug, right?

Yes. And with that, datastore.log is very chatty, which is good.

But when the datastore code hits an unhandled exception, the exception
does not get logged. DBus restarts the process transparently, but that
isn't so useful :-/

In terms of progress, I have these tracks underway,

 - Add a "ds_clean" flag file, rm it on the start of every
create/update/delete op, create it on completion. Complements
"index_updated", and between the two we get good coverage. Missing
either one, we get an index rebuild. This is done, and working well.

 - During index rebuild, delete incomplete entries. Not working well yet.

With the two above, we can operate with the system at ENOSPC, and
essentially we don't lose existing user data. New files fail to save,
and metadata changes fail to get stored (so your edit, ie: a rename,
is reverted in the journal, visibly). But we don't lose the existing
Journal entries -- so behaviour is _almost_ civilized now :-)

Behind the scenes, however, the datastore is dying and respawning.
Which complicates things because what we are missing is to be able to
delete entries while we are at ENOSPC. A delete is pretty
straightforward, because we are removing files, and that succeeds even
on ENOSPC. Except that we need to be able to maintain the xapian
indexes.

My plan right now is to catch the Xapian-is-hitting-ENOSPC exception,
and trigger a Xapian index re-creation, putting the Xapian files in
/tmp (which is a tmpfs in our builds and in Fedora upstream too).
We'll make sure that the index_updated file on disk is removed so
after a reboot the index is recreated once more.

With Xapian index on /tmp, deleting Journal entries should just work.
All dbus sockets are on tmpfs.

However, to do all of the above with some sense that it's working
correctly... yeah, I need to see the exceptions I hit. Right now I am
guessing "what could be going wrong in this black box...?" which isn't
a productive mode of life.



m
-- 
 martin.langhoff at gmail.com
 martin at laptop.org -- Software Architect - OLPC
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff