[Sugar-devel] Datastore index corruption

[Bernie Innocenti]

We've found an XO-1 running Fedora 11 + Sugar 0.84 with an interesting
datastore corruption issue.

The journal was showing just one object, but the
~/.sugar/default/datastore directory contained 4-5 invisible entries.
After removing index_updated and restarting Sugar, the entries
reappeared.

There was no time to analyze the problem in detail, but I have strong
feeling that this isn't an isolated case. It would explain other reports
of files disappearing and mysteriously reduced disk space.

Some reflections:

 * the corruption could be caused by flash problems. I have found
   laptops in the field that wouldn't boot because /sbin/lvm was
   corrupted

 * we can't exclude jffs2 problems too: when it's almost full, it does
   slow garbage collection passes on boot which kids interrupt by
   power cycling. I wonder how robust jffs2 is in this case.

 * there might be a bug in xapian. If so, we'll see this issue also
   on the XO-1.5

 * I'm skeptical it's a new issue in 0.84 or F-11: the older builds
   had so many data loss issues that a subtler problem like this
   could have easily gone unnoticed.

 * can the datastore detect index corruption in the most obvious cases?
   If so, what would it do?

 * how long does it take to rebuild the index on a busy journal?
   Can we afford to rebuild from time to time? On every boot?

 * finally, if we can't find a 100% robust solution, would it make
   sense to add a "Reindex Journal" button somewhere? Where would
   you put it?

-- 
   // Bernie Innocenti - http://codewiz.org/
 \X/  Sugar Labs       - http://sugarlabs.org/