[Sugar-devel] Datastore index corruption!

[Bernie Innocenti]

On Mon, 2010-08-02 at 16:04 -0400, Martin Langhoff wrote:
> There are much smarter ways to tackle this. Make the index rebuild
> automatic on Sugar startup, with some kind of spinner indicating we're
> working on something, conditional on
> 
>  - look at the newest file in ~/.sugar/default/datastore, compare
> mtime with the index files
> 
>  - read & parse /usr/bin/last output -- a boot without a prior
> shutdown or reboot entry indicates a hard shutdown

These are indeed very good ideas. I'll see if we can get them
implemented in time.

I think we still need a rebuild index function to cope with a few other
cases of corrupted indexes that I've observed in the field. If you're
interested in analyzing datastore corruption, I could share some
tarballs from my private collection:


bernie at giskard:~/src/olpc/datastore-horror-show$ for f in *.txt; do echo -e "\n== $f =="; cat $f ; done

== datastore-empty-downloads.txt ==
This datastore contains two downloads hung at 0% and a couple
more files that can't be deleted.

If you try to reindex, you get two more 0% downloads :-)

There are also a lot more entries in the checksums directory
than files. What are these used for? Is there a mechanism for
garbage collecting checksums when they get out of sync with
the entries?


== datastore-invisible-entries.txt ==
This datastore contains a bunch of entries that just do not show in the
Journal.

Removing the index_updated file is sufficient to make them show up
again. It's not clear how the index could get out of sync in this
way. Maybe a crash or a filesystem full condition. I would recommend
analyzing the index db for consistency issues.

Suggested resolution: if we can't get the datastore to autodetect
and repair these issues, we should at least provide a cheat-code
to recreate the index. For example, we could use "both shifts" on
startup in the style of the old Mac System 7.  This could be done
within Sugar or in a startup script such as olpc-configure.
I would favor a Sugar solution so it would work with SoaS too.


== datastore-no-update.txt ==
This is a journal that won't update to the new format version 5
introduced by Andres' sort-by patch set.

Not necessarily a corruption issue.

== datastore-orphans.txt ==
This datastore contains a bunch of files that arn't visibile
in the Journal.

If we regenerate the index, the files reappear. But since the
user has no way to cause an index rebuild from the GUI, they
are left with the only option of reflashing to recover the
lost space.

== datastore-patto.txt ==
This is a journal created by a formador with Sugar 0.82.

It is not known to have corruption issues (but I wouldn't be surprised
it it did have some minor consisntency issues).  It's useful as a sample
dataset to test the journal format upgrade procedure.

== datastore-paz.txt ==
This is a journal created by a formador with Sugar 0.82.

It is not known to have corruption issues (but I wouldn't be surprised
it it did have some minor consistency issues).  It's useful as a sample
dataset to test the journal format upgrade procedure.

== datastore-phantom.txt ==
This is the opposite bug of a datastore with invisible entries:
there are more visible entries than there are files!

I found this on a machine that was 99% full due to Wine, browser
caches and Gnome downloads. So my guess is that the datastore
failed to create the inodes, but happily continued to update the
index.

Whatever the reason for this corruption was, the datastore seems
to choke on failed filesystem operations and therefore does not
let you delete any of these phantom entries.

Regenerating the index fixes the issue.

-- 
   // Bernie Innocenti - http://codewiz.org/
 \X/  Sugar Labs       - http://sugarlabs.org/