[Bugs] #1566 sugar-datastore UNSP: Fix metadata corruption (was: Call fsync() for metadata writes)
Sugar Labs Bugs
bugtracker-noreply at sugarlabs.org
Fri Sep 21 14:23:52 EDT 2012
#1566: Fix metadata corruption
------------------------------------------+---------------------------------
Reporter: sayamindu | Owner: martin.langhoff
Type: defect | Status: assigned
Priority: Unspecified by Maintainer | Milestone: Unspecified by Release Team
Component: sugar-datastore | Version: Unspecified
Severity: Unspecified | Keywords: r!
Distribution: Unspecified | Status_field: Unconfirmed
------------------------------------------+---------------------------------
Changes (by martin.langhoff):
* cc: erikos, sascha_silbe (added)
* owner: alsroot => martin.langhoff
* status: new => assigned
Old description:
> While the ds index is force flushed every 60 seconds, the metadata files
> can often be in a non-flushed state for an undefined time interval. A
> force power off in such a situation makes the Journal bring up odd
> looking entries.
New description:
(Originally titled: Call fsync() for metadata writes)
While the ds index is force flushed every 60 seconds, the metadata files
can often be in a non-flushed state for an undefined time interval. A
force power off in such a situation makes the Journal bring up odd looking
entries.
--
Comment:
Re-titling to reflect the problem instead of a diagnosis that was likely
premature/faulty.
After working on this codebase on the patchseries addressing #2317 and
related dataloss/datacorruption issues with the datastore, my tendency is
to NAK the proposed patch and the whole logic behind it.
The original report talks about "the Journal bringing up odd-looking
entries", I take it to mean that we are working on corrupt/mismatched
metadata between index and metadatastore.
A likely root cause for this is the implementation of propery storage in
metadatastore.py ; a patch addressing this is at
http://lists.sugarlabs.org/archive/sugar-devel/2012-September/039729.html
, and commentary on the erroneous use of try/except/finally at
http://lists.sugarlabs.org/archive/sugar-devel/2012-September/039746.html
. The metadatastore implementation is rather prone to data corruption /
loss.
Once the writes to metadatastore are done defensively (atomically, in the
patch linked), it is valid to ask whether it is worth calling fsync() to
keep in rough sync with the index. The answer is no: all the filesystems
used in Linux today have fsync() on a per-filesystem basis. All "dirty"
buffers for the relevant filesystem are written out, not just the
filehandle you passed.
So if the metadata changes are pending to be flushed when the index is
flush()ed, the fsync() that Xapian invokes will _also_ flush metadata
changes to disk.
I'll also note that currently we flush the Xapian db to disk every 5s,
much lower than the 60s in use when originally reported.
--
Ticket URL: <http://bugs.sugarlabs.org/ticket/1566#comment:4>
Sugar Labs <http://sugarlabs.org/>
Sugar Labs bug tracking system
More information about the Bugs
mailing list