[Sugar-devel] [PATCH sugar-datastore] Ensure we return valid internal / calculated properties

[Sascha Silbe]

Excerpts from James Cameron's message of 2011-11-02 23:42:59 +0100:

> I'm glad to see fixes for corrupted metadata, but I wonder if the
> application (the datastore process) is properly asking the kernel to
> retain the data?  I wouldn't like to see it block on a sync, but I
> thought there were ways to ask for the data to be written soon.

We're not using fsync() or sync() because we expect serious performance
regressions if we do (since the entire data store would block on the
fsync() call, not just a single API call). The current data store layout
splits everything over lots of different files which we'd have to
fsync() individually.

If you know of a better way to improve reliability, I'm all ears.
Neither madvise() nor posix_fadvise() do what we need. sync_file_range()
isn't applicable to newly-created files.

FWIW, if fsync() were fast enough, the algorithm I'd use with the
current data store layout would be:

1. Clear "index updated" flag.
2. Create a temporary directory for the new entry, keeping the file
   descriptor open.
3. Write all metadata, keeping the file descriptors open.
4. Write the data, keeping the file descriptor open.
5. fdatasync() and close all metadata files.
6. fdatasync() and close the data file.
7. fsync() and close the temporary directory.
8. rename() the new entry (temporary directory) into place.
9. Update the Xapian index.
10. Set "index updated" flag.
11. Signal completion to caller.

The intention of the interleaved FIFO order for the writes and fsync()s
would be to let the kernel write as much data as possible to the disk in
the background and only wait for completion when there's nothing else
for us to do.

With the above sequence (and assuming no data is lost inside the storage
device, e.g. due to integrated write caches), the data store would
provide at least three of the four ACID properties: Atomicity (final
rename()), Consistency (fsync() + final rename()), Durability
(signalling completion after everything has reached permanent storage).
It might even be considered having the Isolation property, but I'd have
to think some more about that.

Sascha

-- 
http://sascha.silbe.org/
http://www.infra-silbe.de/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
URL: <http://lists.sugarlabs.org/archive/sugar-devel/attachments/20111106/aa2557dc/attachment.pgp>