[sugar] DS possibilities
Tue Apr 29 15:10:31 EDT 2008
On Wed, Apr 23, 2008 at 10:19 PM, Benjamin M. Schwartz
<bmschwar at fas.harvard.edu> wrote:
> Based on these options, I personally recommend that the priorities be:
> 1. Write a new datastore implementation to the same API.
> 1. Set up a datastore test system.
> 2. Improve logging.
> - --- Future ---
> 3. Write a new datastore implementation with a new API.
> 3. Notify the user when things go wrong.
> My recommendation is not very strong. I can see many other reasonable
> arguments. However, I prefer this one:
> After glancing at Michael's Simple Data Store code, I am convinced that it
> would be easy to turn it into a complete implementation of the current
> datastore API, with a backend that is simply individual files in the
Where would metadata be stored?
> Search could be implemented by grep -R, or something of
> similar complexity.
Hmm, I see filtering and fulltext search as really important features.
Do we really want to grep instead of using some kind of index?
> I would be perfectly happy to replace the current
> datastore layer with this one (though the upgrade mechanism will take some
> careful thought). If some code from the existing implementation is still
> applicable, then it may of course be reused. Indeed, the rewrite vs. fix
> distinction is something of a false dichotomy; the major issue is whether
> or not to use Xapian and other advanced algorithmic designs.
I think that most of our problems come from using Xapian for storing
the metadata, but not from using it as what it is intended: a fulltext
search engine. I actually think that it has worked quite well for that
> The principle objection to this path, it seems to me, is the possibility
> of introducing new (worse) bugs. A datastore test system would greatly
> increase confidence in any new implementation. If the test system can
> also reproduce crashes in the current implementation, then we may
> determine with some confidence whether the new implementation is more
> reliable (under those circumstances). Logging goes hand-in-hand with
> testing; it is needed in order to determine the results. Having better
> logging in production laptops will be a positive side benefit.
Agreed, but I would like to note that the DS is (to my knowledge) the
component in Sugar with most extensive unit tests.
> I recognize the importance of providing version control functionality
> within the datastore, but the deadline for this work appears to be July,
> pending an August release. It seems unlikely to me that any
> future-proofed version control system could be completed, and integrated
> with the rest of the system, in two months. If anyone wishes to argue the
> opposite point, I will be happy to listen, since I desire this feature
> greatly. If the version control project is sufficiently important, it may
> be acceptable to place an expert (presumably Scott) on this full-time
> targeting a December release.
I think that it depends on which features we wish to add. This scheme
may be enough to provide the user experience that Eben has devised:
- Store a version number as a metadata property. Create a branch every
time that a non-tip revision is resumed. Suggested format: 3.1.1.
- Use xdelta to store backwards deltas between versions of the same
file. Probably calculate a hash of the content so we can optimize out
the no-changes case.
- That's all ;)
See here for the last work from Ben Saller related to versions:
> Notifying the user when critical system components stop working is a good
> goal as long as it doesn't distract from the more important work of making
> the critical system components work reliably. It seems to me that the UI
> team's work on the wide variety of frequent, user-interaction
> notifications is far more important, and also sets the stage for this work
Yes, the "journal full" notification is one example.
More information about the Sugar-devel