[Sugar-devel] datastore situation (was Re: Hypothetical sugar-0.90 material, draft 1.)

[Michael Stone]

Dear Tomeu and other Gentle Readers,

Thanks for bringing this thread to my direct attention by CC'ing me. 

Since, I'm not exactly sure what feedback you'd like from me, I've tried to
respond in a way that will lead to a fun and productive discussion on 

   "where do we want to go over the next few years and, concretely, what might
    we do to get there?"

I hope this is helpful. Therefore, if it's not, please let me know and we can
try something else.

Finally, please note that, in the interests of perspicacity and getting at
least a few hours of sleep tonight, I've left out the background of /why/ I
think we want the things that I claim we want below. If you'd like me to fill
this in, please speak up.

Regards,

Michael

...........................

On Mon, Jun 21, 2010 at 12:06:49PM +0200, Tomeu Vizoso wrote:
>On Thu, Jun 10, 2010 at 17:48, Martin Langhoff
><martin.langhoff at gmail.com> wrote:
>>
>
>I think this is an unfair recount of my work, so I'm going to put some
>light on it and ask that my frankness is excused.

>>  - Reworking the datastore... while I welcome efforts in a new
>> datastore... _every Sugar release has a new DS implementation_
>
>Not sure how you are counting, only 2 datastore implementations have
>ever been released as part of a Sugar release (more details below).

Agreed.

>> they get little testing and I've seen extremely light thinking about
>> what is _actually_ needed.
>
> ...
>
> What should I have done instead, limit myself to publish papers and give
> talks about my prototype and just re-release the old implementation in every
> Sugar release?

You did good work in your rewrite.

>> We need _a good, polished DS that covers
>> many aspects sanely_... a new DS is unlikely to do so. IOWs the
>> barrier to merge a new DS should be high. It just triggers my CADT
>> alarms http://www.jwz.org/doc/cadt.html

Martin: CADT doesn't accurately describe the DS situation so please either
recalibrate your alarms or be more specific about what "immortal bugs" concern
you.

>Not sure how we can agree on rewriting or not rewriting if we haven't
>agreed yet on what features should have the future releases of the DS?

So let's talk about what we might want to see in later 0.9x releases...

For me, the three biggies concern the data model, the compatibility story, and
the safety story. Here are my thoughts on each:

On the data model:

   model v0.8x: 

      in the grammar of sugar-0.8x, the journal records one-word sentences in
      which instances simultaneously represent both nouns and verbs.

   claim:

      in the short term, we want to get to a Journal that looks like Eben's
      slideshow and that works like Scott's journal2 mockup.

   model v0.9x:

      in the grammar of sugar-0.9x, the journal records multiword sentences in
      which "instances" are verbs and "objects" are nouns.

   Walter's paraphrased claim:

      in the long run, we're going to want a richer data model that includes
      grammatical features like adverbs and adjectives. however, it seems likely
      that we're not going to be able to get a richer model right before first
      gaining some experience with the simpler model described above.

On the compatibility story:

   we need to maintain compatibility with

      a) the current D-Bus API and
      b) the ability to loselessly import older DS data.

   however, to grow our universe of apps, we need to create compatibility with
   existing software based on hierarchically structured files and directories.

   to do merge this with the v0.9x data model, we should create a new 0.9x
   activity API which specifies the interpretation of top-level files,
   directories, and file metadata in the activity root. 

   as a simple, hypothetical, example:

     activity sessions are identified by "session" dirs with extension ".xos".

     activity sessions are resumed by executing ./resume in the session dir.

     bespoke resume-related data are stored in hidden files or subdirs in the
     session dir.

     the reusable objects of an activity session are non-hidden files in
     standard formats in the session dir or its subdirs.

     activity metadata is stored in ./metadata.json in the session dir.

       (alternate design: metadata is stored as xattrs on whatever files or dirs
       it applies to)

     limited versioning is enabled when we find a top-level hidden dir with
     supported vcs state. for 0.9x, this would mean basic support for .git dirs.

On the safety story:

   * the overriding themes of DS safety are undoability and limited isolation:

     a) session-level undo (versioning)
     b) system-level undo (backup/restore)
     c) isolation (sandboxing + ACLs)
  
   the session-level undo I'm thinking of is adequately provided by the current
   keep button and an additional toolbar that, when opened, displays an
   MRU-sorted list of previous commits. (The keep button should record a commit
   message, then call something like "git add *; printf <msg> | git commit -F -"

   * I don't yet have detailed thoughts on the system-level undo functionality
   other than to point out that there's lots of prior work on backup/restore for
   Plain Old Directories Of Files.

   * the basic story on data isolation is that activities should run under fresh
   accounts and that the account from which Sugar was launched should be
   empowered to modify ACLs on each session dir's tree of files in order to make
   files simultaneously available to multiple sessions.

Thoughts on this outline?

P.S. - I'm also interested in (but too tired to explain) several topics of
considerable secondary interest including the publishing, assimilation, and
indexing stories. Ask me later for details.