[sugar] bundles and parts (was Re: [PATCH] Journal able to use "open with" for activity bundles)

[Jameson "Chema" Quinn]

The "objects and translators" model, as described, does not have any easy
way to retar a bundle; nor is it easy to figure out how to assign different
mime types to different "kinds" of tar files (this one is a web page, that
one is an activity bundle). Note that the "rebundle" problem is not so
trivial - what do you do with the metadata on the subfiles?

Let me try to sketch out the problem of bundles, with everything a good
solution should enable.

1. In journal, the "most default" view shows only bundles, is not cluttered
with all subfiles.
2. Bundles have their own metadata, and a mime-type that reflects their
holistic nature (a set of photos, a web page, an activity)
3. Subfiles inherit (some?) bundle metadata, but can also have their own
metadata.
4. Some subfiles, which make no sense on their own, are hidden by default
(dotted files)
5. Possible to grab a piece of a bundle and start treating it as a separate
top-level object.
6. Possible to edit a piece of a bundle as if it were a top-level object,
yet keep it as part of the bundle. This creates a new version of the whole
bundle, with the edited file.
7. Possible to take a top-level object and add it to a bundle.
8. Possible to remove (delete) an object from a bundle.
9. Possible to import/ export bundles as single files (zip/tar) to external
media/over the net
10. Possible to import/export bundles as directories to external media.
11. For easier porting of legacy apps: ability to open a subfile in context,
with the directory structure of the bundle extracted, so that relative paths
work.

...

I think it is clear from the above list that moving back and forth from
bundle to file view should be as transparent as possible. Internal storage
could be as separate files or as zipped/tarred groups; either way, there
would need to be a storage of stubs/pointers to stand in for the other
model.

Here's one proposal to kick things off:
External format:
metadata for a traditional file is represented on external media by an
invisible file in the same directory, with a related name, which contains
json.
bundles are represented on external media by a zip, which politely extracts
itself to a subdirectory.
metadata for bundles is represented by an invisible file next to the zip.
This file is a zip which extracts to the same subdirectory, putting global
metadata in an invisible file in the root, and subfile metadata in dotfiles
as above.

Internal format:
Same as external format. Glucose knows how to unpack and repack the bundles.
There is no pretense that Glucose maintains any integrity/signatures for the
bundles. When editing as in option 6 above, the bundle is extracted (see
option 11 above) and then repacked on *each save* of the subfile. Each
non-hidden subfile is also represented by a metadata-only datastore pointer
into the bundle, for journal searches.

Note: this proposal is less than ideal in some ways. For one instance: small
changes to large files inside a zip are not good for delta-based storage.
For another, zip has no way to include empty directories, which might be
important for directory structure in some legacy cases. It assumes
relatively large atomic saves in any subfile.

Jameson

On Wed, Jun 11, 2008 at 9:37 AM, Benjamin M. Schwartz <
bmschwar at fas.harvard.edu> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I have a proposal for how to manage objects in the datastore, to resolve
> the various issues we have discussed.  I call it the "Objects and
> Translators" model.  The fundamental idea is this: the datastore is a
> flat, non-hierarchical listing of objects (versioned, but that does not
> come into play here).  The datastore provides these objects for use by
> Activities.  However, the datastore (or perhaps the Journal) also contains
> a new element: Translators.
>
> A Translator is a non-interactive object processing machine that takes an
> object as input and returns a set of objects as output.  Each Translator
> has a specified set of input types on which it can act.  The user, having
> selected a particular object, should be provided with a list of all
> compatible Translators.  Clicking on a Translator should show the list of
> output objects that Translator would generate for this input object, and
> the user may repeat this process on each of those objects as well.
>
> This model allows us to reproduce the usual sorts of directory
> hierarchies, simply by using a group object (I think of it as a .tar file)
> and an untar Translator.  To the user, this looks very much like a
> directory hierarchy.  However, because each "directory" is in fact a
> first-class object, it can also be associated with Activity instances, or
> passed from one XO to another over the network, etc.
>
> The Translator system is appealing to me because it can provide far more
> than a simple storage hierarchy.  For example, I have often had to extract
> a particular image from a PDF file, in order to include it in a class
> project.  I, and most people, have done this by blowing up the image and
> performing "Print Screen", or perhaps opening the PDF in the GIMP, which
> rasterizes it.  However, there is a much better solution, provided by
> xpdf's command-line utils, which allows one to split a PDF into its
> component images and text, losslessly.
>
> Almost nobody uses these tools, because they are command-line only, and
> obscure.  It would be trivial to wrap them up into a Translator, though,
> and then every PDF, in addition to showing options for viewing in Read,
> would show an option for splitting into it components, for further
> editing.  Nothing could be more in keeping with our goal of creating an
> editable world.
>
> You can easily think of many other interesting Translators.  For example,
> one could also provide a PDF->PNG rasterizer, a ODT->PDF renderer, or a
> Theora->JPG frames decompressor.  The APIs should be designed so that
> Translators can be lazy, computing the contents of output objects only
> when necessary.
>
> Translators provide tremendously powerful object management, with a simple
> path to implementation.
>
> - --Ben
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAkhP8SkACgkQUJT6e6HFtqRQrACfcd5ntZ1ZXovZtTVI5k4LEG96
> oDwAnjswhuiV1uKJ5cWhVV9N6uFNgT6E
> =x7ie
> -----END PGP SIGNATURE-----
> _______________________________________________
> Sugar mailing list
> Sugar at lists.laptop.org
> http://lists.laptop.org/listinfo/sugar
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.laptop.org/pipermail/sugar/attachments/20080611/f33af1bf/attachment.htm