[Sugar-devel] Sugar-Server enhancement

[Jerry Vonau]

> On April 15, 2016 at 12:36 AM James Cameron <quozl at laptop.org> wrote:
> 
> 
> On Fri, Apr 15, 2016 at 10:25:59AM +0800, Tony Anderson wrote:
> > Hi Manash
> > 
> > The registration process is awkward but not the problem.
> 
> This is unfair scope creep.  Manash began by asking about bug #362 and
> has been working to fix that.  Now you're asking him to consider a
> much larger task; not a coding task, but a redesign of Sugar Journal
> and Backup interaction.  This is huge.
> 
> And as far as I can tell, students aren't even accepted yet [1].
> 
> 1.  https://en.wikipedia.org/wiki/Google_Summer_of_Code#2016
> 
> What you propose is from a set of tasks [2] you added to the Wiki,
> which have not undergone any design review according to Sugar Labs
> design practice and feature policy.  I do not see any consensus on
> these; we're yet to build a consensus.
> 
> 2.  https://wiki.sugarlabs.org/go/Summer_of_Code/2016#Sugar_on_the_Ground
> 
> Or, it looks like you're trying to make your own fork of Sugar, which
> I'm fine with, it's open source after all, but to push that on others
> without their input is wrong.
> 
> If you proceed without consensus as a sole designer, then OLPC will
> fork Sugar (as we already have so that XO-1s will go faster), and
> you'll be making your own builds.
> 
> > The problem is that rsync is used to create backups of the Journal
> > and no effective means is offered to restore.
> 
> Agreed.  We have no restore from server feature in Sugar 0.108, along
> with no way to start a backup to server, and no selective restore.
>
> (We have backup to media, restore from media, but no selective
> restore from media.  Also, restore from media replaces Journal!)
> 

Think the low hanging fruit is backup to server with a non-selective
restore, mimicking what the media based backend does as a starting point.
Selective restore should be added to both methods once a common storage
format is worked out.

> > However, the ultimate problem is thinking of the problem as one of
> > backup. If you try to solve the wrong problem, often the result is a
> > wasted effort.
> > 
> > The Journal is single place where Sugar users save their documents.
> > This is done by the Sugar activities when they close. The majority
> > of XOs are still XO-1s with a 1GB store.
> 

Or something like a remote network datastore if I understand what Tony is
in favor of.

> This point in your argument is void, because XO-1 are 45% of the XO
> laptops manufactured so far.  I have the numbers.
> 
> Also, many XO-1 have been upgraded with an SD card.
> 
> > If the available store is less than 50GB,
> 
> No, that's 50 MB, not 50 GB.  See _SPACE_TRESHOLD (sic) in
> sugar:src/jarabe/journal/journalactivity.py [3].
> 
> 3.
>  https://github.com/sugarlabs/sugar/blob/master/src/jarabe/journal/journalactivity.py#L56
> 
> > Sugar effectively shuts down.
> 
> This point in your argument is void, because this has been fixed [4,
> 5, 6], please upgrade to Sugar 0.108 which is in OLPC OS 13.2.7 [6].
> 
> 4.  http://dev.laptop.org/ticket/9623
> 5.  https://bugs.sugarlabs.org/ticket/1720
> 6.  http://wiki.laptop.org/go/Release_notes/13.2.7#Fixes
> 
> > This typically results in the deployment reflashing the XO erasing
> > all of the documents created by that user - a tragedy.
> 
> It was a known bug, so that's a training issue.  You previously
> proposed to train a teacher to use "rm -rf" to delete a known_hosts
> file instead of Manash coding up an "ssh-keygen -R" command.  It is
> inconsistent to be able to do one and not the other.
> 
> > What I am proposing is to use the school server as the primary store
> > for the Journal with its effectively unlimited storage capacity. The
> > ds_backup script needs to read the datastore uploading any new or
> > modified documents. The local datastore can then be viewed as a
> > cache for current working documents.
> 
> I'm favour of this ideal in principle, but it remains a huge design
> and consensus challenge, not a coding challenge.
> 
> However, with the XO-1, XO-1.5 and XO-1.75 using IEEE 802.11g the
> local wireless network will collapse sooner due to this new load.
> 
> > On the XO, the datastore is shown in the Journal. The 'keep' star
> 
> There's no such thing.  There's a favorite star [7].  It has a defined
> purpose.  Are you proposing to destroy that purpose, or add another
> column to the journal?  There's even less room now that the multiple
> selection checkbox was added.
> 

Think the "keep star" is something out of dextrose DX3
http://lists.sugarlabs.org/archive/dextrose/2011-June/001287.html
with later DX4 using a tick box.

> 7.  https://help.sugarlabs.org/en/journal.html#journal-features
> 
> > could be used to show whether there is a local copy of that document
> > or not. If the document is not needed locally, the user can clear
> > the star. In this case, the backup script could delete the local
> > copy. If there is no local copy of the document, then the user could
> > set the star. In this case the backup script could download the
> > document.
> 
> My preference would be for the flag to be in the Journal detail
> view [8], where there is available display space.
> 
> 8.  https://help.sugarlabs.org/en/journal.html#journal-detail-view
>

from DX4 
http://wiki.sugarlabs.org/go/File:Journal_multiselect_lock_prevent_interacting_with_error.png
 
> > This capability could be used to set a quota on the amount of space
> > used by the Journal. If the space is exceeded, the 'backup' script
> > could delete local copies of document by LRU until the quota is met.
> > Similarly, there should be a quota on Sugar activities which could
> > also automatically be pruned back LRU. Managing the store
> > automatically is consistent with keeping the Sugar UI as simple as
> > possible.
> 
> This should be built into Sugar rather than in the non-Sugar backup
> script.  They should be maintained together.
> 
> This would be a code change to git repository sugar-datastore and the
> Journal activity in repository sugar.
> 
> > As always, there are complications. The original OLPC concept was
> > that there would be one XO per user. As a result the software was
> > designed for a single user identified by the XO serial number.
> 
> The multiple user feature is supported by Fedora and Sugar, but we
> removed it for OLPC OS.
> 
> > Today, many XO deployments provide enough XOs for a classroom.
> > During the day, different students use the XO as their class goes to
> > the computer lab or as the computers are distributed from classroom
> > to classroom. However, all of the documents created are in a single
> > Journal with only the user's memory to indicate which document goes
> > with which user.
> 
> OLPC did not design OLPC OS to be used in this scenario, so no
> surprise you've hit that.  But it's not a Sugar problem.  Don't
> conflate Sugar with OLPC OS.
> 
> > The OLPC Ubuntu Sugar 14.04 Trusty LTS (to use its official name)
> > solves this problem at the laptop side by using standard gnu/linux
> > logins.
> 
> The multiple user feature is supported by Ubuntu and Sugar, and I
> haven't removed it yet.  I know how to; small configuration change to
> lightdm package.
> 
> Don't forget SoaS.  The Fedora 23 SoaS is easily installed to disk and
> has multiple user capability.  The Fedora 24 SoaS is shaping up to be
> just as good or better, since it is based on Sugar 0.108.
> 
> > Each user has her own username and password. The Sugar activities
> > have been moved to common space in the file system so only one copy
> > is needed to support multiple users. Users are not 'olpc' but
> > identified by their username.  However, the datastore is part of the
> > user space (one datastore per user).
> 
> Yes.  ODPU.
> 
> > This is problematic since the backup script uploads to
> > /library/user/serial-number on the school server.
> 
> No, you're wrong.  In the Ubuntu scenario, the register_laptop
> function will invent a serial number because it won't find Open
> Firmware [1].  So it wouldn't be a problem.  It doesn't sound like
> you've tested this.
> 
> 1.
>  https://github.com/sugarlabs/sugar/blob/master/src/jarabe/desktop/schoolserver.py#L110
> 
> > So, one strategy would be to upload to /library/user/username. This
> > requires that usernames be unique across all laptops using a given
> > schoolserver. This could be enforced at registration on the school
> > server.
> 
> Starting to sound very complicated.  Single-sign-on (SSO) across a
> school.  These are truly amazing teachers with lots of free
> administration time.
> 
> (There are deployments using Sugar with SSO already, but as it's
> outside the scope of Sugar we don't hear about them at Sugar Labs, and
> we don't provide the facility in OLPC OS, but that doesn't stop them.)
> 
> > However, the Sugar releases for the XO 
> 
> We call that OLPC OS, which includes Sugar and Gnome desktops.
> 
> > still maintains Sugar activities in /home/olpc/Activities. So, one
> > requirement is to restructure Sugar as was done for OLPC Ubuntu
> > Sugar 14.04 Trusty LTS.
> 
> That would not block implementing a server datastore, since the
> implementation would not care what $HOME is set to.
> 
> (And besides, it's already done for SoaS, so the Fedora activity
> packages can be used immediately.)
> 
> > Another approach might be to create directories for each user of a
> > single XO (e.g.  /library/user/serial-number/user1).
> 
> That would require authentication service by the server datastore.
>

Think idmgr could be tweaked, '/library/user/serial-number' is just
homedir= from
http://dev.laptop.org/git/projects/idmgr/tree/scripts/create_user but does
not handle nickname from
http://dev.laptop.org/git/projects/idmgr/tree/registration-server 
 
> > Another complication is that the Browse activity downloads files
> > from the school server to the Journal (e.g. pdfs, mp3). These
> > documents do not need to be saved to the users Journal backup on the
> > school server since they can be restored from the school server
> > 'library'. Also, such documents when downloaded should be stored in
> > a common space available to all users of that laptop. Fortunately,
> > the source of a document is provided in the metadata.
> 
> What you describe here can also be solved by deduplication.
> 
> The Journal Git backend proposed by Martin and Walter could help with
> deduplication of journal objects across multiple journals.
> 
> https://wiki.sugarlabs.org/go/Summer_of_Code/2016#Sugar_Core
> 
> > One approach would be to divide the datastore into two directories
> > on the laptop, one in common space and the other local to the user.
> > The Journal could show both sets of objects.
> 
> Or the server datastore would recognise content hashes of server
> artefacts and know it need not send the content from the client to the
> server before LRU local deletion.  It could hard link it.
> 
> > Finally, each Journal object consists of a metadata file and an
> > optional document. The metadata files tend to clutter the Journal
> > display (mine has hundreds of Terminal activity and Log activity
> > entries). I would propose that the Journal show only objects which
> > have a document with a user-supplied name (a metadata flag). The
> > script should backup the metadata files for those objects without a
> > document to a 'log' on the school server for statistical analysis
> > but delete them from the local datastore. Journal objects saved
> > without a user-supplied name (but something like Write.activity)
> > should have their document deleted. As part of GSOC there is an
> > initiative to require users to supply a name for documents they wish
> > to save - so this problem may not be part of the 'backup' scheme.
> > Whether a document is saved or deleted, the metadata can be saved to
> > the log and displayed by the existing statistical tools.
> 
> I'm against any classification of journal objects in this way.  We
> cannot know how useful a Terminal and Log activity object is to the
> learner.
> 
> However, I would like a way for expert users to terminate an activity
> without saving a journal object.
> 
> > As an old crumudgeon, I still believe design precedes coding.
> > 
> > Reading the existing code is always a good idea:
> > 
> > Sugar
> > 
> >     *
> > /usr/lib/python2.7/site-packages/jarabe/desktop/schoolserver.py
> > #registers server - notice transition from gconf to gsettings
> >     * /usr/bin/ds_backup.sh    #primarily decides if backup can be run
> >                                              #backup logic is needed
> > because an rsync can use a lot of bandwidth in a local network
> >     * /usr/bin/ds_backup.py    #actually does the backup using rsync
> > (note: -d option AFAIK deletes an object from the backup if it is
> > deleted in the source,
> >                                              #this has the effect of
> > limiting the size of the datastore to the available space on the XO
> > not on the school server).
> > 
> > Server (xsce6)
> > 
> >     * /usr/libexec/idmgr         #contains a number of utilities
> > used in registration
> >     * /library/users                 #contains a directory per
> > serial-number of registered user
> >                                             #use ls -a to see files
> > created. The idmgr creates a public/private key pair which is used
> > by sftp to authenticate - avoiding password
> > 
> > Note: if you look at the server code, you can see why registering
> > the laptop on each connection works (and can avoid any need for a
> > registration menu item).
> > 
> > When you get to know your way around the existing process, I'll send
> > you a copy of the ds_backup.py code I use to implement the item by
> > item backup.
> 
> You should start using GitHub like the rest of us.
>

Thanks James for the thoughtful response.

Jerry