[Sugar-devel] Sugar-Server enhancement

[James Cameron]

On Fri, Apr 15, 2016 at 10:25:59AM +0800, Tony Anderson wrote:
> Hi Manash
> 
> The registration process is awkward but not the problem.

This is unfair scope creep.  Manash began by asking about bug #362 and
has been working to fix that.  Now you're asking him to consider a
much larger task; not a coding task, but a redesign of Sugar Journal
and Backup interaction.  This is huge.

And as far as I can tell, students aren't even accepted yet [1].

1.  https://en.wikipedia.org/wiki/Google_Summer_of_Code#2016

What you propose is from a set of tasks [2] you added to the Wiki,
which have not undergone any design review according to Sugar Labs
design practice and feature policy.  I do not see any consensus on
these; we're yet to build a consensus.

2.  https://wiki.sugarlabs.org/go/Summer_of_Code/2016#Sugar_on_the_Ground

Or, it looks like you're trying to make your own fork of Sugar, which
I'm fine with, it's open source after all, but to push that on others
without their input is wrong.

If you proceed without consensus as a sole designer, then OLPC will
fork Sugar (as we already have so that XO-1s will go faster), and
you'll be making your own builds.

> The problem is that rsync is used to create backups of the Journal
> and no effective means is offered to restore.

Agreed.  We have no restore from server feature in Sugar 0.108, along
with no way to start a backup to server, and no selective restore.

(We have backup to media, restore from media, but no selective
restore from media.  Also, restore from media replaces Journal!)

> However, the ultimate problem is thinking of the problem as one of
> backup. If you try to solve the wrong problem, often the result is a
> wasted effort.
> 
> The Journal is single place where Sugar users save their documents.
> This is done by the Sugar activities when they close. The majority
> of XOs are still XO-1s with a 1GB store.

This point in your argument is void, because XO-1 are 45% of the XO
laptops manufactured so far.  I have the numbers.

Also, many XO-1 have been upgraded with an SD card.

> If the available store is less than 50GB,

No, that's 50 MB, not 50 GB.  See _SPACE_TRESHOLD (sic) in
sugar:src/jarabe/journal/journalactivity.py [3].

3.  https://github.com/sugarlabs/sugar/blob/master/src/jarabe/journal/journalactivity.py#L56

> Sugar effectively shuts down.

This point in your argument is void, because this has been fixed [4,
5, 6], please upgrade to Sugar 0.108 which is in OLPC OS 13.2.7 [6].

4.  http://dev.laptop.org/ticket/9623
5.  https://bugs.sugarlabs.org/ticket/1720
6.  http://wiki.laptop.org/go/Release_notes/13.2.7#Fixes

> This typically results in the deployment reflashing the XO erasing
> all of the documents created by that user - a tragedy.

It was a known bug, so that's a training issue.  You previously
proposed to train a teacher to use "rm -rf" to delete a known_hosts
file instead of Manash coding up an "ssh-keygen -R" command.  It is
inconsistent to be able to do one and not the other.

> What I am proposing is to use the school server as the primary store
> for the Journal with its effectively unlimited storage capacity. The
> ds_backup script needs to read the datastore uploading any new or
> modified documents. The local datastore can then be viewed as a
> cache for current working documents.

I'm favour of this ideal in principle, but it remains a huge design
and consensus challenge, not a coding challenge.

However, with the XO-1, XO-1.5 and XO-1.75 using IEEE 802.11g the
local wireless network will collapse sooner due to this new load.

> On the XO, the datastore is shown in the Journal. The 'keep' star

There's no such thing.  There's a favorite star [7].  It has a defined
purpose.  Are you proposing to destroy that purpose, or add another
column to the journal?  There's even less room now that the multiple
selection checkbox was added.

7.  https://help.sugarlabs.org/en/journal.html#journal-features

> could be used to show whether there is a local copy of that document
> or not. If the document is not needed locally, the user can clear
> the star. In this case, the backup script could delete the local
> copy. If there is no local copy of the document, then the user could
> set the star. In this case the backup script could download the
> document.

My preference would be for the flag to be in the Journal detail
view [8], where there is available display space.

8.  https://help.sugarlabs.org/en/journal.html#journal-detail-view

> This capability could be used to set a quota on the amount of space
> used by the Journal. If the space is exceeded, the 'backup' script
> could delete local copies of document by LRU until the quota is met.
> Similarly, there should be a quota on Sugar activities which could
> also automatically be pruned back LRU. Managing the store
> automatically is consistent with keeping the Sugar UI as simple as
> possible.

This should be built into Sugar rather than in the non-Sugar backup
script.  They should be maintained together.

This would be a code change to git repository sugar-datastore and the
Journal activity in repository sugar.

> As always, there are complications. The original OLPC concept was
> that there would be one XO per user. As a result the software was
> designed for a single user identified by the XO serial number.

The multiple user feature is supported by Fedora and Sugar, but we
removed it for OLPC OS.

> Today, many XO deployments provide enough XOs for a classroom.
> During the day, different students use the XO as their class goes to
> the computer lab or as the computers are distributed from classroom
> to classroom. However, all of the documents created are in a single
> Journal with only the user's memory to indicate which document goes
> with which user.

OLPC did not design OLPC OS to be used in this scenario, so no
surprise you've hit that.  But it's not a Sugar problem.  Don't
conflate Sugar with OLPC OS.

> The OLPC Ubuntu Sugar 14.04 Trusty LTS (to use its official name)
> solves this problem at the laptop side by using standard gnu/linux
> logins.

The multiple user feature is supported by Ubuntu and Sugar, and I
haven't removed it yet.  I know how to; small configuration change to
lightdm package.

Don't forget SoaS.  The Fedora 23 SoaS is easily installed to disk and
has multiple user capability.  The Fedora 24 SoaS is shaping up to be
just as good or better, since it is based on Sugar 0.108.

> Each user has her own username and password. The Sugar activities
> have been moved to common space in the file system so only one copy
> is needed to support multiple users. Users are not 'olpc' but
> identified by their username.  However, the datastore is part of the
> user space (one datastore per user).

Yes.  ODPU.

> This is problematic since the backup script uploads to
> /library/user/serial-number on the school server.

No, you're wrong.  In the Ubuntu scenario, the register_laptop
function will invent a serial number because it won't find Open
Firmware [1].  So it wouldn't be a problem.  It doesn't sound like
you've tested this.

1.  https://github.com/sugarlabs/sugar/blob/master/src/jarabe/desktop/schoolserver.py#L110

> So, one strategy would be to upload to /library/user/username. This
> requires that usernames be unique across all laptops using a given
> schoolserver. This could be enforced at registration on the school
> server.

Starting to sound very complicated.  Single-sign-on (SSO) across a
school.  These are truly amazing teachers with lots of free
administration time.

(There are deployments using Sugar with SSO already, but as it's
outside the scope of Sugar we don't hear about them at Sugar Labs, and
we don't provide the facility in OLPC OS, but that doesn't stop them.)

> However, the Sugar releases for the XO 

We call that OLPC OS, which includes Sugar and Gnome desktops.

> still maintains Sugar activities in /home/olpc/Activities. So, one
> requirement is to restructure Sugar as was done for OLPC Ubuntu
> Sugar 14.04 Trusty LTS.

That would not block implementing a server datastore, since the
implementation would not care what $HOME is set to.

(And besides, it's already done for SoaS, so the Fedora activity
packages can be used immediately.)

> Another approach might be to create directories for each user of a
> single XO (e.g.  /library/user/serial-number/user1).

That would require authentication service by the server datastore.

> Another complication is that the Browse activity downloads files
> from the school server to the Journal (e.g. pdfs, mp3). These
> documents do not need to be saved to the users Journal backup on the
> school server since they can be restored from the school server
> 'library'. Also, such documents when downloaded should be stored in
> a common space available to all users of that laptop. Fortunately,
> the source of a document is provided in the metadata.

What you describe here can also be solved by deduplication.

The Journal Git backend proposed by Martin and Walter could help with
deduplication of journal objects across multiple journals.

https://wiki.sugarlabs.org/go/Summer_of_Code/2016#Sugar_Core

> One approach would be to divide the datastore into two directories
> on the laptop, one in common space and the other local to the user.
> The Journal could show both sets of objects.

Or the server datastore would recognise content hashes of server
artefacts and know it need not send the content from the client to the
server before LRU local deletion.  It could hard link it.

> Finally, each Journal object consists of a metadata file and an
> optional document. The metadata files tend to clutter the Journal
> display (mine has hundreds of Terminal activity and Log activity
> entries). I would propose that the Journal show only objects which
> have a document with a user-supplied name (a metadata flag). The
> script should backup the metadata files for those objects without a
> document to a 'log' on the school server for statistical analysis
> but delete them from the local datastore. Journal objects saved
> without a user-supplied name (but something like Write.activity)
> should have their document deleted. As part of GSOC there is an
> initiative to require users to supply a name for documents they wish
> to save - so this problem may not be part of the 'backup' scheme.
> Whether a document is saved or deleted, the metadata can be saved to
> the log and displayed by the existing statistical tools.

I'm against any classification of journal objects in this way.  We
cannot know how useful a Terminal and Log activity object is to the
learner.

However, I would like a way for expert users to terminate an activity
without saving a journal object.

> As an old crumudgeon, I still believe design precedes coding.
> 
> Reading the existing code is always a good idea:
> 
> Sugar
> 
>     *
> /usr/lib/python2.7/site-packages/jarabe/desktop/schoolserver.py
> #registers server - notice transition from gconf to gsettings
>     * /usr/bin/ds_backup.sh    #primarily decides if backup can be run
>                                              #backup logic is needed
> because an rsync can use a lot of bandwidth in a local network
>     * /usr/bin/ds_backup.py    #actually does the backup using rsync
> (note: -d option AFAIK deletes an object from the backup if it is
> deleted in the source,
>                                              #this has the effect of
> limiting the size of the datastore to the available space on the XO
> not on the school server).
> 
> Server (xsce6)
> 
>     * /usr/libexec/idmgr         #contains a number of utilities
> used in registration
>     * /library/users                 #contains a directory per
> serial-number of registered user
>                                             #use ls -a to see files
> created. The idmgr creates a public/private key pair which is used
> by sftp to authenticate - avoiding password
> 
> Note: if you look at the server code, you can see why registering
> the laptop on each connection works (and can avoid any need for a
> registration menu item).
> 
> When you get to know your way around the existing process, I'll send
> you a copy of the ds_backup.py code I use to implement the item by
> item backup.

You should start using GitHub like the rest of us.

-- 
James Cameron
http://quozl.netrek.org/