[Sugar-devel] Sugar-Server enhancement

[Tony Anderson]

Hi, James

This thread was getting long so I replied only to the most recent 
communication. I am sure you have the full thread which shows the scope 
of the discussion.

According to trac bug #362 was opened seven years ago against 0.82 and 
last looked at three years ago. Several competent people looked at it 
and left comments. I see none that signify consensus.

To have a design discussion, it is valuable to have a proposed design.

I have tried to explain my proposal in detail. If there are questions, I 
would be happy to try to respond. Fixing the 'Journal is Full' dialog is 
a major help. However, what do you recommend to deployments when this 
happens? The bottom line is that a reasonably active user is likely to 
need more room to store her Journal than is available on the XO.

In the Journal code a filled star sets the 'keep' flag in the metadata. 
The cleared star clears the 'keep' flag in the metadata. Using this 
feature greatly simplifies the coding and the Journal view. As far as I 
know, the only use of this at the moment is to support the Portfolio 
activity.

I think the detail view is inappropriate exactly as it would be to move 
the multiple selection checkbox there. These controls need to be 
immediately available.

The 'backup/sync' script is a good place to do check storage quotas 
because the script needs to touch the datastore on a regular basis. It 
has access to the amount of store in use and the LRU information. For 
example, if the user wants a document downloaded, the script knows its 
size and whether some other local copies need to be deleted to make room.

While an implementation detail, so far no change has been necessary to 
the datastore class. Actually, since the 'keep' or favorite star sets 
the metadata,
so far there has been no need to change the Journal.

"The multiple user feature is supported by Fedora and Sugar, but we removed it for OLPC OS."

I think I am beginning to understand. OLPC OS is your generic name for 
the images to be installed on each model of the XO. I am deploying build 
13.2.5 with Sugar 0.106 on all models. So you are saying that we, users 
of Sugar or ' OLPC OS' could have a multiple user version of Sugar if 
'you', as developers, didn't remove it.


As I understand it, you propose to generate unique serial-numbers per 
user. So SSO would be guaranteed since no two users could have the same 
serial-number. This would certainly work and probably involve very 
little change to the existing code. What will be needed is a 'dns' to 
map serial-numbers to usernames.

Every school I have worked with keeps a careful record of students 
(often in paper ledgers). Currently I provide a name record in a Django 
database on the server (along with an XO inventory by serial number).

Agreed that determining which Journal objects need to be saved to the 
school server is not a difficult problem. However, datastore is a class 
so each user's datastore and the common datastore would be instances. So 
this seemed like a simple thing to implement.

Actually, The deletion of Journal objects without an associated document 
works amazingly well. The number of objects in the Journal view goes 
from hundreds to only a few (often less than 20). Moreover, these 20 are 
the obviously interesting ones. Nothing is lost as the metadata is saved 
to the school server. It becomes much easier to 'reflect' when you are 
only looking at the documents you created. Meanwhile the myriad of 
objects can be subject to statistical analysis.

For many activities, such as the Terminal, the document saved is 
actually 'state' information. This allows the Terminal activity to be 
restored with tabs and pwd. There are many game activities such as 
Memorize that also store state. It would seem more appropriate to save 
this state information in the metadata. For example, a json could be 
created in the metadata to hold state information. The script could keep 
these objects to enable the user to resume.

Tony

On 04/15/2016 01:36 PM, James Cameron wrote:
> On Fri, Apr 15, 2016 at 10:25:59AM +0800, Tony Anderson wrote:
>> Hi Manash
>>
>> The registration process is awkward but not the problem.
> This is unfair scope creep.  Manash began by asking about bug #362 and
> has been working to fix that.  Now you're asking him to consider a
> much larger task; not a coding task, but a redesign of Sugar Journal
> and Backup interaction.  This is huge.
>
> And as far as I can tell, students aren't even accepted yet [1].
>
> 1.  https://en.wikipedia.org/wiki/Google_Summer_of_Code#2016
>
> What you propose is from a set of tasks [2] you added to the Wiki,
> which have not undergone any design review according to Sugar Labs
> design practice and feature policy.  I do not see any consensus on
> these; we're yet to build a consensus.
>
> 2.  https://wiki.sugarlabs.org/go/Summer_of_Code/2016#Sugar_on_the_Ground
>
> Or, it looks like you're trying to make your own fork of Sugar, which
> I'm fine with, it's open source after all, but to push that on others
> without their input is wrong.
>
> If you proceed without consensus as a sole designer, then OLPC will
> fork Sugar (as we already have so that XO-1s will go faster), and
> you'll be making your own builds.
>
>> The problem is that rsync is used to create backups of the Journal
>> and no effective means is offered to restore.
> Agreed.  We have no restore from server feature in Sugar 0.108, along
> with no way to start a backup to server, and no selective restore.
>
> (We have backup to media, restore from media, but no selective
> restore from media.  Also, restore from media replaces Journal!)
>
>> However, the ultimate problem is thinking of the problem as one of
>> backup. If you try to solve the wrong problem, often the result is a
>> wasted effort.
>>
>> The Journal is single place where Sugar users save their documents.
>> This is done by the Sugar activities when they close. The majority
>> of XOs are still XO-1s with a 1GB store.
> This point in your argument is void, because XO-1 are 45% of the XO
> laptops manufactured so far.  I have the numbers.
>
> Also, many XO-1 have been upgraded with an SD card.
>
>> If the available store is less than 50GB,
> No, that's 50 MB, not 50 GB.  See _SPACE_TRESHOLD (sic) in
> sugar:src/jarabe/journal/journalactivity.py [3].
>
> 3.  https://github.com/sugarlabs/sugar/blob/master/src/jarabe/journal/journalactivity.py#L56
>
>> Sugar effectively shuts down.
> This point in your argument is void, because this has been fixed [4,
> 5, 6], please upgrade to Sugar 0.108 which is in OLPC OS 13.2.7 [6].
>
> 4.  http://dev.laptop.org/ticket/9623
> 5.  https://bugs.sugarlabs.org/ticket/1720
> 6.  http://wiki.laptop.org/go/Release_notes/13.2.7#Fixes
>
>> This typically results in the deployment reflashing the XO erasing
>> all of the documents created by that user - a tragedy.
> It was a known bug, so that's a training issue.  You previously
> proposed to train a teacher to use "rm -rf" to delete a known_hosts
> file instead of Manash coding up an "ssh-keygen -R" command.  It is
> inconsistent to be able to do one and not the other.
>
>> What I am proposing is to use the school server as the primary store
>> for the Journal with its effectively unlimited storage capacity. The
>> ds_backup script needs to read the datastore uploading any new or
>> modified documents. The local datastore can then be viewed as a
>> cache for current working documents.
> I'm favour of this ideal in principle, but it remains a huge design
> and consensus challenge, not a coding challenge.
>
> However, with the XO-1, XO-1.5 and XO-1.75 using IEEE 802.11g the
> local wireless network will collapse sooner due to this new load.
>
>> On the XO, the datastore is shown in the Journal. The 'keep' star
> There's no such thing.  There's a favorite star [7].  It has a defined
> purpose.  Are you proposing to destroy that purpose, or add another
> column to the journal?  There's even less room now that the multiple
> selection checkbox was added.
>
> 7.  https://help.sugarlabs.org/en/journal.html#journal-features
>
>> could be used to show whether there is a local copy of that document
>> or not. If the document is not needed locally, the user can clear
>> the star. In this case, the backup script could delete the local
>> copy. If there is no local copy of the document, then the user could
>> set the star. In this case the backup script could download the
>> document.
> My preference would be for the flag to be in the Journal detail
> view [8], where there is available display space.
>
> 8.  https://help.sugarlabs.org/en/journal.html#journal-detail-view
>
>> This capability could be used to set a quota on the amount of space
>> used by the Journal. If the space is exceeded, the 'backup' script
>> could delete local copies of document by LRU until the quota is met.
>> Similarly, there should be a quota on Sugar activities which could
>> also automatically be pruned back LRU. Managing the store
>> automatically is consistent with keeping the Sugar UI as simple as
>> possible.
> This should be built into Sugar rather than in the non-Sugar backup
> script.  They should be maintained together.
>
> This would be a code change to git repository sugar-datastore and the
> Journal activity in repository sugar.
>
>> As always, there are complications. The original OLPC concept was
>> that there would be one XO per user. As a result the software was
>> designed for a single user identified by the XO serial number.
> The multiple user feature is supported by Fedora and Sugar, but we
> removed it for OLPC OS.
>
>> Today, many XO deployments provide enough XOs for a classroom.
>> During the day, different students use the XO as their class goes to
>> the computer lab or as the computers are distributed from classroom
>> to classroom. However, all of the documents created are in a single
>> Journal with only the user's memory to indicate which document goes
>> with which user.
> OLPC did not design OLPC OS to be used in this scenario, so no
> surprise you've hit that.  But it's not a Sugar problem.  Don't
> conflate Sugar with OLPC OS.
>
>> The OLPC Ubuntu Sugar 14.04 Trusty LTS (to use its official name)
>> solves this problem at the laptop side by using standard gnu/linux
>> logins.
> The multiple user feature is supported by Ubuntu and Sugar, and I
> haven't removed it yet.  I know how to; small configuration change to
> lightdm package.
>
> Don't forget SoaS.  The Fedora 23 SoaS is easily installed to disk and
> has multiple user capability.  The Fedora 24 SoaS is shaping up to be
> just as good or better, since it is based on Sugar 0.108.
>
>> Each user has her own username and password. The Sugar activities
>> have been moved to common space in the file system so only one copy
>> is needed to support multiple users. Users are not 'olpc' but
>> identified by their username.  However, the datastore is part of the
>> user space (one datastore per user).
> Yes.  ODPU.
>
>> This is problematic since the backup script uploads to
>> /library/user/serial-number on the school server.
> No, you're wrong.  In the Ubuntu scenario, the register_laptop
> function will invent a serial number because it won't find Open
> Firmware [1].  So it wouldn't be a problem.  It doesn't sound like
> you've tested this.
>
> 1.  https://github.com/sugarlabs/sugar/blob/master/src/jarabe/desktop/schoolserver.py#L110
>
>> So, one strategy would be to upload to /library/user/username. This
>> requires that usernames be unique across all laptops using a given
>> schoolserver. This could be enforced at registration on the school
>> server.
> Starting to sound very complicated.  Single-sign-on (SSO) across a
> school.  These are truly amazing teachers with lots of free
> administration time.
>
> (There are deployments using Sugar with SSO already, but as it's
> outside the scope of Sugar we don't hear about them at Sugar Labs, and
> we don't provide the facility in OLPC OS, but that doesn't stop them.)
>
>> However, the Sugar releases for the XO
> We call that OLPC OS, which includes Sugar and Gnome desktops.
>
>> still maintains Sugar activities in /home/olpc/Activities. So, one
>> requirement is to restructure Sugar as was done for OLPC Ubuntu
>> Sugar 14.04 Trusty LTS.
> That would not block implementing a server datastore, since the
> implementation would not care what $HOME is set to.
>
> (And besides, it's already done for SoaS, so the Fedora activity
> packages can be used immediately.)
>
>> Another approach might be to create directories for each user of a
>> single XO (e.g.  /library/user/serial-number/user1).
> That would require authentication service by the server datastore.
>
>> Another complication is that the Browse activity downloads files
>> from the school server to the Journal (e.g. pdfs, mp3). These
>> documents do not need to be saved to the users Journal backup on the
>> school server since they can be restored from the school server
>> 'library'. Also, such documents when downloaded should be stored in
>> a common space available to all users of that laptop. Fortunately,
>> the source of a document is provided in the metadata.
> What you describe here can also be solved by deduplication.
>
> The Journal Git backend proposed by Martin and Walter could help with
> deduplication of journal objects across multiple journals.
>
> https://wiki.sugarlabs.org/go/Summer_of_Code/2016#Sugar_Core
>
>> One approach would be to divide the datastore into two directories
>> on the laptop, one in common space and the other local to the user.
>> The Journal could show both sets of objects.
> Or the server datastore would recognise content hashes of server
> artefacts and know it need not send the content from the client to the
> server before LRU local deletion.  It could hard link it.
>
>> Finally, each Journal object consists of a metadata file and an
>> optional document. The metadata files tend to clutter the Journal
>> display (mine has hundreds of Terminal activity and Log activity
>> entries). I would propose that the Journal show only objects which
>> have a document with a user-supplied name (a metadata flag). The
>> script should backup the metadata files for those objects without a
>> document to a 'log' on the school server for statistical analysis
>> but delete them from the local datastore. Journal objects saved
>> without a user-supplied name (but something like Write.activity)
>> should have their document deleted. As part of GSOC there is an
>> initiative to require users to supply a name for documents they wish
>> to save - so this problem may not be part of the 'backup' scheme.
>> Whether a document is saved or deleted, the metadata can be saved to
>> the log and displayed by the existing statistical tools.
> I'm against any classification of journal objects in this way.  We
> cannot know how useful a Terminal and Log activity object is to the
> learner.
>
> However, I would like a way for expert users to terminate an activity
> without saving a journal object.
>
>> As an old crumudgeon, I still believe design precedes coding.
>>
>> Reading the existing code is always a good idea:
>>
>> Sugar
>>
>>      *
>> /usr/lib/python2.7/site-packages/jarabe/desktop/schoolserver.py
>> #registers server - notice transition from gconf to gsettings
>>      * /usr/bin/ds_backup.sh    #primarily decides if backup can be run
>>                                               #backup logic is needed
>> because an rsync can use a lot of bandwidth in a local network
>>      * /usr/bin/ds_backup.py    #actually does the backup using rsync
>> (note: -d option AFAIK deletes an object from the backup if it is
>> deleted in the source,
>>                                               #this has the effect of
>> limiting the size of the datastore to the available space on the XO
>> not on the school server).
>>
>> Server (xsce6)
>>
>>      * /usr/libexec/idmgr         #contains a number of utilities
>> used in registration
>>      * /library/users                 #contains a directory per
>> serial-number of registered user
>>                                              #use ls -a to see files
>> created. The idmgr creates a public/private key pair which is used
>> by sftp to authenticate - avoiding password
>>
>> Note: if you look at the server code, you can see why registering
>> the laptop on each connection works (and can avoid any need for a
>> registration menu item).
>>
>> When you get to know your way around the existing process, I'll send
>> you a copy of the ds_backup.py code I use to implement the item by
>> item backup.
> You should start using GitHub like the rest of us.
>