[Sugar-devel] Sugar-Server enhancement

[Tony Anderson]

Hi Manash

The registration process is awkward but not the problem. The problem is 
that rsync is used to create backups of the Journal and no effective
means is offered to restore. However, the ultimate problem is thinking 
of the problem as one of backup. If you try to solve the wrong problem, 
often the result is a wasted effort.

The Journal is single place where Sugar users save their documents. This 
is done by the Sugar activities when they close. The majority of XOs
are still XO-1s with a 1GB store. If the available store is less than 
50GB, Sugar effectively shuts down. This typically results in the 
deployment reflashing the XO
erasing all of the documents created by that user - a tragedy.

What I am proposing is to use the school server as the primary store for 
the Journal with its effectively unlimited storage capacity. The ds_backup
script needs to read the datastore uploading any new or modified 
documents. The local datastore can then be viewed as a cache for current 
working documents.

On the XO, the datastore is shown in the Journal. The 'keep' star could 
be used to show whether there is a local copy of that document or not. 
If the document is not needed locally, the user can clear the star. In 
this case, the backup script could delete the local copy. If there is no 
local copy of the document, then the user could set the star. In this 
case the backup script could download the document.

This capability could be used to set a quota on the amount of space used 
by the Journal. If the space is exceeded, the 'backup' script could 
delete local copies of document by LRU until the quota is met. 
Similarly, there should be a quota on Sugar activities which could also 
automatically be pruned back LRU. Managing the store automatically is 
consistent with keeping the Sugar UI as simple as possible.

As always, there are complications. The original OLPC concept was that 
there would be one XO per user. As a result the software was designed 
for a single user identified by the XO serial number. Today, many XO 
deployments provide enough XOs for a classroom. During the day, 
different students use the XO as their class goes to the computer lab or 
as the computers are distributed from classroom to classroom. However, 
all of the documents created are in a single Journal with only the 
user's memory to indicate which document goes with which user.

The OLPC Ubuntu Sugar 14.04 Trusty LTS (to use its official name) solves 
this problem at the laptop side by using standard gnu/linux logins. Each 
user has
her own username and password. The Sugar activities have been moved to 
common space in the file system so only one copy is needed to support 
multiple
users. Users are not 'olpc' but identified by their username. However, 
the datastore is part of the user space (one datastore per user). This 
is problematic since the backup script uploads to 
/library/user/serial-number on the school server.

So, one strategy would be to upload to /library/user/username. This 
requires that usernames be unique across all laptops using a given 
schoolserver. This could be enforced at registration on the school 
server. However, the Sugar releases for the XO still maintains Sugar 
activities in /home/olpc/Activities. So,
one requirement is to restructure Sugar as was done for OLPC Ubuntu 
Sugar 14.04 Trusty LTS. Another approach might be to create directories 
for each user of a single XO (e.g. /library/user/serial-number/user1).

Another complication is that the Browse activity downloads files from 
the school server to the Journal (e.g. pdfs, mp3). These documents do 
not need to be saved to the users Journal backup on the school server 
since they can be restored from the school server 'library'. Also, such 
documents when downloaded should be stored in a common space available 
to all users of that laptop. Fortunately, the source of a document is 
provided in the metadata.

One approach would be to divide the datastore into two directories on 
the laptop, one in common space and the other local to the user. The 
Journal could
show both sets of objects.

Finally, each Journal object consists of a metadata file and an optional 
document. The metadata files tend to clutter the Journal display (mine 
has hundreds of Terminal activity and Log activity entries). I would 
propose that the Journal show only objects which have a document with a 
user-supplied name (a metadata flag). The script should backup the 
metadata files for those objects without a document to a 'log' on the 
school server for statistical analysis but delete them from the local 
datastore. Journal objects saved without a user-supplied name (but 
something like Write.activity) should have their document deleted. As 
part of GSOC there is an initiative to require users to supply a name 
for documents they wish to save - so this problem may not be part of the 
'backup' scheme. Whether a document is saved or deleted, the metadata 
can be saved to the log and displayed by the existing statistical tools.

As an old crumudgeon, I still believe design precedes coding.

Reading the existing code is always a good idea:

Sugar

     * /usr/lib/python2.7/site-packages/jarabe/desktop/schoolserver.py 
#registers server - notice transition from gconf to gsettings
     * /usr/bin/ds_backup.sh    #primarily decides if backup can be run
                                              #backup logic is needed 
because an rsync can use a lot of bandwidth in a local network
     * /usr/bin/ds_backup.py    #actually does the backup using rsync 
(note: -d option AFAIK deletes an object from the backup if it is 
deleted in the source,
                                              #this has the effect of 
limiting the size of the datastore to the available space on the XO not 
on the school server).

Server (xsce6)

     * /usr/libexec/idmgr         #contains a number of utilities used 
in registration
     * /library/users                 #contains a directory per 
serial-number of registered user
                                             #use ls -a to see files 
created. The idmgr creates a public/private key pair which is used by 
sftp to authenticate - avoiding password

Note: if you look at the server code, you can see why registering the 
laptop on each connection works (and can avoid any need for a 
registration menu item).

When you get to know your way around the existing process, I'll send you 
a copy of the ds_backup.py code I use to implement the item by item backup.

Tony