[Sugar-devel] Sugar-Server enhancement
James Cameron
quozl at laptop.org
Fri Apr 15 17:51:38 EDT 2016
Looks good. A few points on first review;
1. you're changing "Register" and "Register again" to "Connect to
server", but it isn't clear in the commit message why it is necessary;
(In my opinion "Register again" does continue to convey the concept
even if the server has never encountered the laptop before. Perhaps
you focused on an implementation detail that is not important to the
user?)
2. please avoid `ssh-keygen -R` if server is unchanged, because
otherwise a rogue server can be introduced after registration,
(Would it be possible to change .ssh/config to add an entry for each
server? If so, the known_hosts file might not need changing at all).
3. please check for `os.command()` failure, which could happen if the
disk is full,
4. please add commit message references to the server-side changes
and invite server-devel at lists.laptop.org people with your feature page
and patch.
On Sat, Apr 16, 2016 at 02:48:59AM +0530, Manash Raja wrote:
> Hi all,
>
> I fixed the bug. I haven't created a PR just yet. [1]https://github.com/
> ManashRaja/sugar/commit/660985d2183416cd3ed758095e92adf82f87a10c
> As of now I am parsing HTML data for json data to look for registered laptops
> at an XS from the webaddress [2]http://schoolserver:5000 that serves a liist of
> registered laptops.
> I will modify it if there is another better method that can be implemented from
> the Sugar side (we don't intend to modify the server side I guess).
>
> And here is the feature page as suggested by James. [3]https://
> wiki.sugarlabs.org/go/Features/Multi_XS-server_registration
>
> Please have a look.
>
> I am very excited to be a part of the discussions that go into the making of
> great features that can affect people down there in deployment. :) There is a
> scope to do so much.
>
> Thanks
>
> On Sat, Apr 16, 2016 at 2:32 AM, James Cameron <[4]quozl at laptop.org> wrote:
>
> On Fri, Apr 15, 2016 at 04:02:39PM +0800, Tony Anderson wrote:
> > Hi, James
> >
> > This thread was getting long so I replied only to the most recent
> > communication. I am sure you have the full thread which shows the
> > scope of the discussion.
>
> You're making it longer, yes, by hijacking it. You can find the full
> thread here:
>
> [5]http://lists.sugarlabs.org/archive/sugar-devel/2016-April/thread.html
>
> > According to trac bug #362 was opened seven years ago against 0.82
> > and last looked at three years ago. Several competent people looked
> > at it and left comments. I see none that signify consensus.
>
> I take it that you don't want Manash to fix this bug, thanks.
>
> I'd like it fixed. I think there are others who want it fixed. It
> would probably help with XS testing as well.
>
> > To have a design discussion, it is valuable to have a proposed design.
> >
> > I have tried to explain my proposal in detail. If there are
> > questions, I would be happy to try to respond. Fixing the 'Journal
> > is Full' dialog is a major help. However, what do you recommend to
> > deployments when this happens?
>
> 1. upgrade to Sugar 0.108 (by RPM or images), or backport the patch
> [#9623, #1720] into your custom builds,
>
> 2. transcode content to play in browser not Journal,
>
> 3. delete any activities that are not needed,
>
> 4. deploy Sugar Network to use the network activity cache,
>
> Also delete the Browse temporary directories, you reported this on
> 19th January, it remains a problem, and you refused to test my fix, so
> I lost interest very rapidly. [#4931]
>
> > The bottom line is that a reasonably active user is likely to need
> > more room to store her Journal than is available on the XO.
>
> No, because it's no longer a significant problem. XO-1 are in the
> minority and getting rarer as they die. Those that haven't died have
> SD cards.
>
> > In the Journal code a filled star sets the 'keep' flag in the
> > metadata. The cleared star clears the 'keep' flag in the metadata.
> > Using this feature greatly simplifies the coding and the Journal
> > view. As far as I know, the only use of this at the moment is to
> > support the Portfolio activity.
>
> You are using an implementation detail in describing the flag. The
> name given to the flag in documentation and user interface is
> "favorite" (sic).
>
> > I think the detail view is inappropriate exactly as it would be to
> > move the multiple selection checkbox there. These controls need to
> > be immediately available.
>
> I disagree; they won't get used, and so it would be a waste of
> valuable vertical space. The reason the checkbox won't get used is;
>
> - most laptops don't have a server,
>
> - an LRU algorithm can maintain the cache effectively.
>
> > The 'backup/sync' script is a good place to do check storage quotas
> > because the script needs to touch the datastore on a regular basis.
> > It has access to the amount of store in use and the LRU information.
> > For example, if the user wants a document downloaded, the script
> > knows its size and whether some other local copies need to be
> > deleted to make room.
>
> I disagree. This script runs infrequently. The LRU must be
> implemented inside the datastore for it to function properly.
> Otherwise the lag between user action and response by the script would
> be too long.
>
> > While an implementation detail, so far no change has been necessary
> > to the datastore class.
>
> That's no reason not to change it.
>
> > Actually, since the 'keep' or favorite star sets the metadata,
> > so far there has been no need to change the Journal.
>
> That's no reason not to change it.
>
> > "The multiple user feature is supported by Fedora and Sugar, but we
> > removed it for OLPC OS."
> >
> > I think I am beginning to understand. OLPC OS is your generic name
> > for the images to be installed on each model of the XO.
>
> Really, you are out of touch! OLPC OS is our name for the operating
> system releases on the XO laptop. We've been using the term at OLPC
> for a very long time, and use it in each release announcement.
>
> > I am deploying build 13.2.5 with Sugar 0.106 on all models.
>
> That's so sad. It was released in July 2015. It has the journal full
> bug you mentioned. I'm not interested in supporting that release,
> because I've already released two others. Upgrade.
>
> > So you are saying that we, users of Sugar or ' OLPC OS' could have a
> > multiple user version of Sugar if 'you', as developers, didn't
> > remove it.
>
> Well done.
>
> > As I understand it, you propose to generate unique serial-numbers
> > per user.
>
> No. I was describing what happens _now_ during registration, by
> reference to the code:
>
> 1. for XO laptops the serial number of the laptop is used,
>
> 2. for non-XO laptops a serial number is generated randomly,
>
> 3. there is no attempt to ensure the random serial number is
> unique, but the width of the random string is sufficient to make it
> unlikely,
>
> > So SSO would be guaranteed since no two users could have
> > the same serial-number. This would certainly work and probably
> > involve very little change to the existing code. What will be needed
> > is a 'dns' to map serial-numbers to usernames.
>
> No, I wasn't proposing that. It's your idea. I don't think it's
> guaranteed to be unique though.
>
> > Every school I have worked with keeps a careful record of students
> > (often in paper ledgers). Currently I provide a name record in a
> > Django database on the server (along with an XO inventory by serial
> > number).
>
> Fail to see relevance. Not all schools will or can do this.
>
> > Agreed that determining which Journal objects need to be saved to
> > the school server is not a difficult problem. However, datastore is
> > a class so each user's datastore and the common datastore would be
> > instances. So this seemed like a simple thing to implement.
> >
> > Actually, The deletion of Journal objects without an associated
> > document works amazingly well. The number of objects in the Journal
> > view goes from hundreds to only a few (often less than 20).
> > Moreover, these 20 are the obviously interesting ones. Nothing is
> > lost as the metadata is saved to the school server. It becomes much
> > easier to 'reflect' when you are only looking at the documents you
> > created. Meanwhile the myriad of objects can be subject to
> > statistical analysis.
> >
> > For many activities, such as the Terminal, the document saved is
> > actually 'state' information. This allows the Terminal activity to
> > be restored with tabs and pwd. There are many game activities such
> > as Memorize that also store state. It would seem more appropriate to
> > save this state information in the metadata. For example, a json
> > could be created in the metadata to hold state information. The
> > script could keep these objects to enable the user to resume.
>
> Your concept of metadata is not of interest to me; journal objects
> must continue reflect the learner's use of the laptop if the journal
> is to meet the designed style of reflection.
>
> Unreflecting adults are not the target user.
>
> >
> > Tony
> >
> > On 04/15/2016 01:36 PM, James Cameron wrote:
> > >On Fri, Apr 15, 2016 at 10:25:59AM +0800, Tony Anderson wrote:
> > >>Hi Manash
> > >>
> > >>The registration process is awkward but not the problem.
> > >This is unfair scope creep. Manash began by asking about bug #362 and
> > >has been working to fix that. Now you're asking him to consider a
> > >much larger task; not a coding task, but a redesign of Sugar Journal
> > >and Backup interaction. This is huge.
> > >
> > >And as far as I can tell, students aren't even accepted yet [1].
> > >
> > >1. [6]https://en.wikipedia.org/wiki/Google_Summer_of_Code#2016
> > >
> > >What you propose is from a set of tasks [2] you added to the Wiki,
> > >which have not undergone any design review according to Sugar Labs
> > >design practice and feature policy. I do not see any consensus on
> > >these; we're yet to build a consensus.
> > >
> > >2. [7]https://wiki.sugarlabs.org/go/Summer_of_Code/2016#
> Sugar_on_the_Ground
> > >
> > >Or, it looks like you're trying to make your own fork of Sugar, which
> > >I'm fine with, it's open source after all, but to push that on others
> > >without their input is wrong.
> > >
> > >If you proceed without consensus as a sole designer, then OLPC will
> > >fork Sugar (as we already have so that XO-1s will go faster), and
> > >you'll be making your own builds.
> > >
> > >>The problem is that rsync is used to create backups of the Journal
> > >>and no effective means is offered to restore.
> > >Agreed. We have no restore from server feature in Sugar 0.108, along
> > >with no way to start a backup to server, and no selective restore.
> > >
> > >(We have backup to media, restore from media, but no selective
> > >restore from media. Also, restore from media replaces Journal!)
> > >
> > >>However, the ultimate problem is thinking of the problem as one of
> > >>backup. If you try to solve the wrong problem, often the result is a
> > >>wasted effort.
> > >>
> > >>The Journal is single place where Sugar users save their documents.
> > >>This is done by the Sugar activities when they close. The majority
> > >>of XOs are still XO-1s with a 1GB store.
> > >This point in your argument is void, because XO-1 are 45% of the XO
> > >laptops manufactured so far. I have the numbers.
> > >
> > >Also, many XO-1 have been upgraded with an SD card.
> > >
> > >>If the available store is less than 50GB,
> > >No, that's 50 MB, not 50 GB. See _SPACE_TRESHOLD (sic) in
> > >sugar:src/jarabe/journal/journalactivity.py [3].
> > >
> > >3. [8]https://github.com/sugarlabs/sugar/blob/master/src/jarabe/journal
> /journalactivity.py#L56
> > >
> > >>Sugar effectively shuts down.
> > >This point in your argument is void, because this has been fixed [4,
> > >5, 6], please upgrade to Sugar 0.108 which is in OLPC OS 13.2.7 [6].
> > >
> > >4. [9]http://dev.laptop.org/ticket/9623
> > >5. [10]https://bugs.sugarlabs.org/ticket/1720
> > >6. [11]http://wiki.laptop.org/go/Release_notes/13.2.7#Fixes
> > >
> > >>This typically results in the deployment reflashing the XO erasing
> > >>all of the documents created by that user - a tragedy.
> > >It was a known bug, so that's a training issue. You previously
> > >proposed to train a teacher to use "rm -rf" to delete a known_hosts
> > >file instead of Manash coding up an "ssh-keygen -R" command. It is
> > >inconsistent to be able to do one and not the other.
> > >
> > >>What I am proposing is to use the school server as the primary store
> > >>for the Journal with its effectively unlimited storage capacity. The
> > >>ds_backup script needs to read the datastore uploading any new or
> > >>modified documents. The local datastore can then be viewed as a
> > >>cache for current working documents.
> > >I'm favour of this ideal in principle, but it remains a huge design
> > >and consensus challenge, not a coding challenge.
> > >
> > >However, with the XO-1, XO-1.5 and XO-1.75 using IEEE 802.11g the
> > >local wireless network will collapse sooner due to this new load.
> > >
> > >>On the XO, the datastore is shown in the Journal. The 'keep' star
> > >There's no such thing. There's a favorite star [7]. It has a defined
> > >purpose. Are you proposing to destroy that purpose, or add another
> > >column to the journal? There's even less room now that the multiple
> > >selection checkbox was added.
> > >
> > >7. [12]https://help.sugarlabs.org/en/journal.html#journal-features
> > >
> > >>could be used to show whether there is a local copy of that document
> > >>or not. If the document is not needed locally, the user can clear
> > >>the star. In this case, the backup script could delete the local
> > >>copy. If there is no local copy of the document, then the user could
> > >>set the star. In this case the backup script could download the
> > >>document.
> > >My preference would be for the flag to be in the Journal detail
> > >view [8], where there is available display space.
> > >
> > >8. [13]https://help.sugarlabs.org/en/journal.html#journal-detail-view
> > >
> > >>This capability could be used to set a quota on the amount of space
> > >>used by the Journal. If the space is exceeded, the 'backup' script
> > >>could delete local copies of document by LRU until the quota is met.
> > >>Similarly, there should be a quota on Sugar activities which could
> > >>also automatically be pruned back LRU. Managing the store
> > >>automatically is consistent with keeping the Sugar UI as simple as
> > >>possible.
> > >This should be built into Sugar rather than in the non-Sugar backup
> > >script. They should be maintained together.
> > >
> > >This would be a code change to git repository sugar-datastore and the
> > >Journal activity in repository sugar.
> > >
> > >>As always, there are complications. The original OLPC concept was
> > >>that there would be one XO per user. As a result the software was
> > >>designed for a single user identified by the XO serial number.
> > >The multiple user feature is supported by Fedora and Sugar, but we
> > >removed it for OLPC OS.
> > >
> > >>Today, many XO deployments provide enough XOs for a classroom.
> > >>During the day, different students use the XO as their class goes to
> > >>the computer lab or as the computers are distributed from classroom
> > >>to classroom. However, all of the documents created are in a single
> > >>Journal with only the user's memory to indicate which document goes
> > >>with which user.
> > >OLPC did not design OLPC OS to be used in this scenario, so no
> > >surprise you've hit that. But it's not a Sugar problem. Don't
> > >conflate Sugar with OLPC OS.
> > >
> > >>The OLPC Ubuntu Sugar 14.04 Trusty LTS (to use its official name)
> > >>solves this problem at the laptop side by using standard gnu/linux
> > >>logins.
> > >The multiple user feature is supported by Ubuntu and Sugar, and I
> > >haven't removed it yet. I know how to; small configuration change to
> > >lightdm package.
> > >
> > >Don't forget SoaS. The Fedora 23 SoaS is easily installed to disk and
> > >has multiple user capability. The Fedora 24 SoaS is shaping up to be
> > >just as good or better, since it is based on Sugar 0.108.
> > >
> > >>Each user has her own username and password. The Sugar activities
> > >>have been moved to common space in the file system so only one copy
> > >>is needed to support multiple users. Users are not 'olpc' but
> > >>identified by their username. However, the datastore is part of the
> > >>user space (one datastore per user).
> > >Yes. ODPU.
> > >
> > >>This is problematic since the backup script uploads to
> > >>/library/user/serial-number on the school server.
> > >No, you're wrong. In the Ubuntu scenario, the register_laptop
> > >function will invent a serial number because it won't find Open
> > >Firmware [1]. So it wouldn't be a problem. It doesn't sound like
> > >you've tested this.
> > >
> > >1. [14]https://github.com/sugarlabs/sugar/blob/master/src/jarabe/
> desktop/schoolserver.py#L110
> > >
> > >>So, one strategy would be to upload to /library/user/username. This
> > >>requires that usernames be unique across all laptops using a given
> > >>schoolserver. This could be enforced at registration on the school
> > >>server.
> > >Starting to sound very complicated. Single-sign-on (SSO) across a
> > >school. These are truly amazing teachers with lots of free
> > >administration time.
> > >
> > >(There are deployments using Sugar with SSO already, but as it's
> > >outside the scope of Sugar we don't hear about them at Sugar Labs, and
> > >we don't provide the facility in OLPC OS, but that doesn't stop them.)
> > >
> > >>However, the Sugar releases for the XO
> > >We call that OLPC OS, which includes Sugar and Gnome desktops.
> > >
> > >>still maintains Sugar activities in /home/olpc/Activities. So, one
> > >>requirement is to restructure Sugar as was done for OLPC Ubuntu
> > >>Sugar 14.04 Trusty LTS.
> > >That would not block implementing a server datastore, since the
> > >implementation would not care what $HOME is set to.
> > >
> > >(And besides, it's already done for SoaS, so the Fedora activity
> > >packages can be used immediately.)
> > >
> > >>Another approach might be to create directories for each user of a
> > >>single XO (e.g. /library/user/serial-number/user1).
> > >That would require authentication service by the server datastore.
> > >
> > >>Another complication is that the Browse activity downloads files
> > >>from the school server to the Journal (e.g. pdfs, mp3). These
> > >>documents do not need to be saved to the users Journal backup on the
> > >>school server since they can be restored from the school server
> > >>'library'. Also, such documents when downloaded should be stored in
> > >>a common space available to all users of that laptop. Fortunately,
> > >>the source of a document is provided in the metadata.
> > >What you describe here can also be solved by deduplication.
> > >
> > >The Journal Git backend proposed by Martin and Walter could help with
> > >deduplication of journal objects across multiple journals.
> > >
> > >[15]https://wiki.sugarlabs.org/go/Summer_of_Code/2016#Sugar_Core
> > >
> > >>One approach would be to divide the datastore into two directories
> > >>on the laptop, one in common space and the other local to the user.
> > >>The Journal could show both sets of objects.
> > >Or the server datastore would recognise content hashes of server
> > >artefacts and know it need not send the content from the client to the
> > >server before LRU local deletion. It could hard link it.
> > >
> > >>Finally, each Journal object consists of a metadata file and an
> > >>optional document. The metadata files tend to clutter the Journal
> > >>display (mine has hundreds of Terminal activity and Log activity
> > >>entries). I would propose that the Journal show only objects which
> > >>have a document with a user-supplied name (a metadata flag). The
> > >>script should backup the metadata files for those objects without a
> > >>document to a 'log' on the school server for statistical analysis
> > >>but delete them from the local datastore. Journal objects saved
> > >>without a user-supplied name (but something like Write.activity)
> > >>should have their document deleted. As part of GSOC there is an
> > >>initiative to require users to supply a name for documents they wish
> > >>to save - so this problem may not be part of the 'backup' scheme.
> > >>Whether a document is saved or deleted, the metadata can be saved to
> > >>the log and displayed by the existing statistical tools.
> > >I'm against any classification of journal objects in this way. We
> > >cannot know how useful a Terminal and Log activity object is to the
> > >learner.
> > >
> > >However, I would like a way for expert users to terminate an activity
> > >without saving a journal object.
> > >
> > >>As an old crumudgeon, I still believe design precedes coding.
> > >>
> > >>Reading the existing code is always a good idea:
> > >>
> > >>Sugar
> > >>
> > >> *
> > >>/usr/lib/python2.7/site-packages/jarabe/desktop/schoolserver.py
> > >>#registers server - notice transition from gconf to gsettings
> > >> * /usr/bin/ds_backup.sh #primarily decides if backup can be run
> > >> #backup logic is needed
> > >>because an rsync can use a lot of bandwidth in a local network
> > >> * /usr/bin/ds_backup.py #actually does the backup using rsync
> > >>(note: -d option AFAIK deletes an object from the backup if it is
> > >>deleted in the source,
> > >> #this has the effect of
> > >>limiting the size of the datastore to the available space on the XO
> > >>not on the school server).
> > >>
> > >>Server (xsce6)
> > >>
> > >> * /usr/libexec/idmgr #contains a number of utilities
> > >>used in registration
> > >> * /library/users #contains a directory per
> > >>serial-number of registered user
> > >> #use ls -a to see files
> > >>created. The idmgr creates a public/private key pair which is used
> > >>by sftp to authenticate - avoiding password
> > >>
> > >>Note: if you look at the server code, you can see why registering
> > >>the laptop on each connection works (and can avoid any need for a
> > >>registration menu item).
> > >>
> > >>When you get to know your way around the existing process, I'll send
> > >>you a copy of the ds_backup.py code I use to implement the item by
> > >>item backup.
> > >You should start using GitHub like the rest of us.
> > >
> >
> > _______________________________________________
> > Sugar-devel mailing list
> > [16]Sugar-devel at lists.sugarlabs.org
> > [17]http://lists.sugarlabs.org/listinfo/sugar-devel
>
> --
> James Cameron
> [18]http://quozl.netrek.org/
> _______________________________________________
> Sugar-devel mailing list
> [19]Sugar-devel at lists.sugarlabs.org
> [20]http://lists.sugarlabs.org/listinfo/sugar-devel
>
> References:
>
> [1] https://github.com/ManashRaja/sugar/commit/660985d2183416cd3ed758095e92adf82f87a10c
> [2] http://schoolserver:5000/
> [3] https://wiki.sugarlabs.org/go/Features/Multi_XS-server_registration
> [4] mailto:quozl at laptop.org
> [5] http://lists.sugarlabs.org/archive/sugar-devel/2016-April/thread.html
> [6] https://en.wikipedia.org/wiki/Google_Summer_of_Code#2016
> [7] https://wiki.sugarlabs.org/go/Summer_of_Code/2016#Sugar_on_the_Ground
> [8] https://github.com/sugarlabs/sugar/blob/master/src/jarabe/journal/journalactivity.py#L56
> [9] http://dev.laptop.org/ticket/9623
> [10] https://bugs.sugarlabs.org/ticket/1720
> [11] http://wiki.laptop.org/go/Release_notes/13.2.7#Fixes
> [12] https://help.sugarlabs.org/en/journal.html#journal-features
> [13] https://help.sugarlabs.org/en/journal.html#journal-detail-view
> [14] https://github.com/sugarlabs/sugar/blob/master/src/jarabe/desktop/schoolserver.py#L110
> [15] https://wiki.sugarlabs.org/go/Summer_of_Code/2016#Sugar_Core
> [16] mailto:Sugar-devel at lists.sugarlabs.org
> [17] http://lists.sugarlabs.org/listinfo/sugar-devel
> [18] http://quozl.netrek.org/
> [19] mailto:Sugar-devel at lists.sugarlabs.org
> [20] http://lists.sugarlabs.org/listinfo/sugar-devel
--
James Cameron
http://quozl.netrek.org/
More information about the Sugar-devel
mailing list