[Sugar-devel] Sugar-Server enhancement
Tony Anderson
tony_anderson at usa.net
Fri Apr 15 20:36:22 EDT 2016
On 04/16/2016 05:51 AM, James Cameron wrote:
> Looks good. A few points on first review;
>
> 1. you're changing "Register" and "Register again" to "Connect to
> server", but it isn't clear in the commit message why it is necessary;
Register has nothing to do with connect. The word 'connect' should not
be used.
>
> (In my opinion "Register again" does continue to convey the concept
> even if the server has never encountered the laptop before. Perhaps
> you focused on an implementation detail that is not important to the
> user?)
The intent of 'register again' is to notify the user that the laptop is
registered.
> 2. please avoid `ssh-keygen -R` if server is unchanged, because
> otherwise a rogue server can be introduced after registration,
This is the reason to keep the 'known_hosts' check in spite of the extra
work. The simplest solution is still:
rm -rf ~/.ssh/known_hosts
This creates an empty folder and the newly registered server becomes the
known host so no additional vulnerability is involved.
>
> (Would it be possible to change .ssh/config to add an entry for each
> server? If so, the known_hosts file might not need changing at all).
I assume you mean .ssh/config on the server. The list of known_hosts is
kept on the client in ~/.ssh/known_hosts.
This whole problem arises from calling each server, schoolserver. All
that is needed in a deployment with multiple servers
is to give each a unique name, eg. schoolserver1, schoolserver2, .... As
always, it is important to look at the use case.
In the use case of a teacher going from school to school, there should
be no problem. Registration is important to backup - but the
hypothetical teacher seems unlikely to want to backup to multiple
servers. The other concern might be the teacher's ability to collaborate
using
ejabberd. I don't know the current status of this, but it hasn't been a
problem, even at a deployment where essentially none to the laptops were
registered.
>
> 3. please check for `os.command()` failure, which could happen if the
> disk is full,
>
> 4. please add commit message references to the server-side changes
> and invite server-devel at lists.laptop.org people with your feature page
> and patch.
So far as I know, this patch does not require any change on the server
side.
Tony
>
> On Sat, Apr 16, 2016 at 02:48:59AM +0530, Manash Raja wrote:
>> Hi all,
>>
>> I fixed the bug. I haven't created a PR just yet. [1]https://github.com/
>> ManashRaja/sugar/commit/660985d2183416cd3ed758095e92adf82f87a10c
>> As of now I am parsing HTML data for json data to look for registered laptops
>> at an XS from the webaddress [2]http://schoolserver:5000 that serves a liist of
>> registered laptops.
>> I will modify it if there is another better method that can be implemented from
>> the Sugar side (we don't intend to modify the server side I guess).
>>
>> And here is the feature page as suggested by James. [3]https://
>> wiki.sugarlabs.org/go/Features/Multi_XS-server_registration
>>
>> Please have a look.
>>
>> I am very excited to be a part of the discussions that go into the making of
>> great features that can affect people down there in deployment. :) There is a
>> scope to do so much.
>>
>> Thanks
>>
>> On Sat, Apr 16, 2016 at 2:32 AM, James Cameron <[4]quozl at laptop.org> wrote:
>>
>> On Fri, Apr 15, 2016 at 04:02:39PM +0800, Tony Anderson wrote:
>> > Hi, James
>> >
>> > This thread was getting long so I replied only to the most recent
>> > communication. I am sure you have the full thread which shows the
>> > scope of the discussion.
>>
>> You're making it longer, yes, by hijacking it. You can find the full
>> thread here:
>>
>> [5]http://lists.sugarlabs.org/archive/sugar-devel/2016-April/thread.html
>>
>> > According to trac bug #362 was opened seven years ago against 0.82
>> > and last looked at three years ago. Several competent people looked
>> > at it and left comments. I see none that signify consensus.
>>
>> I take it that you don't want Manash to fix this bug, thanks.
>>
>> I'd like it fixed. I think there are others who want it fixed. It
>> would probably help with XS testing as well.
>>
>> > To have a design discussion, it is valuable to have a proposed design.
>> >
>> > I have tried to explain my proposal in detail. If there are
>> > questions, I would be happy to try to respond. Fixing the 'Journal
>> > is Full' dialog is a major help. However, what do you recommend to
>> > deployments when this happens?
>>
>> 1. upgrade to Sugar 0.108 (by RPM or images), or backport the patch
>> [#9623, #1720] into your custom builds,
>>
>> 2. transcode content to play in browser not Journal,
>>
>> 3. delete any activities that are not needed,
>>
>> 4. deploy Sugar Network to use the network activity cache,
>>
>> Also delete the Browse temporary directories, you reported this on
>> 19th January, it remains a problem, and you refused to test my fix, so
>> I lost interest very rapidly. [#4931]
>>
>> > The bottom line is that a reasonably active user is likely to need
>> > more room to store her Journal than is available on the XO.
>>
>> No, because it's no longer a significant problem. XO-1 are in the
>> minority and getting rarer as they die. Those that haven't died have
>> SD cards.
>>
>> > In the Journal code a filled star sets the 'keep' flag in the
>> > metadata. The cleared star clears the 'keep' flag in the metadata.
>> > Using this feature greatly simplifies the coding and the Journal
>> > view. As far as I know, the only use of this at the moment is to
>> > support the Portfolio activity.
>>
>> You are using an implementation detail in describing the flag. The
>> name given to the flag in documentation and user interface is
>> "favorite" (sic).
>>
>> > I think the detail view is inappropriate exactly as it would be to
>> > move the multiple selection checkbox there. These controls need to
>> > be immediately available.
>>
>> I disagree; they won't get used, and so it would be a waste of
>> valuable vertical space. The reason the checkbox won't get used is;
>>
>> - most laptops don't have a server,
>>
>> - an LRU algorithm can maintain the cache effectively.
>>
>> > The 'backup/sync' script is a good place to do check storage quotas
>> > because the script needs to touch the datastore on a regular basis.
>> > It has access to the amount of store in use and the LRU information.
>> > For example, if the user wants a document downloaded, the script
>> > knows its size and whether some other local copies need to be
>> > deleted to make room.
>>
>> I disagree. This script runs infrequently. The LRU must be
>> implemented inside the datastore for it to function properly.
>> Otherwise the lag between user action and response by the script would
>> be too long.
>>
>> > While an implementation detail, so far no change has been necessary
>> > to the datastore class.
>>
>> That's no reason not to change it.
>>
>> > Actually, since the 'keep' or favorite star sets the metadata,
>> > so far there has been no need to change the Journal.
>>
>> That's no reason not to change it.
>>
>> > "The multiple user feature is supported by Fedora and Sugar, but we
>> > removed it for OLPC OS."
>> >
>> > I think I am beginning to understand. OLPC OS is your generic name
>> > for the images to be installed on each model of the XO.
>>
>> Really, you are out of touch! OLPC OS is our name for the operating
>> system releases on the XO laptop. We've been using the term at OLPC
>> for a very long time, and use it in each release announcement.
>>
>> > I am deploying build 13.2.5 with Sugar 0.106 on all models.
>>
>> That's so sad. It was released in July 2015. It has the journal full
>> bug you mentioned. I'm not interested in supporting that release,
>> because I've already released two others. Upgrade.
>>
>> > So you are saying that we, users of Sugar or ' OLPC OS' could have a
>> > multiple user version of Sugar if 'you', as developers, didn't
>> > remove it.
>>
>> Well done.
>>
>> > As I understand it, you propose to generate unique serial-numbers
>> > per user.
>>
>> No. I was describing what happens _now_ during registration, by
>> reference to the code:
>>
>> 1. for XO laptops the serial number of the laptop is used,
>>
>> 2. for non-XO laptops a serial number is generated randomly,
>>
>> 3. there is no attempt to ensure the random serial number is
>> unique, but the width of the random string is sufficient to make it
>> unlikely,
>>
>> > So SSO would be guaranteed since no two users could have
>> > the same serial-number. This would certainly work and probably
>> > involve very little change to the existing code. What will be needed
>> > is a 'dns' to map serial-numbers to usernames.
>>
>> No, I wasn't proposing that. It's your idea. I don't think it's
>> guaranteed to be unique though.
>>
>> > Every school I have worked with keeps a careful record of students
>> > (often in paper ledgers). Currently I provide a name record in a
>> > Django database on the server (along with an XO inventory by serial
>> > number).
>>
>> Fail to see relevance. Not all schools will or can do this.
>>
>> > Agreed that determining which Journal objects need to be saved to
>> > the school server is not a difficult problem. However, datastore is
>> > a class so each user's datastore and the common datastore would be
>> > instances. So this seemed like a simple thing to implement.
>> >
>> > Actually, The deletion of Journal objects without an associated
>> > document works amazingly well. The number of objects in the Journal
>> > view goes from hundreds to only a few (often less than 20).
>> > Moreover, these 20 are the obviously interesting ones. Nothing is
>> > lost as the metadata is saved to the school server. It becomes much
>> > easier to 'reflect' when you are only looking at the documents you
>> > created. Meanwhile the myriad of objects can be subject to
>> > statistical analysis.
>> >
>> > For many activities, such as the Terminal, the document saved is
>> > actually 'state' information. This allows the Terminal activity to
>> > be restored with tabs and pwd. There are many game activities such
>> > as Memorize that also store state. It would seem more appropriate to
>> > save this state information in the metadata. For example, a json
>> > could be created in the metadata to hold state information. The
>> > script could keep these objects to enable the user to resume.
>>
>> Your concept of metadata is not of interest to me; journal objects
>> must continue reflect the learner's use of the laptop if the journal
>> is to meet the designed style of reflection.
>>
>> Unreflecting adults are not the target user.
>>
>> >
>> > Tony
>> >
>> > On 04/15/2016 01:36 PM, James Cameron wrote:
>> > >On Fri, Apr 15, 2016 at 10:25:59AM +0800, Tony Anderson wrote:
>> > >>Hi Manash
>> > >>
>> > >>The registration process is awkward but not the problem.
>> > >This is unfair scope creep. Manash began by asking about bug #362 and
>> > >has been working to fix that. Now you're asking him to consider a
>> > >much larger task; not a coding task, but a redesign of Sugar Journal
>> > >and Backup interaction. This is huge.
>> > >
>> > >And as far as I can tell, students aren't even accepted yet [1].
>> > >
>> > >1. [6]https://en.wikipedia.org/wiki/Google_Summer_of_Code#2016
>> > >
>> > >What you propose is from a set of tasks [2] you added to the Wiki,
>> > >which have not undergone any design review according to Sugar Labs
>> > >design practice and feature policy. I do not see any consensus on
>> > >these; we're yet to build a consensus.
>> > >
>> > >2. [7]https://wiki.sugarlabs.org/go/Summer_of_Code/2016#
>> Sugar_on_the_Ground
>> > >
>> > >Or, it looks like you're trying to make your own fork of Sugar, which
>> > >I'm fine with, it's open source after all, but to push that on others
>> > >without their input is wrong.
>> > >
>> > >If you proceed without consensus as a sole designer, then OLPC will
>> > >fork Sugar (as we already have so that XO-1s will go faster), and
>> > >you'll be making your own builds.
>> > >
>> > >>The problem is that rsync is used to create backups of the Journal
>> > >>and no effective means is offered to restore.
>> > >Agreed. We have no restore from server feature in Sugar 0.108, along
>> > >with no way to start a backup to server, and no selective restore.
>> > >
>> > >(We have backup to media, restore from media, but no selective
>> > >restore from media. Also, restore from media replaces Journal!)
>> > >
>> > >>However, the ultimate problem is thinking of the problem as one of
>> > >>backup. If you try to solve the wrong problem, often the result is a
>> > >>wasted effort.
>> > >>
>> > >>The Journal is single place where Sugar users save their documents.
>> > >>This is done by the Sugar activities when they close. The majority
>> > >>of XOs are still XO-1s with a 1GB store.
>> > >This point in your argument is void, because XO-1 are 45% of the XO
>> > >laptops manufactured so far. I have the numbers.
>> > >
>> > >Also, many XO-1 have been upgraded with an SD card.
>> > >
>> > >>If the available store is less than 50GB,
>> > >No, that's 50 MB, not 50 GB. See _SPACE_TRESHOLD (sic) in
>> > >sugar:src/jarabe/journal/journalactivity.py [3].
>> > >
>> > >3. [8]https://github.com/sugarlabs/sugar/blob/master/src/jarabe/journal
>> /journalactivity.py#L56
>> > >
>> > >>Sugar effectively shuts down.
>> > >This point in your argument is void, because this has been fixed [4,
>> > >5, 6], please upgrade to Sugar 0.108 which is in OLPC OS 13.2.7 [6].
>> > >
>> > >4. [9]http://dev.laptop.org/ticket/9623
>> > >5. [10]https://bugs.sugarlabs.org/ticket/1720
>> > >6. [11]http://wiki.laptop.org/go/Release_notes/13.2.7#Fixes
>> > >
>> > >>This typically results in the deployment reflashing the XO erasing
>> > >>all of the documents created by that user - a tragedy.
>> > >It was a known bug, so that's a training issue. You previously
>> > >proposed to train a teacher to use "rm -rf" to delete a known_hosts
>> > >file instead of Manash coding up an "ssh-keygen -R" command. It is
>> > >inconsistent to be able to do one and not the other.
>> > >
>> > >>What I am proposing is to use the school server as the primary store
>> > >>for the Journal with its effectively unlimited storage capacity. The
>> > >>ds_backup script needs to read the datastore uploading any new or
>> > >>modified documents. The local datastore can then be viewed as a
>> > >>cache for current working documents.
>> > >I'm favour of this ideal in principle, but it remains a huge design
>> > >and consensus challenge, not a coding challenge.
>> > >
>> > >However, with the XO-1, XO-1.5 and XO-1.75 using IEEE 802.11g the
>> > >local wireless network will collapse sooner due to this new load.
>> > >
>> > >>On the XO, the datastore is shown in the Journal. The 'keep' star
>> > >There's no such thing. There's a favorite star [7]. It has a defined
>> > >purpose. Are you proposing to destroy that purpose, or add another
>> > >column to the journal? There's even less room now that the multiple
>> > >selection checkbox was added.
>> > >
>> > >7. [12]https://help.sugarlabs.org/en/journal.html#journal-features
>> > >
>> > >>could be used to show whether there is a local copy of that document
>> > >>or not. If the document is not needed locally, the user can clear
>> > >>the star. In this case, the backup script could delete the local
>> > >>copy. If there is no local copy of the document, then the user could
>> > >>set the star. In this case the backup script could download the
>> > >>document.
>> > >My preference would be for the flag to be in the Journal detail
>> > >view [8], where there is available display space.
>> > >
>> > >8. [13]https://help.sugarlabs.org/en/journal.html#journal-detail-view
>> > >
>> > >>This capability could be used to set a quota on the amount of space
>> > >>used by the Journal. If the space is exceeded, the 'backup' script
>> > >>could delete local copies of document by LRU until the quota is met.
>> > >>Similarly, there should be a quota on Sugar activities which could
>> > >>also automatically be pruned back LRU. Managing the store
>> > >>automatically is consistent with keeping the Sugar UI as simple as
>> > >>possible.
>> > >This should be built into Sugar rather than in the non-Sugar backup
>> > >script. They should be maintained together.
>> > >
>> > >This would be a code change to git repository sugar-datastore and the
>> > >Journal activity in repository sugar.
>> > >
>> > >>As always, there are complications. The original OLPC concept was
>> > >>that there would be one XO per user. As a result the software was
>> > >>designed for a single user identified by the XO serial number.
>> > >The multiple user feature is supported by Fedora and Sugar, but we
>> > >removed it for OLPC OS.
>> > >
>> > >>Today, many XO deployments provide enough XOs for a classroom.
>> > >>During the day, different students use the XO as their class goes to
>> > >>the computer lab or as the computers are distributed from classroom
>> > >>to classroom. However, all of the documents created are in a single
>> > >>Journal with only the user's memory to indicate which document goes
>> > >>with which user.
>> > >OLPC did not design OLPC OS to be used in this scenario, so no
>> > >surprise you've hit that. But it's not a Sugar problem. Don't
>> > >conflate Sugar with OLPC OS.
>> > >
>> > >>The OLPC Ubuntu Sugar 14.04 Trusty LTS (to use its official name)
>> > >>solves this problem at the laptop side by using standard gnu/linux
>> > >>logins.
>> > >The multiple user feature is supported by Ubuntu and Sugar, and I
>> > >haven't removed it yet. I know how to; small configuration change to
>> > >lightdm package.
>> > >
>> > >Don't forget SoaS. The Fedora 23 SoaS is easily installed to disk and
>> > >has multiple user capability. The Fedora 24 SoaS is shaping up to be
>> > >just as good or better, since it is based on Sugar 0.108.
>> > >
>> > >>Each user has her own username and password. The Sugar activities
>> > >>have been moved to common space in the file system so only one copy
>> > >>is needed to support multiple users. Users are not 'olpc' but
>> > >>identified by their username. However, the datastore is part of the
>> > >>user space (one datastore per user).
>> > >Yes. ODPU.
>> > >
>> > >>This is problematic since the backup script uploads to
>> > >>/library/user/serial-number on the school server.
>> > >No, you're wrong. In the Ubuntu scenario, the register_laptop
>> > >function will invent a serial number because it won't find Open
>> > >Firmware [1]. So it wouldn't be a problem. It doesn't sound like
>> > >you've tested this.
>> > >
>> > >1. [14]https://github.com/sugarlabs/sugar/blob/master/src/jarabe/
>> desktop/schoolserver.py#L110
>> > >
>> > >>So, one strategy would be to upload to /library/user/username. This
>> > >>requires that usernames be unique across all laptops using a given
>> > >>schoolserver. This could be enforced at registration on the school
>> > >>server.
>> > >Starting to sound very complicated. Single-sign-on (SSO) across a
>> > >school. These are truly amazing teachers with lots of free
>> > >administration time.
>> > >
>> > >(There are deployments using Sugar with SSO already, but as it's
>> > >outside the scope of Sugar we don't hear about them at Sugar Labs, and
>> > >we don't provide the facility in OLPC OS, but that doesn't stop them.)
>> > >
>> > >>However, the Sugar releases for the XO
>> > >We call that OLPC OS, which includes Sugar and Gnome desktops.
>> > >
>> > >>still maintains Sugar activities in /home/olpc/Activities. So, one
>> > >>requirement is to restructure Sugar as was done for OLPC Ubuntu
>> > >>Sugar 14.04 Trusty LTS.
>> > >That would not block implementing a server datastore, since the
>> > >implementation would not care what $HOME is set to.
>> > >
>> > >(And besides, it's already done for SoaS, so the Fedora activity
>> > >packages can be used immediately.)
>> > >
>> > >>Another approach might be to create directories for each user of a
>> > >>single XO (e.g. /library/user/serial-number/user1).
>> > >That would require authentication service by the server datastore.
>> > >
>> > >>Another complication is that the Browse activity downloads files
>> > >>from the school server to the Journal (e.g. pdfs, mp3). These
>> > >>documents do not need to be saved to the users Journal backup on the
>> > >>school server since they can be restored from the school server
>> > >>'library'. Also, such documents when downloaded should be stored in
>> > >>a common space available to all users of that laptop. Fortunately,
>> > >>the source of a document is provided in the metadata.
>> > >What you describe here can also be solved by deduplication.
>> > >
>> > >The Journal Git backend proposed by Martin and Walter could help with
>> > >deduplication of journal objects across multiple journals.
>> > >
>> > >[15]https://wiki.sugarlabs.org/go/Summer_of_Code/2016#Sugar_Core
>> > >
>> > >>One approach would be to divide the datastore into two directories
>> > >>on the laptop, one in common space and the other local to the user.
>> > >>The Journal could show both sets of objects.
>> > >Or the server datastore would recognise content hashes of server
>> > >artefacts and know it need not send the content from the client to the
>> > >server before LRU local deletion. It could hard link it.
>> > >
>> > >>Finally, each Journal object consists of a metadata file and an
>> > >>optional document. The metadata files tend to clutter the Journal
>> > >>display (mine has hundreds of Terminal activity and Log activity
>> > >>entries). I would propose that the Journal show only objects which
>> > >>have a document with a user-supplied name (a metadata flag). The
>> > >>script should backup the metadata files for those objects without a
>> > >>document to a 'log' on the school server for statistical analysis
>> > >>but delete them from the local datastore. Journal objects saved
>> > >>without a user-supplied name (but something like Write.activity)
>> > >>should have their document deleted. As part of GSOC there is an
>> > >>initiative to require users to supply a name for documents they wish
>> > >>to save - so this problem may not be part of the 'backup' scheme.
>> > >>Whether a document is saved or deleted, the metadata can be saved to
>> > >>the log and displayed by the existing statistical tools.
>> > >I'm against any classification of journal objects in this way. We
>> > >cannot know how useful a Terminal and Log activity object is to the
>> > >learner.
>> > >
>> > >However, I would like a way for expert users to terminate an activity
>> > >without saving a journal object.
>> > >
>> > >>As an old crumudgeon, I still believe design precedes coding.
>> > >>
>> > >>Reading the existing code is always a good idea:
>> > >>
>> > >>Sugar
>> > >>
>> > >> *
>> > >>/usr/lib/python2.7/site-packages/jarabe/desktop/schoolserver.py
>> > >>#registers server - notice transition from gconf to gsettings
>> > >> * /usr/bin/ds_backup.sh #primarily decides if backup can be run
>> > >> #backup logic is needed
>> > >>because an rsync can use a lot of bandwidth in a local network
>> > >> * /usr/bin/ds_backup.py #actually does the backup using rsync
>> > >>(note: -d option AFAIK deletes an object from the backup if it is
>> > >>deleted in the source,
>> > >> #this has the effect of
>> > >>limiting the size of the datastore to the available space on the XO
>> > >>not on the school server).
>> > >>
>> > >>Server (xsce6)
>> > >>
>> > >> * /usr/libexec/idmgr #contains a number of utilities
>> > >>used in registration
>> > >> * /library/users #contains a directory per
>> > >>serial-number of registered user
>> > >> #use ls -a to see files
>> > >>created. The idmgr creates a public/private key pair which is used
>> > >>by sftp to authenticate - avoiding password
>> > >>
>> > >>Note: if you look at the server code, you can see why registering
>> > >>the laptop on each connection works (and can avoid any need for a
>> > >>registration menu item).
>> > >>
>> > >>When you get to know your way around the existing process, I'll send
>> > >>you a copy of the ds_backup.py code I use to implement the item by
>> > >>item backup.
>> > >You should start using GitHub like the rest of us.
>> > >
>> >
>> > _______________________________________________
>> > Sugar-devel mailing list
>> > [16]Sugar-devel at lists.sugarlabs.org
>> > [17]http://lists.sugarlabs.org/listinfo/sugar-devel
>>
>> --
>> James Cameron
>> [18]http://quozl.netrek.org/
>> _______________________________________________
>> Sugar-devel mailing list
>> [19]Sugar-devel at lists.sugarlabs.org
>> [20]http://lists.sugarlabs.org/listinfo/sugar-devel
>>
>> References:
>>
>> [1] https://github.com/ManashRaja/sugar/commit/660985d2183416cd3ed758095e92adf82f87a10c
>> [2] http://schoolserver:5000/
>> [3] https://wiki.sugarlabs.org/go/Features/Multi_XS-server_registration
>> [4] mailto:quozl at laptop.org
>> [5] http://lists.sugarlabs.org/archive/sugar-devel/2016-April/thread.html
>> [6] https://en.wikipedia.org/wiki/Google_Summer_of_Code#2016
>> [7] https://wiki.sugarlabs.org/go/Summer_of_Code/2016#Sugar_on_the_Ground
>> [8] https://github.com/sugarlabs/sugar/blob/master/src/jarabe/journal/journalactivity.py#L56
>> [9] http://dev.laptop.org/ticket/9623
>> [10] https://bugs.sugarlabs.org/ticket/1720
>> [11] http://wiki.laptop.org/go/Release_notes/13.2.7#Fixes
>> [12] https://help.sugarlabs.org/en/journal.html#journal-features
>> [13] https://help.sugarlabs.org/en/journal.html#journal-detail-view
>> [14] https://github.com/sugarlabs/sugar/blob/master/src/jarabe/desktop/schoolserver.py#L110
>> [15] https://wiki.sugarlabs.org/go/Summer_of_Code/2016#Sugar_Core
>> [16] mailto:Sugar-devel at lists.sugarlabs.org
>> [17] http://lists.sugarlabs.org/listinfo/sugar-devel
>> [18] http://quozl.netrek.org/
>> [19] mailto:Sugar-devel at lists.sugarlabs.org
>> [20] http://lists.sugarlabs.org/listinfo/sugar-devel
More information about the Sugar-devel
mailing list