[sugar] [pymaemo] modifications to python and use on OLPC's XO

[Tomeu Vizoso]

[cc'ing sugar]

On Wed, Oct 29, 2008 at 6:39 PM, Anderson Lizardo
<anderson.lizardo at openbossa.org> wrote:
> Hi,
>
> On Wed, Oct 29, 2008 at 10:48 AM, Tomeu Vizoso <tomeu at tomeuvizoso.net> wrote:
>> Hi all,
>>
>> I'm a developer in the Sugar [0] project and we have some performance
>> issues in the OLPC's XO [1] that I think are similar to those in
>> Nokia's tablets because of using jffs2 (with autocompression enabled),
>> having limited memory and no swap, and having a slow processor.
>>
>> I'd like to know if PyMaemo has made significant changes to the stock
>> python codebase and share some experiences that may be useful in both
>> projects.
>
> AFAIK (by looking at the patches), the python package from pymaemo
> differs from stock python source only for bug fixes and a few patches
> that disable linking to some libraries not relevant to maemo, to save
> some space. We would be interested on patches that could improve
> performance though, feel free to share your findings on XO here!

Sure, you are as well invited to sugar at lists.laptop.org ;)

In general, raw python performance is not hurting us so much. By using
cProfile, we can see why some operation is being slow, and that's
generally because of something stupid on our side. Only in very rare
cases we have resorted to rewriting something in C.

I think we have two big issues with performance related to python:

- startup time: loading the interpreter takes some time, but the main
problem here is how python modules eagerly load all their dependencies
(imports at the top) and even worst, have the habit of performing
heavy initializations on the module scope. As an example, by making
lazy all re.compile() calls in modules, we can make apps start 0.5s
faster.

- memory usage: related to eager import, python processes take up much
more memory than they really need, and that memory is not shared
between processes. It may not matter so much in systems with 100s of
MB of swap, but we don't have that.

We have been able to improve on both issues by using a python launcher
similar to Maemo's pyLauncher. By preloading most dependencies and
forking a process for each app, we save a substantial part of the
initial startup time, and even best, the forked processes share a big
deal of memory between them (in the order of 3.5MB each).

The downside is that how this is implemented is a bit hacky, as the
environment where initial loading happens is not the same as the one
where the application will run later, so there need to happen some
reinitializations and some stuff breaks sometimes.

We would like to see how we could drop our hack and move to
pyLauncher, but there are some issues regarding how our security
framework (Rainbow) prepares the environment in which apps will run.

http://wiki.laptop.org/go/Rainbow

> Now regarding the packaging, one thing I remember is that pymaemo's
> python stores the most used modules in a zip file, thus allowing
> (supposedly) faster access because a directory lookup is not necessary
> during import. See [1] for a brief explanation. We haven't measured
> how much performance gain does this give to the Nokia tablets though,
> it is just something that has been there since forever...
>
> [1] http://mail.python.org/pipermail/python-dev/2005-November/057970.html

That sounds interesting, I'm going to look into it.

Thanks!

Tomeu