[Sugar-devel] Killing activities when memory gets short

[Lucian Branescu]

2010/8/10 NoiseEHC <NoiseEHC at freemail.hu>:
>
>> We used to do that, the problem is that we don't control our platform
>> as Google controls Android and you need to make sure that resources
>> that need to be specific of each child process aren't shared (dbus and
>> X connections, etc).
>>
>> I'm personally more interested in reducing the amount of resources we
>> need to start activities (which is quite insane right now), but
>> sharing more of those resources sounds like a good idea to me.
>>
>
> Since I spent quite a bit of time analyzing the Python runtime here is my
> conclusion, maybe it will make wasting all that time less painful.
>
> First, most of the memory consumed by an activity is the process heap. While
> the Python runtime and native libraries are shared among processes, the heap
> is not. Even if you fork processes it will not work because reference
> counting will dirty the shared pages and my instinct tells me that just
> loading a Python source file will reference almost all the shared pages
> because they are mostly used to store already loaded modules. What I finally
> did not do is to actually check the hypothesis that most of the heap is
> filled by strings which are the identifiers in the loaded Python modules. My
> goal was to somehow make identifiers constants and just readonly-mmap them
> into the process (replace the pathetic .pyc loading mechanism). Note that my
> plan was just a little subset of the .jar -> .dex converter.
>
> Second, Python has a special memory manager which works with buckets so
> there are separate heaps for different object sizes. Somehow this special
> memory manager is turned off in release mode and it uses the standard C heap
> for some reason. Either it is a bug or just it turned out to be faster (more
> work was put into glibc) I do not know, but handling the linked free list
> can dirty shared pages as well or I am just mistaken...
>
> Third, the fact that Python is a dynamic language does not help any
> prefetching or memory sharing. I am not too convicted either that this
> dynamic nature is used at all in the Sugar codebase but you know I cannot
> program in Python and frankly I do not feel the need to learn another
> language. Just now, at my age of 34, I finally gave up and learned LISP
> (more specifically clojure) and I hope that it will be the last programming
> language I will have to learn (other than assembly languages of course)...
> :) Now this point is interesting because if you thought that the Dalvik VM
> could run the Sugar codebase via Jython then it just will not work. The
> Dalvik VM just cannot load .jar files and Jython just generates them on the
> fly because of the dynamic nature of Python.
>
> Fourth, moving Python (theoretically) to a GC environment (instead of
> reference counting) also would not work if the GC is compacting because it
> would also dirty the shared pages. So a mark and sweep nonmoving GC with
> separately stored "visited" bits is the only feasible solution. Now guess
> what the Dalvik VM does?
> For more info (45 min + questions):
> http://www.youtube.com/watch?v=ptjedOZEXPM
>
> So *my* conclusion is that solving this sharing problem is *very* hard. I am
> not sure that simply translating all activities from Python to Java would
> take more time.
>
> Another interesting thing is that meantime the unladen-swallow project
> progresses (just more slowly than planned). Their plan is to make it the
> default Python runtime so if it will happen (I cannot comment on that) then
> the Python VM will use even more memory (though it will be faster) so Sugar
> will be even less interesting on the myriad of low spec cheap ARM tablets
> which will flood the market soon.
>
> I think that is all I can say about this subject so I just finish it here.

The PyPy project has a fully-featured python 2.5 interpreter that has
much lower memory usage and a proper GC (so less dirty pages). They
also have an x86 JIT which makes it much faster than CPython, at the
cost of a bit of memory (still less than CPython). The only issue
right now is extension support: ctypes is fully supported, but
C/Python extension support is not complete by far.

As for Jython on Android, Jython has a Java bytecode JIT. It should be
entirely possible to write a dalvik backend to this JIT.

So not only would rewriting everything to Java be a huge step
backwards, but it would also be more work.