[Sugar-devel] test automation

[Sascha Silbe]

Sridhar Dhanapalan <sridhar at laptop.org.au> writes:

> I am keen to explore ways to improve the quality and delivery time of
> code. Is there any work being done to automate testing of code?

Yes, I've been working on this whenever I have time. We already have
a data store test suite [3-5] and a semi-automated set of tests for the
WLAN part of our NetworkManager UI (Neighbourhood, Frame).

The remainder of this mail will focus on UI tests for Sugar (and
Activities). The public D-Bus API for Sugar is quite limited; the
datastore test suite already covers the important part. Most of the rest
should be replaced by other protocols that have since been agreed on by
other desktop environments. We could test the public (Python) API
provided by sugar-toolkit, but I haven't looked yet into that yet. There
may be some promising targets in the non-UI part of sugar-toolkit
(e.g. the Activity / Journal / Journal Entry Bundle related
code). However, automated UI tests would exercise at least some of the
more commonly used code paths in sugar-toolkit.

The basic framework for Sugar UI tests is available [6], but not of much
use with current upstream Sugar. There are two problems:

1. Ensuring the "accessibility" helpers are running.

   The test suite already enables them, but they need to actually get
   started. This is done automatically by the upstream version of
   gnome-session, but sugar-toolkit ships an ancient copy that won't
   start the helpers. Starting them manually was problematic. For this
   reason (in addition to generally being a good idea), I've ported
   Sugar over to upstream gnome-session. Upstream now provides [7] API
   to run alternative (i.e. non-Gnome) sessions. A WIP patch for using
   the new API is available [8] in my sugar clone on git.sl.o. The
   remaining blockers are the Shutdown and Reboot actions. Upstream
   gnome-session currently doesn't [9] provide API to trigger shutdown
   or reboot from outside of gnome-session, either without showing a
   dialog (for shutdown) or at all (for reboot). I've cleaned up a patch
   for this recently in the hope of pushing it forwards (the RFE was
   filed over three years ago), but upstream doesn't seem interested in
   it.

   The systemd support landed in sugar-toolkit doesn't help with this,
   BTW. It's used with our own copy of gnome-session and will be called
   after we already did the session part, closing all activities. With
   upstream gnome-session, we wouldn't get that chance (nor need to, if
   upstream just added the two small functions we need).


2. The Zoom Views (i.e. Home View, Group View, Neighbourhood) and the
   Journal are based on hippo-canvas, which doesn't support the
   GTK/Gnome accessibility framework. This means pretty much all of the
   core functionality isn't testable by this approach, either because it
   uses hippo-canvas or because it can only be triggered via some other
   part of the UI that uses hippo-canvas.

   This has been the major blocker for a long time. However, Simon
   recently ported sugar from hippo-canvas to plain GTK (thanks a lot,
   Simon!) as a milestone of the GTK3 migration and during the last
   Development Team meeting [10] we agreed on a plan that enables us to
   make use this intermediate result in upstream Sugar. I still haven't
   seen the cleaned-up patches, but Caspar and me talked about it
   extensively last weekend and we may simply accept the patches with
   only minor cleanups. We'd fully expect major bugs, but the hope is
   that a) the accumulated amount of manual testing (we're early in the
   development cycle for 0.98) and b) the ability to write automated
   tests would more than offset this risk. If we start right away, we
   might even have some simple test cases during the GTK3 migration,
   speeding up testing each individual change and thus the entire
   development phase. And once we have automated tests, we can more
   confidently clean up the current Zoom Views related code. The Views
   all pretty much provide the same functionality in terms of layout,
   but they're doing it in different ways. A single (set of) base
   class(es) used by all Zoom Views would help a lot to make the code
   more manageable and prevent Zoom View specific bugs. Like the
   GTK2→GTK3 migration, this is a major change. But with sufficient test
   coverage, the risk is low.


> We recently had some university students working with us to create an
> activity [1], and they were using the Robot Framework [2].

For activities, we're in a much better shape because virtually none of
them use hippo-canvas. With some glue, you may be able to start right
away - as your students apparently have done. I'd be quite interested in
their usage of the Robot Framework, as I've taken a look at it before
and it didn't seem to do much that would be useful enough in the context
of Sugar UI testing (or even any of the automated testing I do for my
customers). Neither AU#634 nor the NoteBoard source code gives any hint
about the automated testing done for this activity.


For implementing the test cases, taking a look at Strongwind [9] may be
a good idea. It's similar to dogtail, but apparently easier to use.

Just for completeness, I'd like to point out that there are alternatives
to the GTK/Gnome "accessibility" approach, e.g. XTEST based tools like
xautomation [10] or hooking into the Python side like SugarBot [11]
does. However, they come with a significant maintenance overhead: tests
implemented using low-level functionality need to be updated on many
minor UI changes and Python side hooking is likely to break during the
GTK2→GTK3 migration.

Sascha

> [1] https://dev.laptop.org.au/issues/634
> [2] https://code.google.com/p/robotframework/
[3] https://patchwork.sugarlabs.org/patch/708/
[4] https://patchwork.sugarlabs.org/patch/642/
[5] https://patchwork.sugarlabs.org/patch/640/
[6] https://people.sugarlabs.org/silbe/git/sugar-ui-tests.git
[7] https://bugzilla.gnome.org/show_bug.cgi?id=633276
[8] http://git.sugarlabs.org/~sascha_silbe/sugar/silbe/commits/gnome-session
[9] https://bugzilla.gnome.org/show_bug.cgi?id=575880
[10] http://meeting.sugarlabs.org/sugar-meeting/meetings/2012-06-26
[11] https://medsphere.org/community/project/strongwind
[12] http://hoopajoo.net/projects/xautomation.html
[13] https://wiki.sugarlabs.org/go/BugSquad/SugarBot
-- 
http://sascha.silbe.org/
http://www.infra-silbe.de/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: <http://lists.sugarlabs.org/archive/sugar-devel/attachments/20120702/c9c4d442/attachment.pgp>