[Sugar-devel] Automated testing

[Daniel Narvaez]

Hello,

I made a lot progress in the last few days on automated testing, so I
thought I would give a quick update to the list. Hopefully this will
also provide an higher level description of the many patches I sent.
I'm working on a few cases which I consider paradigmatic but of course
we will need everyone to buy into it for this to really work. After
all, if you worked with a unit tested code base before you know it
takes less time to write code with tests than without!

Case 1. Non-UI component.

http://git.sugarlabs.org/~dnarvaez/sugar-toolkit-gtk3/dnarvaez/blobs/master/tests/test_bundlebuilder.py

Nothing special here. We are setting up a git repository with a sample
activity and running all the bundlebuilder commands on it, both
in-source and out-of-source.

Case 2. UI component, functional testing.

http://git.sugarlabs.org/~dnarvaez/sugar/dnarvaez/blobs/master/tests/test_activitieslist.py

This is more interesting. We are using at-spi (the gtk accessibility
framework) to get a tree of the UI. We can check if certain widgets
exists, click on them etc. At the moment it looks like for a palette
for the activities list palette for example

  [test_activitieslist.py | application]
    [ | frame]
      [ | filler]
        [ | filler]
          [ | filler]
            [ | panel]
              [ | icon]
          [ | panel]
            [ | filler]
              [mock | accelerator label]
              [ | label]
        [ | filler]
          [ | separator]
          [ | filler]
            [ | filler]
              [ | panel]
                [ | filler]
                  [ | filler]
                    [ | filler]
                      [ | icon]
                      [ | panel]
                        [Make favorite | label]
              [ | panel]
                [ | filler]
                  [ | filler]
                    [ | filler]
                      [ | icon]
                      [ | panel]
                        [Erase | label]
              [ | panel]
                [ | filler]
                  [ | filler]
                    [ | filler]
                      [ | icon]
                      [ | panel]
                        [Start new | label]
          [ | filler]


Case 3. UI component, visual testing.

No code to share here yet. The idea is to take a screenshot of the UI
and compare with it at testing time. It should be pretty easy to
update it when styles and sizes changes. I never experimented with it
yet, so I will have to play and see if it really works. I'm probably
going to use one of the sugar3.graphics widget to start with.

Case 4. Full UI, functional testing.

http://git.sugarlabs.org/sugar-build/sugar-build/blobs/master/tests/sugar/tree.py

In sugar-build we are starting all the activities from the activities
list view and the close them. Timing is tricky but I think we can make
it work reliably. Of course we will need to write more and more
complex tests.

Case 5. Full UI, visual testing.

I have not thought too much about this. I think a combination of
at-spi and screenshots could work. But I hear humitos is doing some
research on it :)

I will keep the list updated as I make progress on this.

-- 
Daniel Narvaez