[Sugar-devel] [Testing] Fedora Sugar Test Day - Test case content, location

[Kalpa Welivitigoda]

On Sun, Feb 19, 2012 at 4:28 AM, Samuel Greenfeld <greenfeld at laptop.org> wrote:
> There are a few things I think need to be clarified:
>
> What are we looking as a community to get out of this Fedora Test Day event?
> What is the purpose of a "test case" and/or "test plan"?  How detailed
> should they be?  Presuming we want them, how should they be stored in
> general, and for this particular scenario?
>
>
> For #1, I disagree that having users start out for this occasion by learning
> how to use Sugar on their own is appropriate for a few reasons:
>
> The amount of time that any given tester will be available to help us out
> likely is limited.   Time spent doing discovery will not be spent on other
> tasks.
> Not everyone will start at the same time.  Perhaps calling this a "Test Day"
> is a misnomer because there is no guarantee that testers will be in the time
> zone or country.
> Sugar is a relatively stable platform with a few known recurring UI
> disputes.  It is not Nell, the Helicopter experiment, or another one where
> the user interface could potentially require major changes.
>
> With this test day, my personal view is that we need to get feedback
> verifying basic Sugar and activity functionality in Fedora.  When Peter
> Robinson, Kalpa Welivitigoda, or someone else updates a Sugar software
> package in Fedora, these packages often go through the verification process
> without a single person commenting on if the proposed update worked or not.
>

It's true most of the time. Actually for all the activities I have
packaged for fedora only one got a comment in it's reviewing phase.
It's the same with the update phase. At the same time, I didn't get
any feedback from any of the users either.

> Mind you, usage feedback is appreciated; but is more of a secondary concern
> to me.  Fedora had "Fit and Finish" test days during Fedora 12 cycle where
> they asked for general usage feedback; perhaps we can propose that they do
> another round of those aimed at the various desktop environments with Fedora
> 18.
>
>
>
> For #2, I have used similar test templates to the the New Zealand's to
> verify activities in the past, and was thinking of making one available in
> this case.   Translating one into a Wiki template would make it
> straightforward to clarify which activities support sharing, webcam usage,
> etc.
>
> The reason I am interested in maintaining test cases with a system to keep a
> historical log of who did what when is because I want to be able to
> parallelize tasks.  Although I recognize I could be more efficient, there
> simply is too much material in Sugar and the XO platform for one person to
> focus on.  And yet I get regularly asked "do you remember bug #123" or "when
> was the last time someone looked at Q"?
>
> I am not looking for detailed test cases to the point of listing which
> buttons to click when; but rather simple ones like "Does it install?" and
> "Can it open a saved document?".
>
> Fedora takes a curious approach to this in that they write a series of test
> cases which could be parallelized, but then offer to have everyone run the
> same set of test cases.  And usually, pretty much everyone runs most of the
> available items.
>
> For comparison, look at the last GNOME 3 test day
> (http://fedoraproject.org/wiki/Test_Day:2011-04-21_GNOME3_Final) versus the
> last Sugar one (http://fedoraproject.org/wiki/Test_Day:2010-08-19_Sugar).
>
> I'm open to taking suggestions back to the Fedora Testing mailing list if
> someone has an idea on how we could do things better, but I'm trying to
> avoid cross-posting too much.   We could also inquire on the Fedora QA
> mailing list as to who might participate and what their skill levels are so
> we can better tune our approach.
>
>
>
>
> On Sat, Feb 18, 2012 at 5:04 AM, Tabitha Roder <tabitha at tabitha.net.nz>
> wrote:
>>
>> On 17 February 2012 08:36, Samuel Greenfeld <greenfeld at laptop.org> wrote:
>>>
>>> On March 22 there will be a Sugar test day for Fedora 17.  This means
>>> that the Fedora community in general will be gathering to look at Sugar and
>>> see what issues we have close to the end of the Sugar 0.96 cycle.
>>>
>>
>> While test cases can be useful, I always try to start with some discovery
>> time as this is when you can get some feedback on design and intuitive
>> behaviour (though this is impacted by use of other systems with many
>> users).  Something like:
>> "Find a friend. Work together to discover how to open the laptop if you
>> have an XO, or start Sugar. Together try clicking on things and see if you
>> can learn how to play any games or complete any activities. Can you find
>> ways to take photos, write stories, make music."
>> After that, get their feedback on how that went before giving them a test
>> case. First time users of Sugar can also give you feedback on their
>> experience of first use of an activity while following testing instructions.
>> There have been a number of occasions when I have said "oh, you have to
>> click on that first and then click on that other thing" and they have said
>> "why is it designed like that?" which really makes us rethink about the
>> design of activities.
>> Our basic activity testing template (written a long time ago) is here -
>> http://wiki.laptop.org/go/Activity_testing_template#The_NZ_activity_test
>>
>> On the topic of tracking testing, we have looked at a number of options
>> here in NZ and I think Australia also looked at a number of options. In NZ
>> we tried writing them on wiki.laptop.org but that didn't really work. My
>> personal method of managing test requests is to try to tag the requests (or
>> potential requests) in my email inbox when they arrive and then test them on
>> Saturday, archiving off emails as things are tested. This only works for us
>> because we meet in one place, not a suitable solution for multiple testing
>> locations. I personally don't think we should add anymore systems, but look
>> at ways to use existing systems to manage testing - such as the two bug
>> trackers we already use or the activities.sugarlabs.org site.
>>
>> Hope this helps
>> Tabitha
>
>
>
> _______________________________________________
> Sugar-devel mailing list
> Sugar-devel at lists.sugarlabs.org
> http://lists.sugarlabs.org/listinfo/sugar-devel
>



-- 
Best Regards,

Kalpa Pathum Welivitigoda
http://about.me/callkalpa