[Sugar-devel] Test protocol for "Resume" vs. "Start new"

[Wade Brainerd]

If possible, it might be helpful to record the students completing
this test, and then post the results to the wiki (as links to YouTube
etc).

We video tape focus testing sessions all the time in the video game
industry.   You just have to be careful about interpreting the
results, and it does take some time to go through watching them all.
But it reveals a lot of hesitations, failed attempts, etc. that don't
come out in a survey.

Anyway, either way I agree that this test would be helpful in
determining the best home view behavior.

Wade


On Mon, Jan 25, 2010 at 11:35 AM, Christian Marc Schmidt
<christianmarc at gmail.com> wrote:
> Thanks Gary, I'll have a look at this. I'm worried that conducting tests
> with novice users might be inconclusive, given that we are testing
> schoolchildren. But I am also not an educator. I'd be happy to work with
> someone with an educational background to put something together that might
> be suitable as a test for new users.
>
> For now, though, I'd suggest we stick to our plan to see where it gets us.
>
> Christian
> On Mon, Jan 25, 2010 at 11:20 AM, Gary C Martin <gary at garycmartin.com>
> wrote:
>>
>> Hi Christian,
>>
>> On 23 Jan 2010, at 15:01, Christian Marc Schmidt wrote:
>>
>> > Hi all:
>> >
>> > For our meeting at 10:30pm EST today, I put together a draft of a test
>> > protocol (in email body below). Please look it over and feel free to edit or
>> > add to. Thanks,
>> >
>> > Christian
>> >
>> > --
>> > Sugar Home View Concept Testing
>> > Draft 01/23/10
>> >
>> > Goal: Evaluate the Home view for "Resume activity" and "Start new
>> > activity" functionality.
>> > Protocol: Alternate between two builds, one per student: The current
>> > build [Build A], which makes "Resume" the default action, and a test build
>> > [Build B] making "Start new" the default action.
>> > Duration: 5 mins per student
>> > Student Profile: Should be a current user (not a first-time user). Age
>> > and grade should be recorded, along with duration of overall Sugar
>> > experience.
>> >
>> > 1. Begin at Home screen after boot up
>> >
>> > 2. Test cognitive model
>> >       • "Describe what you see in this view."
>> >       • "What are the symbols in the ring?"
>> >       • "Describe what each symbol in the ring is."
>> >       • "Do you know why the symbols are different colors?"
>> >       • "What activity did you last work on?"
>> > 3. Test pathways
>> > [Alternate between two pathways, one per student]
>> >
>> > Pathway 1:
>> >       • "Show how you would go back to the activity you last worked on.
>> > Describe what you are doing." [Student shows pathway to resume activity.]
>> >       • "Now, show what you would do if you wanted to start a new
>> > [Write/Draw/Browse, etc.; should be of the same kind] activity. Describe
>> > what you are doing." [Student shows pathway to start a new activity of the
>> > same kind.]
>> >       • "Now, show what you would do if you wanted to start a new
>> > [Write/Draw/Browse, etc.; should be a different activity than the current
>> > one] activity." [Student shows pathway to start a new activity different
>> > from the current activity.]
>> > Pathway 2:
>> >       • "Show how you would start a new activity. Describe what you are
>> > doing." [Student opens a new activity.]
>> >       • "Now, show what you would do if you wanted to go back to the
>> > activity you last worked on. Describe what you are doing." [Student shows
>> > pathway to resume a previous activity.]
>> >       • "Now, show what you would do if you wanted to start a new
>> > [Write/Draw/Browse, etc.; should be of the same kind] activity. Describe
>> > what you are doing."[Student shows pathway to start a new activity of the
>> > same kind.]
>> > 4. General questions
>> > "If you could change anything about Sugar, what would make it easier for
>> > you to go back to a previous activity?"
>> > "What would make it easier for you to start a new activity?"
>> > "Is there anything else about Sugar that you would change?"
>>
>> I know we've drifted from a usability study model, to a focus group model
>> for practical classroom reasons – but I though Jakob Nielson's latest
>> article was a timely piece worth a quick read through given we are going to
>> be getting feedback from existing Sugar users.
>>
>> "Summary: It's more difficult to conduct usability studies with
>> experienced users than with novices, and the improvements are usually
>> smaller. Still, improving expert performance is often worth the effort."
>>
>>        http://www.useit.com/alertbox/experienced-users.html
>>
>> Regards,
>> --Gary
>
>
> --
> anything at christianmarcschmidt.com
> 917/ 575 0013
>
> http://www.christianmarcschmidt.com
> http://www.linkedin.com/in/christianmarcschmidt
> http://twitter.com/cms_
>
> _______________________________________________
> Sugar-devel mailing list
> Sugar-devel at lists.sugarlabs.org
> http://lists.sugarlabs.org/listinfo/sugar-devel
>
>