[IAEP] Testing meeting follow up

[Marco Pesenti Gritti]

Decisions:

* Neither Sugar Labs, OLPC or any other distribution can take
responsibility over *all* Activity testing. Only sanity checking (and
needed testing) on the subset they've decided to be responsible for.
We should work together to encourage Activity maintainers to test
their own Activities, provide them the necessary tools, policies and
information.
* During these coordination meetings and a few email threads, we made
huge progress on upstream/downstream responsibility division on
testing. We are all comfortable to move ahead with the plan now. Yay!

Actions:

* Marco to run point on "Activity testing infrastructure" at Sugar Labs.
* Marco to keep championing activity infrastructure and maintenance
work flows at Sugar Labs.
* Marco to schedule a meeting about automated testing, probably in the
context of development team meetings.
* Mel to communicate with OLPC management and internal testing group
about the outcome of these meetings.
* Mel to sleep at night. She was falling asleep in the middle of the
conversation and almost derailed Marco plans to finish in one hour.

Lessons learned:

* Always write your minutes just after the meeting, or you will end up
wasting time to reread the whole log. (if you have a flaky memory like
mine, at least).
* Make sure that everyone take actions, Simon has apparently been lazy
this time!

Thanks:

* To Simon and Mel for the entertaining and productive conversation.
I'm excited that we removed pretty much all the road blockers. There
will be more coordination to do, I'm sure, but we can start running
now!

Full log:

<marcopg> ok I guess we can start the next meeting now
 I'm supposed to run it, but I don't know how to run meetings
 but it's only a few of us, so it won't be too bad!
 so activities
 the question last time was... who tests activities?
 marcopg mchua_ mungewell1 meeting m_stone mtd morgs
 mchua_: is the olpc community testing group going to do it?
<mchua_> "Not OLPC," says the Mel. "Only when we want to verify
Activities we're shipping on the default image for XOs," says the Mel.
<marcopg> mchua_: OLPC as org or as community too?
<mchua_> marcopg: as org, and as far as the org will support community
testing, at least for now.
 marcopg: there's a reason I started off community testers with
g1g1-2008 activities - it's a really easy way to argue that this,
clearly, is something OLPC Should Have Tested, as we Ship It.
<marcopg> mchua_: so OLPC strategy is to rely on upstream worth
activities quality?
 mchua_: that's what I think too, if you ship something you need to test it
 from the upstream point of view
<mchua_> marcopg: Well, it's tricky. We definitely don't *support*
Activities, even the few we ship.
<marcopg> mchua_: I understand, but I've not been able to make sense
out of that yet
<mchua_> marcopg: and with limited resources, I would prioritize OLPC
testing towards XO-specific code (like our build, say.)
 marcopg: hm, then I should try to clarify... is there any particular
point you had questions on?
 marcopg: or should I start trying to explain my current understanding of it?
<marcopg> mchua_: prioritizing make sense clearly... but I don't see
how you can not support something you ship
 I guess what I'm unclear on is
 how does it make sense to ship a product, and *not* support a very
relevant part of it?
 think to the extreme case
 if no activities would work
 what good would it be if the OS/core Sugar was perfect?
<mchua_> Oh, I agree that without Activities, a perfect ship of
Sugar/XO-OS would be useless.
<marcopg> mchua_: but you disagree that OLPC should ensure that
Activities are working well?
* erikos can not kick the bot - just tried
<mchua_> Maybe a similar case would be... I'm not sure how Red Hat
support works, but if someone writes $random_third_party_application,
Red Hat doesn't necessarily support it automagically, right?
<marcopg> mchua_: totally agreed about random third parties
 mchua_: but making every activity a random third party seems to push it too far
<mchua_> marcopg: I disagree that it is the responsibility of OLPC to
ensure all Sugar Activities are working well. I agree it is the
responsibility of OLPC - a responsibility towards its customers - to
ensure a small subset of Sugar Activities they're likely to care about
are working well.
<marcopg> mchua_: then we completely agree :)
<mchua_> marcopg: I think that this responsibility can best be carried
out by making sure that a non-OLPC test/dev group for that Activity
(which may include OLPC employees) makes sure that Activity works
well,
<marcopg> so OLPC, through his community or not, should make sure that
this small subset is tested, right?
<mchua_> since it's likely for Activities we ship (and therefore have
a responsibility to sanity-check) to change from build to biuld.
<mchua_> marcopg: Yep.
--- andresambrois is now known as aa
<mchua_> marcopg: Kind of like how OLPC QA has a vested interest in
having SL's BugSquad triage awesomely, even though SL's BugSquad is
Not Us. (In fact, we have a huge incentive to keep the two separate.)
<marcopg> mchua_: make sure that the activity works well on the XO
 (I guess)
 so this is still some kind of downstream testing, prolly
 mchua_: did you already start to work on this? or is it just a plan?
<mchua_> marcopg: Yep. But OLPC QA won't really care if the Activity
works well in, say, Debian. And probably can't justify resources spent
to do that. Just on the stuff we ship, and on whether it works on the
XO.
<marcopg> mchua_: sure, that's why I call it downstream testing
<mchua_> marcopg: Well, the G1G1-2008 Activity testing is my first
attempt at that limited sphere of responsibility (and it's far from
perfect, but better than the almost-nothing we had before.)
<marcopg> mchua_: my feeling is that this kind of testing should *not*
go through SL basically
<mchua_> marcopg: I don't want that to be interpreted as "OLPC will do
all Activity testing, ever!" though - and want to work with SL to find
ways to encourage Other People to do it.
<mchua_> it'll make both of our lives easier.
<mchua_> marcopg: I agree it shouldn't be a SL thing either.
<marcopg> mchua_: and that from the upstream point of view it should
probably work just like core testing
 i.e. we get and triage the bug reports
 (and fix them hopefully!)
<erikos> agreed here
<marcopg> OK
 this is really cool
 mchua_: thanks a lot, I was really struggling to understand olpc
position on activities!
<mchua_> marcopg: well, wait... is Activity development (making sure
Activity bugs get fixed) the responsibility of core-sugar devs?
<mchua_> basically, "Who's Responsible For Activities?"
<marcopg> mchua_: nope, of the activity upstreams
 well
 not exactly
 so
<mchua_> Is it SL? The Activity maintainer (as an independent project?)
<marcopg> we have a group of activities
 which are part of the SL release process
<mchua_> It's definitely not OLPC, except for the "we need to check
what we ship" bit.
* mchua_ listens
<marcopg> the group is called Fructose
 I'm not completely sure it's the right setup
 and I'm willing to question that at some point
 but, as long as, things are setup this way
 I think SL as a community is responsible for the Fructose activities
 and only for those
<erikos> development and triaging wise
<mchua_> marcopg: in the same way OLPC is responsible for the
Activities we ship in signed images?
<marcopg> for the other activities we basically provide services
<erikos> mchua_: yup
<marcopg> mchua_: not quite
 mchua_: in the same way we are responsible for the core
<erikos> mchua_: they are officially part of the release
<marcopg> SL doesn't have customers, so we are not responsible in the
same sense as olpc
<erikos> oh right
<marcopg> we don't do support etc
 mchua_: is it clear how it's supposed to work? And I'd also like your
opinion about how good is this setup, or how you'd see it work
 (we can't change it for 0.84 at this point, but we might reconsider
down the way)
 my feeling lately is that this setup is too inflexible
 different deployments might want pretty different set of activities
<erikos> marcopg: yeah - but we have only a very small core
 marcopg: and sugar without that core is nothing
<marcopg> erikos: yup, but it would grow ideally
<erikos> marcopg: the core?
<mchua_> marcopg: but so would the maintenance you have to do
<mchua_> (you == SL)
<marcopg> mchua_: sure and that's part of the reason I'm starting to
dislike it ;)
<mchua_> So, as I understand it, what other open-source projects have
done to solve this problem is to market it as "You've got to make your
Activity good enough to include in our Release of Awesome."
<marcopg> we might spend time maintaining things which would not be
generally useful
<mchua_> (which includes stuff like testing and docs.)
<marcopg> mchua_: very correct
 and also
 the application should be somewhat generally useful
 and obviously not duplicate anything else in the core
 s/core/release
<mchua_> Being chosen for Fructose, or the G1G1 image, is an *honor.*
It's an honor we don't have as much transparency on (as in, "I wish my
Activity could be included, what do I have to do?" does not exist.)
<marcopg> mchua_: it does exist :)
 perhaps not well communicated though!
<mchua_> marcopg: hm, I am missing many things then :) where do I find this?
<marcopg> somewhere on the wiki
 :P
 (re not well communicated...)
 let me see
 mchua_: http://sugarlabs.org/go/DevelopmentTeam/Release#New_modules_proposal
 mchua_: I mailed the list about it a couple of times
 and we had a few proposals about it
 but well, yeah, the documentation is very poor etc
 mchua_: the fact that you had no idea about it, is obviously an
indication that something is broken communication wise :(
<mchua_> marcopg: oh! I mostly meant for OLPC (Greg Smith has done an
admirable job of starting to collect proposals and make that process
more transparent for 8.2.0, though).
<marcopg> mchua_: oh I thought you was referring to SL. SL sucks at
that anyway :) Greg has been doing better!
<mchua_> marcopg: but, yeah, I think it applies to both SL and OLPC
(although in slightly different degrees/ways).
<mchua_> <3 greg smith. I don't know what we'd do without him.
<marcopg> so the consensus here is that we should keep Fructose but be
careful to not expand it too much?
 that's at least my understanding of what erikos and mchua_ was saying
<erikos> sounds good with me
<erikos> marcopg: as well - activities can be dropped or replaced in my opinion
<marcopg> erikos: sure
 (though in practice removal is sort of difficult)
<mchua_> from the perspective of OLPC, Fructose probably doesn't
matter as much (only inasmuch as it overlaps with what we've chosen to
ship, really. ;)
 but that sounds good to me.
 back to the original question of "who tests Activities?" - it sounds
like neither SL nor OLPC can take responsibility over all Activity
testing, only sanity checking (and needed testing) on the subset we've
decided to be responsible for.
 so my question is what we can do to work together to encourage
Activity maintainers to test their own Activities, to ease the load on
both of us as "downstream."
<marcopg> mchua_: the idea is that it will tend to overlap a lot with
what you decided to ship, otherwise Fructose is going to be pretty
useless
<mchua_> ...until OLPC doesn't make up nearly all of SL's user base, but yeah.
<marcopg> yeah that's a good question
 activity testing sprints sounds like a great thing, independently
from who organize them
<erikos> yup testing sprints sounds great
<marcopg> stuff like SoaS will also get in quite a bit of testing on
activities hopefully
<mchua_> I think Activity testing infrastructure helps too; that's
what I've been trying to set up from the OLPC side - I hope that what
we've come up with is migrateable.
<marcopg> maybe activity testing sprints should be co-organized by SL and OLPC
 (and by SoaS etc)
<mchua_> marcopg and erikos, can SL host the Activity development and
testing resources?
<marcopg> a bunch of people testing the same activities on a bunch of
different distros
 lots of fun!
 mchua_: I think so
<mchua_> like their trac instances, and where we report bugs on them,
and gitorious and such for them? (I know some of this is already in
the works.)
<erikos> mchua_: development ressources we do host
<marcopg> mchua_: hosting at SL sounds like a good idea
<erikos> mchua_: we just moved git
<mchua_> Excellent. How can I move the current Activity testing
resources upstream, then?
 Both in terms of infrastructure, but also process and the people who
are doing the testing?
<marcopg> can you enumerate the infra you need to move?
 or I can do it, if you like
* mchua_ is trying to steer helpful resources in the SL-hat direction,
and trying to make it as easy as possible to take and clone (and then
we'll delete it here)
<mchua_> marcopg: the move should take place after Dec. 25, when our
current Activity test sprint ends. But basically,
http://wiki.laptop.org/go/G1G1_Activity_testing and all the resources
that it links to.
<mchua_> (or things that serve the same purpose.)
<marcopg> mchua_: wiki pages only?
 I'm planning to help moving dev resources here
 git and trac compoennts
<mchua_> marcopg: also a trac infrastructure for reporting bugs, but
it sounds like you already have that down
 marcopg: maybe a more coherent way of rephrasing it would be
<mchua_> "if SL is hosting Activity development/testing as a service
to the Activity development/user community, there are some things I
think it should provide"
<marcopg> mchua_: things == infrastructure?
<mchua_> marcopg: yeah
<mchua_> Some are things that are also dev resources, like code
hosting (and an easy way for testers to download/install both
latest-stable and bleeding-edge packages) and Trac (and procedures for
reporting bugs, yay BugSquad!)
<marcopg> oh I'm pretty sure we can get infra team to give us what we need :)
<mchua_> Some are things that are a little bit more test specific,
like... *searches brain for list*
<marcopg> mchua_: I guess my question is how do we want to go about this then
<mchua_> ...a way for each Activity to keep track of its own test
cases, test plans, and test results. And a way for the test community
(SL's BugSquad, but also downstream projects) to search for and look
at information about Activity test results across multiple Activities.
<marcopg> i.e. how do we get to the infra team stuff we require
<mchua_> And a way of contacting maintainers so it's easier to
escalate any issues.
<erikos> hmm
<mchua_> marcopg: Yeah... if this sounds like the arrangement we want
to have, I'm super-happy to work out the tickets that we need to file
for the infrastructure team, but I'm not sure if this is the
arrangement that you want, too.
<erikos> the bug communication happen over trac
<marcopg> mchua_: I like it. Can you think of any reason I'd not want it? :)
<mchua_> erikos: Yep. (Another reason to figure out the
upstream/downstream magic trac plugin - chalk a +1 to my
motivation-to-work-on-that list.)
<erikos> testing plans - would be good to have upstream i agree
 mchua_: ;p
<marcopg> erikos: and activity user/devel info
<mchua_> marcopg: more stuff to think about and maintain (as
infrastructure), mostly.
<erikos> marcopg: yeah we just need to enhance the Modules page
<marcopg> well, activities are my nightmare anyway
<mchua_> marcopg: the setup costs are high. But I believe they'll be a
well-placed investment (and eventually, a necessary one.)
<marcopg> it's the biggest blocker for SL atm
 (ask caroline!)
<mchua_> marcopg: how so?
<marcopg> it's a pain to get them into distributions because they are
poorly maintained...
* erikos is interested as well
<marcopg> it's not even clear how to contact the maintainers
<erikos> oh yeah :/
<marcopg> maintainers don't know how to package stuff properly
 etc
 SL needs to start to work with activity maintainers
 at all levels
 packaging, testing, user documentation
* tomeu1 adds a Activities category to amo
<marcopg> which doesn't mean we should do the work
<erikos> ok i think - lets fix the glucose one first
<marcopg> but provide the infra, the policies, the info that activity
authors will use to do a better work
<erikos> marcopg: we could do a sprint as well
<marcopg> and I think the idea of upstreaming a bunch of activity
related infra pieces, fits perfectly into this
<erikos> once we have the basic infrastructure up
<mchua_> From the OLPC side, I'm going to try to push Activity testing
resources upstream, and help the SL-hosted infrastructure get started.
That'll help us concentrate on 8.2.1 and 9.1.
 Is someone going to run point on "Activity testing infrastructure?"
<marcopg> "run point"?
 my english sucks
<mchua_> oh! sorry, "take as an ongoing action item"
<marcopg> (great about the OLPC side)
 mchua_: I can take it
 I want to have some discussion about it in tomorrow dev meeting
<mchua_> marcopg: I'm not sure how common the phrase "be the point
person for this" is
<marcopg> mchua_: oh that I understand, it's the "run" thing which
confused me ;)
<mchua_> marcopg: Thanks. I'm too hosed with CTW and 8.2.1 to really
concentrate on Activity testing beyond our G1G1-2008 run right now,
but can probably jump back in more helpfully in January.
<marcopg> oh wait Activit testing infra :)
 damn, I took the action without knowing what it was exactly :P
<mchua_> marcopg: So yeah, please, please do ping me on that if you
need a hand after... say... Jan. 5. I should have more of a life then.
<marcopg> mchua_: OK, I will take it
 I suspect I'll have to bug you a lot, though :)
<mchua_> marcopg: (If you're too hosed and need to back out, that's
okay. I understand. We /can/ come back to this right before FUDCON.)
<marcopg> mchua_: I will do my best :)
<mchua_> marcopg: That's not a problem at all - I just need someone
else to take responsibility for the remembering and
pushing-this-forward right now, because I'm staggering under too many
things to keep track of right now. :)
 marcopg: thanks, Marco.
<marcopg> time running out!
* marcopg think mchua_ is trying to make me a 2 h meetin got make fun of me
<marcopg> (can't type)
* mchua_ didn't know we set a time limit for this one :)
<mchua_> I'll stay as long as it takes.
<marcopg> ok it seem like we have a plan on activities, yay!
<mchua_> (...well, if it runs over 4 hours, I might have to get
dinner, but... yeah.)
* marcopg wonder if mchua_ doesn't remember what I said about time of
this meeting or she is just making fun of me
<mchua_> marcopg: yeah, I'm feeling *much* better about Activity
testing now, thanks. And I'll make sure this gets communicated to the
OLPC testing group and community.
 marcopg: oh, are we doing an hour? ah! sorry!
* mchua_ has... a brain of great scatteredness today
<erikos> marcopg: other points?
<marcopg> mchua_: hehe I had said this one was going to be *exactly*
one hour, thought you was kidding me ;)
 yeah OLPC
* marcopg leaves the floor to mchua_ about this
<marcopg> we discussed a part of that already with activities I guess
 mchua_: I guess my question is... how do you think what we discussed
in this two meetings will be in regard to OLPC
 mchua_: to OLPC org in particular
 mchua_: it's our first customer, so we have to care about it :)
 mchua_: kim in particular seemed to be worried about
upstream/downstream separation of the bug tracker
 mchua_: do you think we are clear enough on how to coordinate, to be
able to reassure them?
 mchua_: anything you would like us to do, to make OLPC life easier
 erikos: these OLPC people are evil, see now she runned away so that I
can't close my meeting in time!
* mchua_ is here! really! just thinking...
<marcopg> mchua_: :))
<erikos> :)
<marcopg> mchua_: sleep helps to think faster :P
<mchua_> do you know what kim's concerns were, and/or would you like
me to talk with her about those?
<mchua_> marcopg: :P
 marcopg: ...says the person who was also up 'till 4.
<marcopg> (you was up much later!)
 erikos: do you remember?
 we had a thread on techteam
<mchua_> As far as I'm concerned, I'm completely confident that we'll
be able to coordinate upstream/downstream stuff in whatever way we
need.
 But I'm only one individual.
<erikos> marcopg: that was the past
<marcopg> mchua_: perhaps the best would be to try and talk with Kim
<erikos> if there are no current issues - lets just forget about it
<marcopg> mchua_: now we have a better plan, we didn't when we first
discussed it
<erikos> right
<mchua_> And I certainly can't speak for all of OLPC, or even all of
OLPC's QA team... I do currently liason between the OLPC test
community and OLPC as part of my job responsibilities, so "officially"
that's what I can speak as.
<marcopg> I propose that we go ahead with plan. And if/when mchua_ has
time she sync a bit OLPC management about it
<erikos> sounds good to me
<mchua_> erikos, marcopg, I think you're right; we don't have any
current issues, we just resolved the *huge* one of Activity testing to
the satisfaction of everyone here, we just need to propagate that
"yay!" feeling out to SL and OLPC communities (and OLPC-HQ.)
* cjb waves.
<mchua_> marcopg proposal + 1
<erikos> i have learned - you have to act first
<mchua_> ask forgiveness, not permissoin!
<cjb> mchua_: that sounds right
<marcopg> ok, fanstastic
 next point!
<mchua_> s/permissoin/permission
<marcopg> 4 mins left
* mchua_ waves at cjb
<erikos> if someone opposes he will do it - if not you were not blocked
<mchua_> automation!
<marcopg> hello cjb
<erikos> hello cjb
<marcopg> ok so my agenda for automation would be to figure out how we
go forward
<mchua_> I have no bandwidth for it! cjb proposed we send a call to
the community on what automation we need!
<marcopg> which team/people start looking into it etc
<cjb> hi all.  don't let me distract your last four minutes.  :)
<marcopg> the sugarbot guy seem to be back have time, btw
<mchua_> on Sugar and Activity testing, I'd propose as a first step to
have a sprint to try to spec out Things We Need To Build / Automate
<erikos> yeah seen a blog post
<mchua_> I don't think the problem space is well defined (from the
perspective of OLPC and of SL, both)
 w00t sugarbot!
<cjb> I think the lowest hanging fruit is sending out mail saying "hey
this guy wrote this sugarbot thing and none of us have had time to
look at it but I bet it'd make a good framework for activity testing"
<marcopg> mchua_: I second that proposal
<cjb> and, you know, you offer to make someone the head of the
automated testing team, and so on
<mchua_> as in, "I really don't know what we have/want/need re:
automated Sugar/Activity testing for OLPC" - I have a reasonable idea,
but it's likely to be out of date, and has huge holes.
<marcopg> cjb: did someone actually look into sugarbot?
<mchua_> cjb: +1, though I'd like to find ways to fit that into a
larger, more coherent "so, Activity testing... here's the big picture"
thing.
<cjb> marcopg: I think we've all seen the screencast, but never tried running it
<marcopg> I only have a vague idea of what it does
<mchua_> marcopg: I did, a *veryveryvery* little bit. No patches.
Just... I got it to run, once, on my desktop, which no longer has it
installed (I wiped it and reinstalled ubuntu when it was getting
flaky, months ago.)
<marcopg> cjb: yeah was thinking more of code sanity :)
<mchua_> marcopg: so, practically, "no."
<mchua_> I also recall the setup process to be sort of painful for me,
but that's probably due to n00bness in large part.
 I'd have to do it again to really say.
<marcopg> so time is expired and my mum will get angry if I don't go
to dinner :(
 pizza!
<mchua_> (in other words, "I didn't document it, so it didn't happen.")
<marcopg> what about if I take action to schedule an automation
discussion in one of the next sugar development team meetings
<cjb> mchua_: oh, you did more than us, at least :)
<mchua_> marcopg: and I'll take the parallel action item to schedule
the same for the olpc internal test group, with a warning that that
will probably happen in January
<marcopg> and pester mchua_, cjb, the sugarbot guy and other
interested people to participate
 mchua_: sounds good
 ok great!
 we are done I think
<mchua_> marcopg: agree
<marcopg> thanks everyone for coming
 mchua_ in particular to make me much more clear about activities situation :)
<mchua_> marcopg, your meeting-fu increases. :)
<marcopg> happy to see things moving there!
<mchua_> Likewise!
<marcopg> mchua_: I will work, at some point! :)
 gah
 rock
 I need to learn to type, first