[sugar] Automated testing of activities

[Mitch Bradley]

Kent Quirk wrote:
> Is automated testing intended for more than just battery life testing? 
> If not, is it really necessary for every activity to support it? If so, 
> what do you expect to accomplish? Will it actually save more than the 
> amount of time taken to implement it for a given activity?

As one of the instigators of the push toward automated testing, I should 
chime in with my two cents worth:

Fully-automated testing of activities could indeed be a huge time sink, 
but I think there are some easy steps that should more than pay for 
themselves:

a) Activity developers should write down use cases for each major 
feature, including a description for how to exercise that feature.  This 
part is just engineering 101, and it is easily justified.  It will help 
the testing team do manual testing too.

b) It would also be helpful if each activity listed its assumptions 
about system files and its effect on the filesystem.  We are thinking 
about removing unused system files to reduce bloat, so we need some way 
to determine whether removal of a system file causes activity breakage.  
If we had an exerciser script or storyboard (doesn't have to be 
automated; could be executed by a human) as well as some way to detect 
failure, then we could do regression testing.

Automated testing is a laudable long-term goal, but in the short run we 
need something that is reasonably comprehensive, whether automated or 
not.  It seems like a bad idea to depend on the testers to come up with 
ad-hoc test sequences for activities.