[IAEP] Teaching that the error condition is normal, and the wisdom of crowds

[Samuel Greenfeld]

Earlier this year, one of the public transit systems that I take to work
decided to upgrade their fare collection system.  The exact name or nature
of this system is not important, apart from the fact it is a regional train
line.

But as someone hired to test Sugar & its related items, I saw this as an
interesting opportunity.  We often talk about learning to use Sugar, and
here I would be able to observe people learning to use a system they have
never used before.

Now I do not want to unnecessarily criticize the system since they have
solved most of the significant issues, so instead of giving a full
explanation and critique, this email will focus on the issue I find the most
intriguing.  For the system seems to have successfully taught many people
that an error condition is normal, and they seem to ignore it when it
occurs.


How did this happen?  To explain this we need to take a few steps back.  The
old system was paper-based with printed information, similar to (but not
compatible with) what many other public transportation agencies in the area
use.  The new system (somewhat simplified) uses contactless smartcards
compatible with an upgrade previously done by one neighboring transportation
agency, but issued somewhat independently.  Paper tickets are still
available, but only for single-day use.

Since the system in question uses mostly unattended stations with a
proof-of-payment(*) approach, there is a need to verify that users have the
paid appropriate fare when a guard on the train asks you for it.  With paper
tickets this is easy, as the ticket shows your start and end destination,
along with any other information (expiration date, trips used, etc.)
needed.  But with smartcards, this information is only stored
electronically.  In order to prove you got on and off where you were
allowed, the new system introduced computerized fare validators installed on
every train platform which the contactless smartcards have to be touched
against.  Failure to use a validator prior to getting on or after getting
off a train may result in the maximum one-way fare being charged against the
card.

The fare validators are smaller than the fare payment machines, and to the
best of my knowledge only provide information in one language.  To otherwise
communicate, they use a series of beeps.  There are two main sounds, one a
good "ok" sound, the other a bad "error" sound, which differ noticeably in
tone, sequence, and volume.  Information is displayed on a status display
when a smartcard is touched against the card reader on each validator, but
disappears as soon as the card is out of range.

The problem stems from the fact that the error sound has been heard far too
often, and for far too many reasons.  A user which moves their card in front
of the reader too quickly or keeps it too far away may result in an error
tone, along with a possibly incorrect error message.  The lack of a proper
payment method on a card will cause it.  Validators near high pedestrian
traffic areas seem to often break down, reporting "Card not valid" or
"Contact customer service" whenever a card is touched against them.  When
this happens, it is rare to see someone actually attach a note to the reader
labeling it as broken, yet the broken condition may last for days.

The handheld checkers used by guards on the train can also be configured to
make the same "ok" and "error" sounds, and guards in some cases do not
challenge riders when they make the error sound, although they definitely do
in others(**).  Since the error sound is so prominent and noticeable
compared to the good sound, everyone seems to expect it.

The end result is a situation much like Windows User Account Control or End
User License Agreements where many people seems compelled to click "Yes".
They just heard the error tones, often do not try a different reader if it
is too far away or if they are in a rush, and get on the train or head to
their destination/transfer while potentially risking a financial penalty.


So what mitigates this?  The other topic I wished to talk about is the
wisdom of crowds.  For I have noticed if someone watches a person touch
their smartcard against a validator, and it makes the error sound, and then
that person walks to another reader which produces the "ok" sound, then the
next person is much more likely to try the other reader to see if it works
with their card as well.  If someone watches another person get the error
sound and the latter does not attempt to go to another validator, then they
are unlikely to try another validator as well.

Advice has changed over time, and it takes a bit of time for the crowd to
catch up.  Early on many monthly pass (unlimited ride/distance) holders were
told or told each other that they did not have to use the fare validators,
as they never would be penalized.  But then some guards told them that
without using the validators, their smartcards would appear to be unloaded.
It is unclear to me if this was an early bug with the handheld checkers, or
what the current state is.  Still, as more people were seen using the
validators, other people started using them, creating a bit of a snowball
effect.

But the wisdom of crowds also fails sometimes.  Remember I said that this
system uses both paper-based (one day only) and smartcard-based (supporting
everything) tickets?  Well the three major transit systems peering with this
one offer discounts to transfer to and/or from this train line, in addition
to discounts which may be taken on this train when coming from one of
them.   Two of the primary peers only use paper tickets themselves, while
the third runs almost a pure smartcard system.   The result is a maze of
transfer discounts and system compatibility which works in some cases but
not others.

Fortunately since the new system has been online a few months things have
been getting better.  So I apologize if I make it sound if things are worse
than they actually are.  And by my rough estimate, the error tones are heard
a bit less then they were before.

(*) http://en.wikipedia.org/wiki/Proof-of-payment
(**) It's worth nothing that this does not mean you can get away with just
an empty card.  The handheld checkers provide a lot of information to a
guard including recent usage & fare loading history, and they are getting
better at using them all the time.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sugarlabs.org/archive/iaep/attachments/20110612/1113a99f/attachment.html>