In the period when prosecutions were wrongly being brought by the Post Office toward its sub-postmasters for errors generated by their Horizon IT systems, the judge in one of the first legal cases declared:
“It is inescapable that the Horizon system was working properly in all material respects.”
In contrast, “No one can say computers cannot go wrong”, declared a union rep for sub-postmasters early on.
Corrie Wise, an expert witness in court cases involving IT systems, commented:
“Almost all IT systems ever built have malfunctioned at some point … it may not have been IT glitches but the way the Post Office handled it”.
In doing so, she neatly underlines the ‘wise’ balance between the tool and how the human handles it.
Advancing by accident
Looking backwards into the history of technology, it is obvious that there were advances only after many accidents. Chimneys were first built of wood and plaster, and only after about a century of house fires was the design made safer and the need to sweep out the soot annually realised. During one of the very first public demonstrations of a steam train, MP William Huskisson, was killed because he unwisely got out to greet another worthy.
Safety principles
Now reading up on safety principles for technical apparatus, it is clear that there are two sides, as Wise pointed out: there is the possible failure of the machine and there is the possible misuse of it by humans. Experts who work in safety, in whatever sphere of technology, would always investigate to see if there were failures in the machine in parallel with human error. In a crashed aircraft, for example, the investigative team would consider a host of factors including pilot error, along with maintenance or ground crew error, and examine this in parallel with mechanical or electronic failure.
Naturally, sometimes it is human error that causes the technical failure. A notorious example is the mis-curvature of the Hubble telescope mirror “a spherical aberration caused by the malfunction of a measurement device used during the polishing of the mirror”. Technicians and quality control at the company that produced the mirror just assumed the curvature was correct and sent it onwards for delivery to space. Fortunately, other engineers were able to work out how to achieve sharp images from Hubble by sending up additional equipment to be attached to it by astronauts – all costing many millions of dollars more, so an expensive human and systems error.
In a month when a commercial moon probe has just failed, it is obvious that this is a technology still in the early stage, working with prototypes that need to be adjusted. But the value of what can eventually be utilised on the moon by humankind is so enormous that there are plenty of other commercial companies willing to risk further investment in moon probes.
The cost of early adoption
The cost of being an early adopter in a new technology can be considerable. The town-centre parish in Ashford installed a heat pump for a large medieval building more than a decade ago. When, two years ago, it failed, it was discovered that spare parts were no longer available. The parish has since gone back to using the old technology of a gas boiler.
The new technology that is much in the news now is Artificial Intelligence (AI). One of the concerns raised over AI is how it is normed (essentially the scripting of the rules and inference parameters that enable it to function) according to the data available when it is developed. Such data may well be incomplete or insufficient. When I worked in South Africa, I tried to be an early adopter of whatever was on offer from computers. For example, there were suites of image graphics available to beautify invitations, but looking at the human figures on them, they were all Barbie-style female figures, North American blondes.
What did you say?
I had a similar problem with audio programmes. It would have saved time to use a programme to turn audio recordings (of meetings) into script that could be saved and circulated. But the programme to do this, ‘Dragon Speaking Naturally’, did not work. One had to train it on one’s own voice, but it completely failed to pick up even my UK English intonation, let alone any of the myriad South African accents.
Our daughter in South Africa still has this problem. Her phone was stolen recently and to get her insurance she had to speak to the company through an app, which completely failed to make sense of her intonation. In all these examples, the response given by the programme is based on what was most frequent in the training data, and so what is most likely to fit the majority of users.
AI-based IT systems: more data, please
Manufacturers who use AI claim that the problem of norming can be solved by even bigger data sets. The development of self-driving vehicles, for example, depends on millions of hours of training with real life road images and incidents. But even so, some exceptional incidents have caused fatalities.
The statistics of norming can be cruel from an individual perspective. Look for a bell curve in any set of data, and there may still be some out-riders or trailing cases. If the data is from education or health, those outrider cases are individuals. When a doctor says
“95% of patients recover from this operation”, with a pang one realizes that 5% do not.
One reason why the Post Office was able to suppress complaints about the Horizon system was because they told each complainant “you are the only one”. Yet once the complainants got together, they were able to see patterns of malfunction. So, it is really important (given funds and staff) to investigate the exceptional cases.
Lessons to be learnt
There are lessons here over what to do with AI. First, we must not be unduly optimistic about its capabilities in the initial stages. Its norming data might be wrong for our situation. Second, we must pay attention to the outriders. They are people too, and they may actually represent a growing number. The manufacturers might be norming their data ignoring a whole slice of the world, such as concentrating on North American producers while, comparatively speaking, ignoring Africa, or even the United Kingdom.
