Most people are familiar with software in their everyday life and workplace, which operates based on rules. They may be complex rules, but they are (bugs aside) predictable. They have been explicitly programmed to follow a set of instructions which turn inputs into outputs.
AI works differently. It ingests data and learns how to interpret it by establishing connections between different data sets. So, an English- Spanish translation AI is not explicitly told word by word ‘perro = dog’, ‘gato = cat’, etc, alongside fixed grammar rules. It is fed texts which have been translated and told to learn what pattern links one to the other (with guidance from language and data experts).
This allows it to learn complex tasks such as translation or image recognition quickly. Many tasks performed with AI would not be possible with traditional software, or would take decades of programming.
However, this approach brings unpredictability because the input data is complex and imperfect, not a set of binary options. To learn a language, an AI needs huge amounts of text and there is not time to manually check it all. Some translations may be poor, or contain mistakes, or deliberately misuse language. Even correct ones
contain nuance, where experts disagree on the precise translation. A phrase can be translated in several ways, depending on the context. Anyone who has used a translation app will know they are good, but not perfect.
Translation is usually low stakes, and we can trust a language translation AI for many applications, even if we can see it makes some mistakes. But for AIs which diagnose disease, spot when a plane engine component needs replacing, or predict drug formulations, we need to be very confident that it has reached the right answer before we can trust it.
Added to this complexity is that AI conclusions may be confusing, but still be correct. NASA used AI to design an antenna against
a defined set of criteria. The result would never have occurred to a human, but it was better aligned to their needs than anything a human came up with. What does one do when an AI recommends something completely counter-intuitive? It could be breakthrough
(as in NASA’s case), or it could be a spectacular oversight in the AI
design or training. How do we know?
All of this raises questions of trust. If we know it is not 100% accurate, we need to reach a decision about how much we trust its recommendation. This comes down to multiple factors including
how accurate we are told it is, how much we believe that claim, how much control we had over the inputs, how well we understand its decision-making, what supplementary information it has provided to back up its recommendations, its past record, the consequences of it being wrong, and the user’s own knowledge of the problem.