Artificial intelligence (AI) has come a long way. Once the exclusive province of speculative fiction and scientific hypotheses, today it is being entrusted with a range of increasingly vital tasks. Search engines, language translation, drug design, financial fraud detection, camera surveillance, medical evaluation and treatment, self-driving vehicles—all of these are beginning to take advantage of AI’s unique potential to automate time-intensive tasks, with further applications continually being explored.
The adoption of any new technology, however, brings with it new risks and vulnerabilities. Some stem from the inherent difficulties of developing machines that can account for the vast diversity, inconsistency, and frequent lack of clear, distinct categories in the real world. For example, things like variations in the shape, size, or color of ideally standardized objects like road signs can confuse AI. Additionally, systemic biases found in datasets routinely create unequal outputs in analysis of human faces and human behavior. Other risks to AI stem from more malicious sources.
All AI systems, from simple chatbots and image generators to highly sophisticated data analysis tools, develop their algorithms by categorizing and processing information from pre-existing datasets, like collections of images or written documents, and recombining that information in response to queries or operation by a human user. If those input datasets are tampered with, the output results could be catastrophic. That’s something Abderrahmen Amich (M.S. ’21) has dedicated his research to preventing.
“AI is now extensively deployed in critical domains throughout the world, and this raises a big question about whether we can trust it, is it secure for these important tasks?” he says. “The answer is no. Research has shown it has privacy and security vulnerabilities. Imagine if a car misclassifies a stop sign: The consequences could be drastic. We need to improve its robustness to defend against adversarial inputs.”
Trust Building Exercises
One of AI’s primary vulnerabilities lies with an attacker’s ability to manipulate its algorithm through something called perturbation attacks, which provide an AI system with data inputs that lie on the fringes of its training and which produce erroneous behaviors. When these attacks are purposefully designed and deployed, they can potentially cause an AI to fail in predictable ways that serve the interests of the attacker rather than the user.
As a Ph.D. student in the Department of Computer and Information Science at the University of Michigan-Dearborn, Amich’s work has been instrumental in developing a number of tools to both protect AI systems from attacks and make their development more robust in the first place.
Perturbation analysis is a routine component of developing AI, in which developers intentionally expose the system to a series of intentional disturbances in order to gauge how it responds to them and what measures need to be taken to improve that response. In his earliest research on the subject, Amich found evidence that this analysis is often inefficient, with most perturbations not producing any consequential effect on the system. In response, he created EG-Booster, a tool that improves perturbation analysis by weeding out those inconsequential attacks, therefore providing developers with more focused data.
Amich (left) has developed several tools to improve the security and reliability of AI, including EG-Booster and Morphence.
While conducting the research that led to EG-Booster, Amich observed that a major reason machine learning is vulnerable to attack lies with something called the out-of-distribution (OOD) assumption.
“In simple terms, AIs are trained on a particular distribution of data, like a set of images,” he explains. “But that means that if an AI encounters images that are sharper or blurrier than the ones it was trained on, they lie outside its distribution and it can have difficulty reacting to them. And if it is purposely fed OOD data via perturbations, that can cause the model to shift in drastic and unknown ways.”
To help this, he and his colleagues developed Out2In, an in-development tool that makes OOD data intelligible to the AI model. For example, Out2In will be able to interpret even a blurry, out-of-focus image of a stop sign and respond accordingly.
“This tool will be useful for a variety of AI applications because it goes beyond defense,” Amich says. “Because it can translate adversarial to clean images, it can not only protect a model from perturbations, it can also translate naturally OOD images that it may encounter in the course of its real-world use.”
A third tool developed by Amich, called Morphence, provides AI with additional real-world defense. Called a moving-target defense, Morphence allows an AI model to shift quickly between different data sets in order to stay ahead of adversarial inputs.
“By using Morphence, the decision function of a model changes from one step to another during a query,” he explains. “That way, if an adversary is intentionally trying to fool the model, it will be ahead of their strategies, and the attacker won’t know which model to target in a given moment.”
AI, Amich says, is only going to continue to grow as a tool, and will continue to impact our lives in increasing ways, which is why ensuring it is reliable and robust is such a vital undertaking.
“The impact of AI in security, in medicine, in automotives, in smartphones, in a wide range of fields, is very high,” he says. “AI is everything right now. And as long as we have software and AI systems, we need them to be safe and secure, and that’s where our work can have great value.”