A lens for AI blurriness

A Book Review of "AI Snake Oil: What AI Can Do, What It Can't, and How to Tell the Difference", by Arvind Narayanan and Sayash Kapoor

After more than 2 years of GenAI hype (or GenAI doomism, depending on your intellectual bent), you may be feeling the need for a grounded analysis of the implications of artificial intelligence for society. “AI Snake Oil: What AI Can Do, What It Can't, and How to Tell the Difference", by Arvind Narayanan, professor of computer science at Princeton, and Sayash Kapoor, PhD candidate at the same institution, is a timely book that fills that gap. As many others do, the book explains the capabilities and risks of current AI technologies. But its novelty comes from unmasking the many times exaggerated, and sometimes plainly deceptive, claims of AI companies regarding those capabilities. These practices, innocently or negligently spread by the media, are the “Snake Oil” that the authors want to unmask.

The authors focus on 3 technologies: Predictive AI, Generative AI and Content Moderation technologies, but the most interesting section was the authors’ analysis of Predictive AI.

The Doom of Predictive AI

Predictive AI’s inherent limitations, the authors’ argue, make it likely to fail to automate decision-making: “predictive AI not only does not work but will likely never work, because of the inherent difficulties in predicting human behavior”. Some of the reasons for this failure can be attributed to the specific way machine learning models are created and deployed. For example, trained on past hospitalization history of an institution, a machine learning model could correctly predict that asthmatic patients with symptoms of pneumonia are at a lower risk of serious pneumonia or death but could totally miss that the reason for such lower risk is that asthmatic patients with symptoms of pneumonia are sent directly to the ICU to avoid complications and hence receive better treatment. Deploying such a model to decide when a patient should be hospitalized would have catastrophic effects in spite of being “correct” in the actual risk of patients based on the data. “AI can make good predictions if nothing else changes” but the world, of course, is constantly changing. There’s no clear path for Predictive AI to surpass this inherent limitation.

When predicting social behavior, additional difficulties arise from the fact that we lack theories to determine the limits of our predictions. In physics, we know that the laws of thermodynamics do not allow us to predict the behavior of individual atoms. Trying to make such predictions would only generate errors; we know the limits of such laws because we know that those errors are “irreducible”. It’s not the same with the theories that explain social behavior. Those theories are not as developed so we don’t really know their limits. Just as with the laws of thermodynamics, there may be “irreducible errors” in their predictions of social behaviors, but we don’t know them yet. And no machine learning model can be sufficiently accurate for important human decisions without a deep understanding of such limits.

Other practical reasons explain why Predictive AI fails: It doesn’t adequately deal with people behaving strategically to game AI systems (a challenge shared by content moderation technologies); it requires permanent human oversight but many systems are deployed without offering any recourse to the subjects of the decision (over-automation); training data will many times come from a different population than the one the model is used on; it can create inequality by using proxies that are based on economic or business incentives instead of measuring the actual data; and vicious feedback loops that derail accuracy.

These are all cogent arguments, but claiming Predictive AI is a failure seems a bit extreme. If the expectation were that the inherent limitations of Predictive AI prevent us from automating all decisions, then yes, Predictive AI is a failure, but beyond that all-or-nothing premise there’s ample room for using these technologies. Even when not accurate, predictions could be very useful in many situations where, for example, great accuracy is not needed, where social behavior is not involved or is involved but no fundamental right is at stake, or when information is so scarce that predictions slightly better than chance are still helpful. These predictions may allow for better decisions even with its inherent limitations. Current regulatory approaches that classify uses according to levels of risk and impose measures only to the levels of high risk suggest that Predictive AI has productive applications beyond the concerns of the “Snake Oil” claims.

A Recount on Gen AI and the Ladder of Generality

Narayanan and Kapoor move to Generative AI next. They offer a nice recount of the history and some of the technical properties of GenAI. If you’re not familiar with any of this, you’ll find their explanations about vectors, tokens and transformers highly readable and useful. And even if you’re familiar with these ideas, you might find some nice insights, such as how ImageNet helped build the current practices for developing AI or how the Google Photos-Gorillas issue¹ has not been really solved yet (!).

Besides this helpful overview, one of the main warnings that Narayanan and Kapoor give us about the discussions brought by Generative AI is against thinking that the main issue to address right now is the “existential threat” of AI or the “catastrophic risks” “Artificial General Intelligence”. The authors argue that “We don’t think AI can be separated into “general” and “not general”. Instead, the history of AI reveals a gradual increase in generality”. AI challenges are better understood through a “ladder of generality” framework that goes from special-purpose hardware in the “floor” through 6 rings of generality all the way to “instruction-tuned models”. This ladder can help us prioritize; we can use real-world data and experiences to defend ourselves from “specific threats” (e.g. cybersecurity and product safety) instead of regulating “existential threats”, that are not likely to being substantial in the near future². The authors also suggest that this approach will be more fruitful than “alignment” efforts, that is, efforts to align technologies to our values so that they don’t help bad actors.

AI for content moderation

For content moderation technologies the analysis is less pessimistic than for Predictive AI but the authors also highlight some inherent difficulties: (i) Content moderation is heavily dependent on context, and AI’s inability to discern that context creates a major limitation for its use in this realm; (ii) Content moderation is also inherently political and dependent on how the world changes;(iii) Some content becomes harmful only in the aggregate, away from the reach of content moderation technologies that review individual posts; (iv) Humans change perception constantly and perceptions vary across borders; (v) communities constantly create their own slangs to prevent outsiders understanding public communications, making it harder for AI to moderate such content, especially because many times members of these communicates intentionally use such language to circumvent content moderation technologies.

The book ends with a reflection of why it is important to fight against these myths. AI’s impact in the world is still up for grabs but, if we don’t do something about it, the grimmest world, one where AI takes over the content of the world unchecked, is the most probable one. So the authors have a call to action to all of us: “We hope it is clear that vastly different futures are possible when it comes to the role of AI in society…we are not okay with leaving the future of AI to the people currently in charge… There is a role for everyone in shaping the future of AI and its role in society. We are playing our small part by writing this book and a newsletter. Join us”.

Snake Oil is a great book for the general reader with no previous technical background on AI or the societal issues that it brings. You won’t find detailed analysis of these issues in the book but this is not a flaw; it’s a feature. Even if you are familiar with these issues and are looking for a deeper analysis, this book may serve as a lens to better see the context and find where to go next.

1. The scandal that involved Google Photos mistakenly classifying black persons as “gorillas” was well documented in the media back in 2015, as shown here.

2. For a more detailed analysis about the book’s principles to address AI risks, we recommend reading the “Six principles for thinking about AI Risk” post in Understanding AI substack, by Timothy B. Lee.