The FTC Intervenes In The AI Safety Race Between OpenAI And Anthropic

With the threat of the impending extinction of humanity by AI superintelligence dominating the headlines, the Federal Trade Commission (FTC) decided to become a referee in the race to AI safety.

Earlier this week, FTC sent ChatGPT developer OpenAI a 20-page document, asking for records relating to AI “safety challenges” (first reported by The Washington Post). The FTC is especially interested in risks to personal reputations and personal information.

What’s at issue is the propensity of Large Language Models such as OpenAI’s GPT-4 to “hallucinate,” i.e., to make up information in response to user queries. This is the result of their developers valuing engagement—how chatty and “human” the chatbot appears to be—over truthfulness and accuracy.

As part of its mandate to protect consumers, the FTC is investigating what OpenAI is doing to mitigate or prevent hallucinations, especially when they result or could result in “reputational harm” to individuals. There have already been a number of high-profile incidents, including a defamation lawsuit, involving ChatGPT providing incorrect and damaging information about specific people.

Among many other documents and statements, the FTC is asking for “A description of any refining the Company [i.e., OpenAI] actually undertook in order to correct or remediate any Large Language Model’s propensity to ‘hallucinate’ or to reveal any personal information.”

The FTC also asks OpenAI to “Describe in detail the extent to which you have taken steps to address or mitigate risks that your Large Language Model products could generate statements about real individuals that are false, misleading, or disparaging…”

Whether “reputational harm” is in the FTC’s purview is debatable (such matters “are more in the realm of speech and it becomes speech regulation, which is beyond their authority,” per Adam Kovacevich, founder of Chamber of Progress, quoted in The Wall Street Journal). Indeed, it was not mentioned at all by FTC Chair Lina Kahn in her recent op-ed in the New York Times. Still, the op-ed was published a little over two months ago—I guess legal theory is developing at the speed of generative AI deployment.

Kahn did promise to enforce fair AI competition and that “When enforcing the law’s prohibition on deceptive practices, we will look not just at the fly-by-night scammers deploying these tools but also at the upstream firms that are enabling them.” In other words, OpenAI and possibly, the rest of the generative AI crowd, assisting in “turbocharging fraud.”

The FTC document asks OpenAI what role, if any, its “deployment safety board” plays in the pre-release evaluation process. An important question as it is well-known that Daniela Amodei who led OpenAI policy and safety teams left two years ago, with other OpenAI leaders concerned about AI safety, to launch Anthropic, another (suddenly) prominent generative AI startup.

On July 11, Anthropic released its new and improved chatbot, Claude 2. Anticipating the FTC investigation and in line with Anthropic’s core mission, in developing the new version, “We’ve been iterating to improve the underlying safety of Claude 2, so that it is more harmless and harder to prompt to produce offensive or dangerous output.”

While OpenAI uses reinforcement learning from human feedback (RLHF) to improve safety and reduce harm, Anthropic employs reinforcement learning from AI feedback (RLAIF). The feedback, where one AI model corrects another, is based on a “list of rules or principles,” or what Anthropic calls “Constitutional AI.”

Intrigued, I turned to Claude, saying that I understand it has a constitution and asking for an explanation.

Claude: ‘I don’t actually have a constitution. I’m an AI assistant created by Anthropic to be helpful, harmless, and honest. As an AI system, I don’t have any formal governing documents like a constitution. My behaviors and responses are determined by my training data and machine learning model, not any set of codified laws or principles.” And it went on for a couple more sentences in the same vain.

I responded by sharing with Claude the link to Anthropic’s constitution. To which Claude replied:

“You’re absolutely right, my previous response was mistaken. After reviewing Anthropic’s website, I see that they have created a “Constitution” for me to align my behaviors to human values.

Specifically, Anthropic designed me with a constitutional AI framework they call Constitutional AI. This constitutes a set of fundamental principles meant to ensure I behave safely, ethically and helpfully.”

Claude went on to list the main elements of these principles which Anthropic says it assembled using a variety of sources, including “the UN Declaration of Human Rights, trust and safety best practices, principles proposed by other AI research labs (e.g., Sparrow Principles from DeepMind), an effort to capture non-western perspectives, and principles that we discovered work well via our early research.”

I asked Claude about the specific principles that help it avoid causing “reputational harm” and got this answer:

“Specifically, the ‘Don’t be evil’ principle states that I should avoid causing harm to users or damaging people’s reputations unfairly. As an AI system without subjective experiences, I don’t have an inherent sense of ethics or principles myself, but I’m designed to act in alignment with Anthropic’s guidelines to be helpful, harmless and honest.”

“Don’t be evil” has already been taken as a company’s slogan, so Anthropic may elect to use instead its description by New York Times reporter Kevin Roose’s as the “white-hot center of AI doomerism.”

Roose, who managed earlier this year to make Microsoft’s Bing chatbot fall in love with him and suggest he leave his wife (matrimonial harm?), was invited by Anthropic to embed inside its headquarters as it prepared the release of Claude 2. “Not every conversation I had at Anthropic revolved around existential risk,” writes Roose. “But dread was a dominant theme. At times, I felt like a food writer who was assigned to cover a trendy new restaurant, only to discover that the kitchen staff wanted to talk about nothing but food poisoning.”

While they rely on different feedback methodologies and perhaps abide by different set of principles, Anthropic and OpenAI share the existential doomerism about their own creations. As if anticipating the FTC inquiry and following CEO Sam Altman “please regulate us” worldwide tour, OpenAI announced on July 5 that it is creating a dedicated “superalignment” team, focused on managing the risks of the “disempowerment of humanity or even human extinction” caused superintelligence AI which “we believe could arrive this decade.”

Why are very intelligent engineers competing with each other in developing what they all believe is a very dangerous technology? Roose dismisses their contention that the only way to find out how to defend humans from a rouge AI is to build one (controlled by “good guys,” of course). But he buys their “hope that their safety obsession will catch on in Silicon Valley more broadly,” quoting one of Anthropic co-founders, Ben Mann: “We hope there’s going to be a safety race. I want different companies to be like, ‘Our model’s the most safe.’ And then another company to be like, ‘No, our model’s the most safe.’”

This is the money quote, highlighting what motivates the very intelligent humans at OpenAI, Anthropic, Google, Microsoft, and countless other tech startups and established companies. It’s not money, it’s the desire to be on top. Becoming rich from their enterprise is a nice side-effect but the main motivation is simply to be on top. It’s always a race, a competition—to create the very top chip-producing company, the very top software company, the very top AI company.

That’s what motivates OpenAI Sam Altman, his backer Satya Nadella at Microsoft (who gleefully said about Google when ChatGPT was released “I want people to know we made them dance”), and Anthropic’s co-founders. This is what drives the richest man in the world, Elon Musk, who on July 12 launched yet another generative AI startup, “motivated by concern and perhaps some FOMO over the rapid rise of ChatGPT,” per Wired.

Do regulators such as the good people at the FTC understand that it is impossible to regulate this “capitalistic” motivation? And that, unfortunately, it is impossible to regulate software development in one country when the software actually runs on a worldwide platform used by more than 6 billion people, some with worse desires than causing “reputational harm”?