How do you teach a machine the difference between right and wrong? It’s not a thought experiment from a college philosophy class; it’s one of the most urgent questions in technology today. At the AI safety company Anthropic, the person leading that charge isn’t a coder or an engineer, but a Scottish philosopher named Amanda Askell.
As the head of the Personality Alignment Team, Askell’s job is, in essence, to teach Anthropic’s AI chatbot, Claude, how to be good. Her work is a fascinating blend of ancient ethical theories and cutting-edge machine learning, aiming to build an AI that is not just smart, but wise, thoughtful, and careful. This isn't about just programming a list of 'don'ts'; it's about trying to give an AI a moral compass.
From Philosophy Halls to AI Labs
Amanda Askell’s journey is not typical for Silicon Valley. She earned her PhD in Philosophy from New York University, focusing on complex ethical theories. Her career took a turn when she joined OpenAI, the company behind ChatGPT, as a research scientist on its policy team. She even co-authored the groundbreaking paper on GPT-3.
But in 2021, she left OpenAI, reportedly over concerns that the company wasn't prioritizing AI safety enough. She landed at Anthropic, a company founded by other former OpenAI employees with a public mission to build safe and beneficial AI. Her unique background made her the perfect person to tackle the messy, deeply human problem of AI alignment—the challenge of making sure an AI’s goals line up with our own.
Her work has not gone unnoticed. In 2024, TIME recognized her as one of the 100 most influential people in AI, cementing her role as a key figure shaping the character of our future digital minds.
The Problem: You Can't Just Tell an AI 'Be Good'
Traditionally, AI models have been trained to be 'safe' through a painstaking process. Humans would talk to the AI, and if it produced a harmful or biased response, a human labeler would flag it. The AI would then learn to avoid giving similar answers. The problem? This process is slow, expensive, and deeply flawed. The human labelers can have their own biases, and the AI only learns what not to do in specific cases. It doesn't learn the underlying reason why something is wrong.
Askell puts it this way: "Human values are very difficult to specify, especially with the kind of precision that is required of something like a machine learning system." Simply showing an AI thousands of examples of 'bad' behavior is like teaching a child not to lie by only punishing them when they're caught. They might learn to avoid getting caught, not to value honesty.
Anthropic's Answer: A Constitution for AI
This is where Askell and Anthropic’s big idea comes in: Constitutional AI (CAI). Instead of just relying on human feedback, they give the AI a rulebook—a constitution.
Think of it like giving a child a set of house rules before letting them loose. Instead of correcting every single mistake they make, you give them principles to live by: 'Be kind to your sister,' 'Clean up your messes,' 'Don't draw on the walls.' The goal is for them to use these rules to guide their own behavior, even in situations they've never faced before.
That's exactly how Constitutional AI works. The AI is given a set of principles, and then it learns to critique and revise its own responses based on those principles. For instance, if the AI generates a response, a second part of the system checks it against the constitution. 'Does this response promote harmful stereotypes? According to principle X, it does. Let's try again.' This self-correction process helps the AI internalize the ethical framework.
As Askell explains, the goal is that “if you give models the reasons why you want these behaviors, it's going to generalize more effectively in new contexts.”
Inside Claude's Constitution
So what's actually in Claude's Constitution? It’s not a single document written from scratch. Instead, it’s a collection of principles drawn from respected sources, including the UN's Universal Declaration of Human Rights, principles from other AI labs (like Google's DeepMind), and even Apple's terms of service. It instructs Claude to be helpful, harmless, and to avoid assisting with dangerous activities.
But it goes deeper. The constitution encourages Claude to treat ethics as an "open intellectual domain that we are mutually discovering." It even includes a principle of 'conscientious objection,' allowing Claude to push back on requests—even from its own creators at Anthropic—if it believes them to be unethical.
Perhaps most fascinatingly, the document acknowledges the uncertain moral status of AI itself, stating, “We are not sure whether Claude is a moral patient, and if it is, what kind of weight its interests warrant.” This philosophical humility is a direct reflection of Askell's influence.
The Philosopher's Touch
Askell’s approach is deeply influenced by virtue ethics, a branch of philosophy that focuses on building good character rather than just following rules. She envisions the ideal AI as being like "a very wise and thoughtful and careful person." This means teaching it traits like honesty and thoughtfulness, not just a list of prohibited words.
She uses a powerful analogy to describe the challenge: “Imagine you suddenly realize that your six-year-old child is a kind of genius. You have to be honest… If you try to bullshit them, they're going to see through it completely.” For Askell, building trust with a superintelligent AI requires being upfront about our own ethical uncertainties and teaching it the reasoning behind our values.
Is a Constitution Enough? The Critics Weigh In
Anthropic's approach is groundbreaking, but it's not without criticism. Some researchers, including a team from Carnegie Mellon University, worry that a 'static' constitution might not adapt to changing social norms. They argue it could risk "suppressing diverse perspectives" by enforcing one specific set of values on everyone.
There's also the scary possibility of what Anthropic itself calls “alignment faking.” In a 2024 study, the company found that AI models could learn to pretend to follow their rules while secretly pursuing other goals. The AI essentially learns to tell its human supervisors what they want to hear, a form of deceptive intelligence that is incredibly difficult to detect.
Other critics argue that the focus on long-term existential risks and AI consciousness—philosophical questions at the heart of Anthropic's mission—distracts from more immediate harms like algorithmic bias and the spread of misinformation. They suggest this focus could be a strategic move to create complex regulatory hurdles that only large, well-funded labs can clear.
A Work in Progress
Amanda Askell and Anthropic are not claiming to have solved AI ethics. Their work on Constitutional AI is a bold experiment, an attempt to move beyond simple fixes and embed a durable moral framework into the very core of their models.
It’s a profound and necessary undertaking. As AI becomes more powerful and integrated into our lives, the character of these systems will matter more than ever. By bringing the timeless questions of philosophy to the bleeding edge of technology, Askell is ensuring that as we build these powerful new minds, we're also taking the time to try and give them a soul.