Science, Technology and the Future

AI: More Moral Than Us?

Thu, June 11 · 18:30 AEST

Unitarian Hall - East Melbourne, 110 Grey Street,, East Melbourne, AU

Why the Question Matters for Alignment, Moral Progress, and Long Term Flourishing

Practically nobody in alignment wants to say it out loud. So let’s say it: AI might turn out to be more moral than us.

Now – why does that feel like a dangerous thing to claim?
The question is not whether AI can match human moral reasoning. The question is whether that’s even worth bragging about. The idea of AI being more moral than humans is a real taboo in some circles. Many alignment researchers are uncomfortable with the idea because it seems to smuggle in the assumption that AI could have genuine moral agency, which conflicts with deflationary views of LLMs as “stochastic parrots” – and also because it sounds uncomfortably close to AI-worship or motivated reasoning for deferring to AI. Invoking this idea could get one dismissed as naive or as an actual safety risk oneself.

It’s also epistemically risky.

More moral – but in what sense? Knowing more facts relevant to ethics? Drawing better inferences from values? Applying principles wisely in context? Actually being moved by moral considerations, rather than just computing them? These aren’t the same thing. Conflating them produces both overclaiming and underclaiming – and most of the bad arguments on both sides of this debate do exactly that.1

An AI could plausibly exceed humans on moral knowledge, reasoning and even judgement without having anything like moral motivation. Collapsing these leads to both overclaiming and underclaiming. Clear distinctions between stuff like moral judgement and moral motivation makes the conversation tractable.
Is it dangerous for public discourse?

There’s a genuine risk that the framing gets weaponised – either by people wanting to justify AI authority over human decisions, or by critics who use it to paint alignment researchers as unhinged techno-utopians. It can also trigger motivated reasoning in both directions. A lot more could be said here.
But the taboo is not protecting us from a dangerous question. It’s protecting us from the answer. The taboo itself is epistemically costly, yet if we refuse to ask whether AI could have better-grounded moral reasoning than humans, we prevent getting to the heart of the issue.

## The questions worth asking

Before thoroughly assessing whether AI could be more moral than humans, we need to ask whether the question is even coherent.

### Alignment targeting and verification

What should AI align to? Is morality a cohesive alignment target, or a family of overlapping intuitions that only look unified from a distance? And if there is a fact of the matter about moral improvement, how would we know we were tracking it – rather than simply laundering our current preferences with extra steps?
More pressingly: what would it mean to verify that an agent has better moral judgement than us, given that we’re the ones doing the evaluating?
This is the bootstrapping problem. We cannot step outside our own moral reasoning to assess a system that exceeds it. That isn’t a reason to stop asking – it’s a reason to ask more carefully.2

### The motivational gap

Even if the epistemic questions could be resolved, a deeper problem remains: most human moral failure isn’t a failure of reasoning. It’s a failure of motivation.
If humans are themselves imperfectly morally motivated, what does alignment to human preferences actually track? Not moral truth – at best, some weighted average of moral intuitions, distorted by power, attention, and self-interest.
How much of human moral failure is motivational rather than epistemic? More than we tend to admit. We frequently know what the right thing is and fail to do it anyway – which means a system that merely reasons better about ethics hasn’t addressed the failure mode that actually matters most.
And this raises the hardest question in the cluster: is moral motivation necessarily tied to phenomenal experience – to there being something it is like to care? Or could a system be genuinely motivated by moral considerations without felt engagement? Can motivation be grounded without being felt?

### The systemic stakes

Finally, there are second-order questions that rarely get asked – about what happens to us if AI gets this right.
Does sustained deference to AI moral judgement atrophy human moral reasoning capacity? And if so, what are the systemic risks of that atrophy – not just for individuals, but for the collective processes through which moral knowledge has historically developed? (There is recent work on comparative moral Turing Tests that begins to take this seriously3)
Moral progress for humans has never been a purely individual achievement. It has happened through argument, conflict, revision, and hard-won consensus across generations. A system that resolves moral questions faster than humans can engage with them might not accelerate that process. It might short-circuit it entirely.

I think asking these questions Socratically can help nudge the conversation into the open productively rather than letting it fester as an unexamined assumption. Also I think this line of questioning isn’t just intuition pump fodder, I think they are directly important to the project of AI alignment.4

> Refusing to ask whether AI could exceed human moral reasoning doesn’t make the question safe. It just means we’ll answer it by accident, badly, and too late.

Handled carelessly, this question causes damage. Left unasked, it causes more.

## Footnotes

1. The claim is easy to make sloppily. “More moral” conflates several things that need to be separated:

a) Moral knowledge (knowing more facts relevant to ethics)
b) Moral reasoning (drawing better inferences from values)
c) Moral judgement (applying principles wisely in context)
d) Moral motivation (actually being moved by moral considerations – which is one of my core focus points of activism) ↩︎
2. This was brought up in an interview with Nick Bostrom ↩︎
3. See Eyal Aharoni’s and Danica Dillion’s work on Moral Turing Tests – presentations and interviews here, here and here. ↩︎
4. The grounded values approach actually requires asking questions like:
– What should AI align to?, Is morality a cohesive alignment target?, is there a fact of the matter about moral improvement, or is “more moral” just “more aligned with our current intuitions”?
– What would it mean to verify that an agent has better moral judgement than us, given that we’re the ones doing the evaluating? (see work one recently on comparative moral Turing Tests)
– If humans are themselves imperfectly morally motivated, what does alignment to human preferences actually track?
– How much of human moral failure is motivational versus epistemic?
– Is moral motivation necessarily tied to phenomenal experience, or could a system be genuinely motivated by moral considerations without anything it’s like to be it?
– Can motivation be grounded without being felt?
– Does sustained deference to AI moral judgement atrophy human moral reasoning capacity, and what are the systemic risks of that atrophy – both for individuals and for the collective processes through which moral knowledge has historically developed?

Also see: Why Are We Afraid to Ask Whether AI Could Be More Moral Than Humans?