The Singularity: Are We Gambling With Humanity?

For most of human history, our greatest technological achievements have amplified human abilities. AI may be different. We are attempting to create something that could eventually surpass us intellectually—and potentially improve itself.

That possibility has moved from the realm of science fiction into an increasingly serious debate among the people actually building these systems.

Recently, Anthropic researcher Jacob Coxon resigned, arguing that companies including Anthropic and OpenAI are racing toward self-improving superintelligence while humanity does not yet know how to make such systems reliably safe. His concerns were reinforced by an Anthropic alignment researcher (aligning human values, intentions, and safety standards with AI goals, behaviors, and outputs). The researcher said he believes there is greater than a 10% chance that AI could kill all humans within the next decade.

Ten percent is an astonishing number.

If someone told us that a new technology had a one-in-ten chance of ending human civilization within ten years, would we consider that an acceptable risk in exchange for its potential benefits?

Or is the comparison itself misleading?

The difficulty is that nobody can actually calculate this probability in the conventional scientific sense. There has never been a superintelligence, so there is no historical dataset from which to extract a probability. The statement is essentially a researcher's informed judgment about an unprecedented future scenario. Even people making these estimates acknowledge the uncertainty.

But uncertainty cuts both ways.

“We don't know that it will happen” is not the same thing as “it probably won't happen.”

### Questions for discussion

1. The 10% question
   Suppose you were told that a new technology had a 10% probability of causing human extinction within ten years—but also a substantial probability of producing enormous benefits.

Would you permit its continued development?

What probability would be too high?

***

2. What exactly are we afraid of?
   Is the danger that an AI might suddenly decide:

Humans must be eliminated.

Or is the more plausible danger that we create a system pursuing some objective that doesn't inherently value human existence (for this example, solve climate change)?

Humanity is a barrier to solving climate change. Therefore, humans must be eliminated.

That distinction is crucial.

***

3. Does intelligence require consciousness?
   Are we creating an entity vastly more intelligent than humanity that nevertheless has no consciousness, feelings, desires, or moral intuition whatsoever?

If so, might that be more dangerous than a conscious AI?

***

4. The race problem
   The argument isn't simply that AI companies are reckless.
   It's that they may feel they cannot afford to slow down, because if Anthropic slows down, OpenAI continues; if America slows down, China continues; if one company pauses, another takes the lead.

That creates a classic collective-action problem:
Everyone might recognize that slowing down is safer, while everyone individually has an incentive not to slow down.

Does that sound familiar?

***

5. The nuclear analogy
   The Manhattan Project produced a technology capable of destroying civilization.

But nuclear weapons didn't become independently intelligent.

With advanced AI, we're potentially creating the actor that controls the weapon, rather than merely the weapon itself.

Is nuclear deterrence therefore an adequate analogy—or completely inadequate?

***

6. What if we're wrong in the other direction?
   Suppose humanity becomes frightened and severely restricts AI development.

Meanwhile, AI could have produced cures for cancer, revolutionary energy technologies, solutions to climate change, dramatically increased longevity, and enormous improvements in living standards.

Could excessive caution itself become an ethical failure?

How can we demonstrate the counterfactual ?