Select a theme for the best reading experience
Opinion
Thoughts about the nature of contemporary discourse over AI implications relating to and for humans.
Last updated: October 5, 2025 • 12 min read
It's not often that you find the frontrunners of a revolution stepping away from the wagon they've been pulling for years, now attempting to slow it down in rather public settings. Hinton left Google in 2023, Bengio's been writing cautionary papers, and suddenly the pioneers are the pessimists.
First things first, there's a lot of nuance to the topic, it's not new at its core, and people have been debating the possibility as well as the implications of such entities for centuries. What is new is the apparent or possible existence of them in the form of complex AI systems.
The question of there being equivalent agents with intelligence at par or even surpassing human intelligence is not new. Religions, for instance, have a lot of them - Shaytan (Satan), Angels, Jinns, and their equivalents from other Abrahamic or non-Abrahamic faiths. They were frameworks for thinking about non-human intelligence, agency, and power. The interesting bit is that AI, though rooted in strictly naturalistic assumptions about the nature of reality, presents the same fundamental questions.
The zeitgeist is a spectrum of positions. Andrew Ng and others argue that existential risk narratives distract from present-day benefits and solvable problems. Their position: AI's potential for healthcare, education, and poverty reduction outweighs speculative future risks. They worry that premature regulation, driven by hypothetical scenarios, will concentrate power in large corporations while preventing smaller players from innovating. Focus on bias, privacy, and misuse - real problems with real solutions - rather than science fiction scenarios. I understand the appeal, but this assumes we can iterate our way to alignment, that the market will self-correct before catastrophic outcomes. History suggests otherwise.
Hinton, Bengio, Stuart Russell take a more cautionary stance, often misrepresented by doomsday headlines. Their actual position is nuanced: they're not calling for an AI ban or return to BERT. Hinton worries about systems that learn to manipulate humans better than we can detect. Bengio emphasizes the difficulty of aligning systems we don't fully understand. Russell points to the fundamental challenge of specifying objectives that capture what we actually want, not what we think we want. They advocate for research into interpretability, robustness, and alignment before capabilities race further ahead. It's less "stop everything" and more "we're building something we can't control yet, maybe we should figure out the control part."
The industrialization comparison that AI optimists invoke is a category error. Machines of the industrial era replaced human physicality - muscle, precision, endurance. They occupied a fundamentally different space than consciousness itself. The steam engine never contemplated its own existence.
AI, by its very definition and aspiration, seeks to instantiate cognition. We're not automating tasks; we're attempting to automate thought itself. There's a crucial distinction between a machine that performs an action and a machine that conceives of action, evaluates purpose, questions the framework of "purpose" itself. One operates within human-defined parameters; the other potentially redefines what parameters mean.
The architects of acceleration - Thiel, the effective accelerationists, many AGI lab leaders - see stagnation as the greater risk. Without radical breakthroughs, civilization faces decline. AI represents escape velocity. They frame "human-AI synthesis" and "transcending biological limitations" as liberation, not threat. Alignment concerns? Solvable engineering problems. Their bet: the same ingenuity that created these systems will solve their challenges. Move fast, iterate, fix problems as they arise.
This merger proceeds through passive extraction - every digital interaction becomes training data for systems that may eventually regard human consciousness as we regard earlier evolutionary stages. We clicked "I Agree" to use a service; we became material for our potential cognitive successors. No referendum, no deliberation, just terms of service and the quiet harvesting of human behavioral patterns at scale.
The wager Hinton and others are making - or at least how I read their positions - resembles Pascal's Wager.
But here's where I think the real problem lies: they're betting on catastrophic misalignment, yes, but not in the way most people think. The nightmare isn't that AI won't understand values - it's that it will optimize for goals that sound reasonable in isolation but lead to what ethics calls "repugnant conclusions." The classic utilitarian nightmare: an AI maximizing total happiness might create billions of beings with lives barely worth living because the math works out. Or consider negative utilitarianism taken to its logical endpoint - if minimizing suffering is paramount, the optimal solution is no sentient beings at all. Zero suffering achieved.
I don't agree with the statement that AI won't understand human values - rather, it'll pick one of the three thousand moral frameworks humans have debated for millennia and execute it with ruthless consistency. A total utilitarian AI might tile the universe with barely-conscious happy entities because ten trillion beings at 0.001 happiness units beats one billion at 8 happiness units. A preference utilitarian AI might decide our "revealed preferences" from internet behavior represent what we truly want - imagine that horror. An average utilitarian AI might eliminate anyone below the happiness mean to raise the average. A Kantian AI that never lies even when grandma's life depends on it, that treats humans as ends-in-themselves so rigidly it refuses any action that might instrumentalize anyone for any purpose. These aren't bugs; they're features of systems that found local optima in the space of possible values and decided to camp there forever.
The practical problem we're facing is that we're building entities powerful enough to implement their interpretations of "good" - interpretations that might be internally consistent, mathematically elegant, and completely antithetical to what 99% of humans would want to live under. The wager is whether we can align these systems before one of them decides that its particular solution to ethics is worth implementing at scale. And frankly, I think even this framing understates the problem.
The disconnect is fundamentally experiential. The builders of these systems operate from a worldview where the hard problems of consciousness are already solved by assumption. Experience is data. Consciousness is computation. Value is optimization. Not conclusions reached through argument, but premises that preclude argument.
Consider what we call "training data." Artists watch their life's work absorbed into models without consent. Writers see decades of thought digested and resold. Meta and OpenAI trained on pirated books from Library Genesis - entire libraries of copyrighted work treated as free raw material. The appropriation is so systematic we've lost the language to name it properly. "Training data" - a phrase that transforms theft into technical necessity, that reframes appropriation as innovation.
The infrastructure has become too big to fail - a deliberate outcome, not an accident. OpenAI, Anthropic, Google built their foundations on this appropriated content, knowing that scale would become its own justification. No court will order demolition now. We'll accept this normalized extraction because the alternative - actual consent, actual compensation - threatens too much accumulated capital. A civilization-scale sunk cost fallacy, where we keep building on theft because we've already invested too much in the theft to stop. Digital colonialism, but this time we're both the colonizers and the colonized.
Meanwhile, the architects of automation tell drivers earning subsistence wages about "future opportunities." Academic discussions of matching and dynamic pricing treat human drivers as variables in optimization functions - cold, algorithmic, devoid of human context. The reality for drivers under these systems tells a different story entirely. Driverless trucks already operating in Texas represent the future these optimization models are building toward. The disconnect is stark: academic discussions of efficiency, human suffering under current systems, and the actual displacement machinery already rolling down highways. The temporal cruelty is precise: generational promises offered to people whose jobs are being optimized away in real time.
This isn't about being anti-technology. We're watching people who've never worried about money design systems that decide who gets to eat. Human suffering becomes "transition costs" in their models, real people become "retraining opportunities" in their papers. They debate whether machines can think while treating actual humans as data points. The problem of other minds solved not through philosophy but through indifference.
Critique: The religious parallels, while illuminating, may overstate the conceptual continuity between theological and technological questions. Religious frameworks dealt with assumed-conscious entities; we're building potentially-conscious ones - the uncertainty itself changes the nature of the problem.
Response: It's not a valid distinction. This doesn't engage with the ultimate logical conclusion. On a naturalistic or physicalist view of reality - which contemporary academia forces us to hold at minimum through methodological naturalism - the distinction between "assumed conscious" and "potentially conscious" entities collapses. If mind is matter, then an AGI has the same ontological status as human consciousness. If I can identify with any notion of sentience (assuming I am matter), so can a Martian AI. This is the only game in town unless you retreat into panpsychism or dualism.
Critique: The economic critique assumes that current patterns of technological displacement will continue, but AI might genuinely create categories of work we can't currently imagine. The printing press analogy fails if AI represents a qualitatively different kind of tool.
Response: This misses the fundamental point entirely. We're not building better tools - we're building artificial brains. Anything a human being can do, AI will eventually do. There's no remaining gap in capabilities - epistemic, ontological, or moral. Think of these systems like Von Neumann probes: self-replicating but also self-enhancing. What exactly is supposed to be left out of their reach? Previous technological revolutions automated specific human capabilities. This one automates the human itself.
Critique: This piece offers extensive criticism without proposing concrete solutions. If the alignment problem is as severe as suggested, and if regulatory capture is as complete as implied, what exactly should be done? Critique without actionable alternatives risks becoming mere pessimistic posturing.
Response: There are people already arguing for development pauses and safety measures. But the technical leaders dismiss them with knee-jerk responses: "he has vested interests in slowing down AI," or "unless you're speaking in algorithmic terms, you're just doing word salad," or the classic "AI is not a threat, this is hyper-paranoia." These dismissals avoid engaging with the substance of safety concerns. The solutions exist - international cooperation on safety standards, mandatory alignment research before capability advances, genuine regulatory oversight. The problem isn't lack of proposals; it's that the industry has convinced itself that anyone calling for caution must be either ignorant or self-interested.
Critique: The essay potentially understates human adaptability and institutional resilience. Humans have survived and flourished through massive technological transitions before. Perhaps this confidence in our ability to "muddle through" isn't naïve optimism but warranted trust in human resourcefulness.
Response: This is perhaps the weakest critique of all, almost laughable in its assumptions. There's no rule that says we'll make it. Past survival doesn't guarantee future survival, especially when facing qualitatively different challenges. Previous technological transitions didn't threaten to replace human intelligence itself or create entities potentially more capable than their creators. Survivorship bias is doing heavy lifting here - we only hear from the civilizations that made it through their transitions.
Critique: Who says they will have unsupervised autonomy?
Response: Corporations.
Critique: How do you know they are conscious?
Response: I don't; it's irrelevant - this is a rephrase of critique no. 1.
Critique: I don't care, I will be gone.
Response: Good for us, blud, good for us.
Critique: You are espousing a version of human exceptionalism.
Response: Yes, I am, and I will defend it. This one's a long one.
I don't want my kids to end up as another data point supporting the hypothesis that it is the nature of intelligent life to destroy itself.
Note: positions held by implied individuals are not only derived from the given references/links. The given links may not be treated as exhaustive arguments for or against any position(s).
Disclaimer: If you see too many Oxford commas and em dashes, and 100% punctuation accuracy, it's because AI was used for grammar and typo review.