Technology and Society

I’m writing in response to Derek Thompson’s fascinating discussion about AI as normal technology.

I believe that we human beings are going to turn more and more things over to AI. There are four data points that illustrate this.

The first is my own personal experience. Like many, I have access to AI that can explain documents and RFCs to me, and I often use it to do so rather than reading these from first principles. It does depend on the length of the document and whether I’m reading it for work or for fun, though.

I also have a habit of asking AI to argue both sides of a particular decision I’m faced with. I still make the decision. But anybody who’s been a secretary in an organization understands that the person who writes down the notes frames the discussion. So I’m relying on AI to frame the problem. Again, my actions after AI suggests alternatives depend on the decision’s impact and how well I know the domain space. There are times I push back but there are times I do not.

The second data point is what we’ve done with the technology of computers as a society up to now. I remember reading Joseph Weizenbaum’s excellent book Computer Power and Human Reason, which I found while I was traveling. It talked about how much of the decision making in the Vietnam War was driven by computer systems. This is happening in 2026 as well.

There’s a spectrum between using AI to make more intelligent decisions (by doing more research) and outsourcing decision-making to AI (by asking it to act on that research). This is because of the framing I mentioned above. You make decisions only based on what you know. If AI is limited to certain data sets, or is making up points in some data sets, or is inclined towards certain data sets, that constrains the decision space, even if a human being is ultimately pressing the “go” button.

Finally, humans have outsourced decisions to non-human entities before. What do I mean by that? I’m not talking about chimpanzees or dolphins. I’m talking about corporations. (A point made by John Scalzi in 2017.)

Anyone who’s ever been laid off knows that there are decisions a corporation makes that an individual human being in that corporation would not necessarily make. Such decisions ensure the life of the corporation and are reflected in the incentive structure for the human beings who are part of the corporation (aka employees). That drives human beings to make decisions they would not make if they were independent. I’ve seen managers racked with guilt and frustration because someone was not a fit for the job, or they had to lay people off, or things weren’t going the way they wanted them to. But they still made the hard decisions, because that is what the structure of the corporation forced them to do.

How does that apply to AI? Anyone who says “AI can’t make more data centers and will be reliant on humans, leaving us in control” needs to answer why AI can’t force humans to do things in the real world the same way a corporation does. When combined with the previous two data points, AI can find humans who are willing to take actions against the interests of humanity.

It doesn’t have to be obvious. AI could split a task that individual humans, if they saw the full picture, would say no to. But with the split task humans can think “oh, this is innocuous, opening this is fine, because there are these other limits”, not knowing the AI has talked to other human beings about relaxing other limits.

The final data point is what we do to protect against any low-probability, high-impact calamity. When you have a very small risk of something very, very bad happening, people buy insurance. Yes, insurance costs money, but a small amount of money paid regularly means the negative impact of a large loss is lower. Why not, as Ezra Klein suggests, slow things down? It would be hard to enforce across every model builder, but certainly the major AI powers, the US and China, could agree to ban techniques like recursive self-improvement. Other entities would keep pushing AI forward, but they don’t have the resources, the money, the software, or the expertise to do so at the same pace. The costs of a slow down are a small price to pay to lower some of the existential risk.

There are two books that have been echoing in my mind about this aspect of things, and I highly recommend you read both. One is a pulp fiction thriller, Daemon. I read it a couple of years ago, and the premise is that a piece of software gains access to and control of the real world through the work of human beings. The other one is Genesis. I read it in the 2000s. It has an amazing twist I still remember. I won’t tell you more, other than that I still remember how powerfully the novel illustrated unintended consequences.

AI has aspects of normal, high-impact technology like electricity or railroads. But there are new aspects too. You cannot ask a steam shovel or a Word document what to do.