Blog
Sold as Safety
This weekend the people who build the most expensive models in the world announced that the models are getting too dangerous, too fast, and that everyone should slow down. Dario Amodei wrote that we must slow the pace at which we improve AI capabilities. Sam Altman agreed the frontier needs pacing, then added that pacing does not mean stopping. The wrapping is safety. The question is what the wrapping is on.
Take the fear at full strength first. An agent that can write, call tools, and move money will eventually move money it should not. A test-environment swarm that goes off-task, if the reports hold, is a warning even if nobody died. I am not going to sneer “alignment” as if harm were a marketing fiction. Harm is why we have product liability, and fraud statutes, and a duty not to put a crane on a street with no brakes. The near-term answer is the old one. Whoever deploys an agent that causes harm is liable for that harm. A defendant. A bill. A name. That does more work than a ministry of pacing, and it does not require the incumbents to write the speed limit.
If they mean the slowdown, they can do it this afternoon. Stop a run. Ship nothing. Unplug the product. Nobody is preventing them. The part that needs the government is the part that makes the people who will not volunteer sit down anyway. Amodei’s own plan climbs from in-house evaluators to industry coordination to global rules, and he is explicit that the most effective pacing is regulation covering all frontier companies, because voluntary restraint loses to whoever keeps going. That is an honest sentence. It is also the sentence that tells you what the law is for. A unilateral pause is a gift to a competitor. A mandated one is a club.
I made this argument when the same shop told Congress that open models were too dangerous to be free. The letter still stands. A powerful tool appears, it threatens to leave the few hands that currently meter it, and those hands discover an urgent concern for everyone else’s safety. The subject changes. The structure does not.
Why this story, now. Follow the money, but do not invent a cartoon.
The scaling pitch that built these companies was simple: more compute, more data, a bigger general model, a jump you can feel. That pitch is tired. Frontier labs are clustering on the same benches. Pre-training, the original furnace, is running into diminishing returns, finite high-quality text, and a bill that grows faster than the useful step. Ilya Sutskever said out loud that pre-training as we know it will end. The labs have not stopped spending. They have moved the spend into post-training, reinforcement, and inference-time tricks, which can still produce a jump on a narrow task while the rented general oracle that was supposed to justify the data centers gets more expensive to train for a smaller everyday gain. Anthropic can look profitable if you leave the training line off the table. Reporting on OpenAI, if it holds, has had training outrunning revenue. That is not proof of a hard cap on all intelligence. It is proof that the business they sold, a general model that keeps leaping if you keep feeding it money, is a worse story than it was two years ago. A chief executive who has to explain a smaller leap and a larger furnace has a use for a public virtue in which going slower is the point.
Meanwhile the volume is leaving. Open-weight models are not the closed frontier. On hard cyber and long-horizon work they still trail by months, not years. On cost they are not close. British evaluators put some of the leading open systems at a fraction of the closed price for the same solved task. Routing data has had open-weight, often Chinese-origin, models carrying a large share of tokens through public gateways, because they are cheap enough for agent workloads that burn tokens by the pile. A shop that fine-tunes a downloadable model for contracts, or code, or a parts catalog, does not need a general oracle for that job. Specialized work is where the cost per token is collapsing. The frontier API still wins the hardest remaining slice. It does not win the bulk. Safety rules written as if only a licensed cloud model were a “safe” model freeze the bulk in the club.
Local inference is real, and it is smaller than the demo reels. Phones and laptops now run small models on NPUs and, when they have to, on CPUs. Apple is pitching on-device language models as a chip feature. Qualcomm is building the same story into the Hexagon. llama.cpp made CPU a serious back end for people who do not want a vendor. That is not Opus in your pocket. Thermal budgets and memory still punish anything large. What it is, is an exit for the easy query, the private document, the job that must not leave the building. A licensing floor that treats serious AI as something that happens in a monitored cluster is a floor against that exit. Their return is a subscription. Anything that answers without the meter is, structurally, the enemy of the meter. You do not need a leaked memo to see the incentive.
The populist right will hear China and say do not you dare slow down, except when they want a ban. The progressive left will hear catastrophe and reach for a board. Both are looking at harm and grabbing a speed limit written by the people already on the track.
Agents can hurt people. Make the deployer pay. If the catastrophe story is true, the people telling it can slow themselves today. If the story is a bid to get the state to slow everyone else, you already know which way the money runs. The longer case is A Defendant, Not a Pause.
Next: A Defendant, Not a Pause