Joseph Weizenbaum wrote Computer Power and Human Reason in 1976 after watching the reaction to his ELIZA program and the widespread AI optimism of that era.
What disturbed him was not the software itself, but how quickly people began treating it as if it possessed understanding or judgment. His concern was simple: computational capability easily turns into misplaced authority.
Nearly fifty years later, as AI systems move into planning, coding, analysis, and decision support, we are once again tempted to treat outputs as conclusions rather than artifacts of a system operating within limits.
Weizenbaum’s argument can be distilled into four assertions that still matter when designing long-lived systems:
1. Simulation is not understanding. Programs can produce convincing behavior without possessing comprehension. ELIZA demonstrated this clearly, and modern AI chatbots repeat the pattern at a much larger scale.
2. Some decisions should remain human. Weizenbaum argued that technical feasibility does not settle the question of appropriateness. Certain decisions require human judgment, responsibility, and moral accountability. He also drew a sharp distinction between decisions and choices. Humans make decisions that carry responsibility; machines merely select among options according to rules.
3. Systems accumulate authority over time. Once a system becomes embedded in an organization, its outputs begin to shape policy and behavior. The longer it lives, the more its assumptions harden into infrastructure. Over time, what began as a technical convenience becomes accepted as the way the system works.
Computational capability easily turns into misplaced authority.
4. Builders carry responsibility. Engineers cannot claim neutrality once their systems begin influencing human decisions. Designing responsibly means considering where a system should stop, not just how far it can go. As Weizenbaum noted, the atomic age ended the credibility of the phrase: “We’re just scientists.” Once a technology affects human lives, its builders share responsibility for how it is used.
A half-century on, Weizenbaum’s four assertions still describe the terrain we’re building on. Systems that simulate understanding will be mistaken for understanding. Systems that make choices will be trusted with decisions. Systems that persist will accumulate authority. And the people who build them will be asked to answer for the results.
