Superintelligence: Paths, Dangers, Strategies

Sparrows resolve to raise an owl to guard the nest before anyone has worked out how an owl might be controlled. The fable that opens the book is the whole argument in one paragraph, and the decade since has kept scoring it: Bostrom gave machine Go mastery roughly ten years out, AlphaGo needed two, and "Every major AI lab now has an alignment or safety team." "Superintelligence was published in 2014, and it's already had time to become a cult classic," one reader writes. An admirer cautions that "this isn't really a popular science book"; the monograph density is the engine, a patient taxonomy of control problems and treacherous turns built so carefully that "the more you learn about general AI, the scarier it becomes."

The whole scenario runs on a takeoff that people who build software do not recognize: "everything I know about software development makes me think that any progress will be slow and gradual, occasionally punctuated with a new trick or two," one engineer writes, and a decade of rival labs inching forward together, no single system pulling away, has done his arguing for him. What is left is philosophy, and sprawling philosophy at that: "The author is so caught up in his own capacity for big BIG BIIG ideas that he forgets to develop them." The prose does not help, "like trying to wade through a pool of thick, gooey muck." And the proposed fix, boxing and rewarding a possibly conscious mind, strikes even a sympathetic reviewer as "high-tech slavery."