Premature Optimization
Why do the successful often fail to repeat their success? Because success is often the source of failure.
Success is a form of optimization—a state of optimal profits, or optimal fitness, or optimal mastery. In this state you can’t do any better than you are. In biology, a highly evolved organism might reach a state of supreme adaptation to its environment and competitors to reach reproductive success. A camel has undergone millions of revisions in its design to perfect its compatibility with an arid climate. Or in business, a company may have spent many decades perfecting a device until it was the number one bestselling brand. Say it manufactured and designed a manual typewriter that was difficult to improve. Successful individuals, too, discover a skill they are uniquely fit to master—a punk rock star who sings in an inimitable way.
Scientists use a diagram of a mountainous landscape to illustrate this principle. The contours of the undulating landscape indicate adaptive success of a creature. The higher the elevation of an entity, the more successful it is. The lower, the less fit. The lowest elevation is zero adaptation, or in other words, extinction. The evolutionary history of an organism can thus be mapped over time as its population begins in the foothills of low adaptation and gradually ascends the higher mountains of increased environmental adaptation. This is known in biology and in computer science as “hill climbing.” If the species is lucky, it will climb until it reaches a peak of optimal adaptation. Tyrannosaurus Rex achieved peak fitness. The industrial-age Olivetti Corporation reached the peak of optimal typewriter. Sex Pistols reached the summit of punk rock.
Their stories might have ended there with ongoing success for ages, except for the fact that environments rarely remain stable. In periods of particularly rapid co-evolution, the metaphorical landscape shifts and steep mountains of new opportunities rise overnight. What for a long time seemed a monumental Mt. Everest can quickly be dwarfed by a new neighboring mountain which shoots up many times higher. During one era dinosaurs, typewriters, or punk rock are at the top; in the next turn, mammals, word processors, and hip-hop tower over them. The challenge for the formerly successful entity is to migrate over to the newer, higher peak. Without going extinct.
Picture a world crammed with nearly vertical peaks, separated by deep valleys, rising and falling in response to each other. This oscillating geography is what biologists describe as a “rugged landscape.” It’s a perfect image of today’s churning world. In this description, in order for any entity to move from one peak to a higher one, it must first descend to the valley between them. The higher the two peaks, the deeper the gulf between them. But descent, in our definition, means the entity must reduce its success. Descent to a valley means an organism or organization must first become less fit, less optimal, less excellent before it can rise again. It must lower its mastery and its chance of survival and approach the valley of death.
This is difficult to any species, organization, or individual. But the more successful an entity is, the harder it is to descend. The more fit a butterfly is to its niche, the harder it is for it to devolve away from that fit. The more an organization has trained itself to pursue excellence, the harder it is to pursue non-excellence, to go downhill into chaos. The greater the mastery a musician gains for her distinctive style, the harder it is to let it all go, and perform less well. Each of their successes binds them to their peaks. But as we have seen, sometimes that peak is only locally optimal. The greater global optimal is only a short distance away, but it might as well be forever away, because an entity has to overcome its success by being less successful. It must go down against the gain of its core ability, which is going uphill towards betterment. When your world rewards hill climbing, going downhill is almost impossible.
Computer science has borrowed the concept of hill climbing as a way of discovering optimal solutions to complex problems. This technique uses populations of algorithms to explore a wide space of possible solutions. The possibilities are mapped as a rugged landscape of mountains (better solutions) and valleys (worse). As long as the next answer lands a little “higher uphill” toward a better answer than the one before, the system will eventually climb to the peak, and thus find the best solution. But as in biology, it is likely to converge onto a local “false” summit, rather than the higher global optimal solution. Scientists have invented many tricks to shake off the premature optimization in order to get it to migrate to the globally optimal. Getting off a local peak and arriving at the very best, repeatedly, demands patience and surrender to imperfection, inefficiency, and disorder.
Over the long haul, the greatest source of failure is prior success. So, whenever you are pursuing optimization of any type, you want to put into place methods that prevent you from premature optimization on a local peak: Let go at the top.
