If there was any doubt that science is driven by people politics as much as anything else, look no further than James Gleick’s 1987 book, Chaos: Making a new science.
The book chronicles the history of chaos theory; but “chaos” is also a good word to describe the scientific community’s embarassingly slow acceptance of the findings and tools of this new mathematical subfield.
The book held particular interest for me because of an unsolved mystery in a research paper I wrote in college, Weather forecasting by computer. Edward Lorenz, who published the first research on what would become chaos theory, calculated in the late 1960s that “even with perfect models and perfect observations, the chaotic nature of the atmosphere would impose a finite limit of about two weeks to the predictability of the weather.” Despite this, I was reading brash predictions in books published in the early 1980s that we would soon be able to forecast the weather months or years into the future. Why did it take more than a decade for this fundamental mathematical result to make its way even to experts writing about weather forecasting?
Gleick wondered the same thing. The fact that his conclusions took the form of an entire book is testament to the many factors at play. (I was a bit relieved to confirm that I wasn’t just missing something obvious.)
Part of the answer is that chaos theory was outside the scope of existing academic disciplines, almost by definition. It tried to make sense of problems that couldn’t be solved using traditional mathematics — the very problems that most researchers (and entire science departments) stayed away from because the chances of progress seemed slim. Over time, disciplinary boundaries developed such that most of these problems were not considered valid topics in physics, biology,… and even weather forecasting.
A second part of the answer is that many of the important results of chaos theory themselves defined limitations on what is possible to know or achieve, especially when seen through the lens of traditional approaches. Scientists and other leaders didn’t want to believe these pessimistic claims, and they were easy to ignore when coming from a suspicious fringe group of career-insensitive mathematicians.
A third part of the answer is that even when the essential properties of chaos theory had been well established by mathematicians, the theory was not useful to mainstream scientists until practical mathematical tools were developed. Several important mathematical results eventually helped to show how disparate data sets all displayed chaotic “bifurcations” and “period doublings,” for example. As scientists were given more concrete patterns to look for, evidence of chaotic behavior became increasingly visible to them.
And yet a fourth part of the answer is that the main tools used to investigate chaos theory — computers — were new and unfamiliar to mainstream scientists. Lorenz was one of very few theorists in the 1960’s who had access to expensive computer time (and the knowledge to use it). And although rigorous mathematical proofs were eventually found for many components of chaos theory, for many years the most important results were simply the outputs of clever computer programs. Running experiments like this via numerical simulation was a totally new approach. Scientists and mathematicians had every reason to be skeptical.
At the time Gleick’s book was published, chaos had finally become broadly accepted in science and had led to a few high-profile applications such as heart pacemakers. Yet even now, 20 years later, chaos theory is not part of the standard curriculum at any level of school. I studied it for a few weeks in high school as part of a special end-of-year diversion; and in college as an elective math course that was only offered one semester every other year. And I went to very progressive schools. When Steven Wolfram unveiled his “new kind of science”, non-experts missed the fact that he was talking about this same line of research. The new science is still in its infancy.
