Mysteries of financial risk, plus: House on fire

The idea floating around (one version proposed by McCain) that the government should buy up "troubled" mortgages is as misguided notion as they come. "Troubled" itself is hard to define. Is it determined by the borrowers' difficulties in making loan payments? Or by loan default? Or is it just a mortgage "under water," valued at more than the house it's attached to? "Troubled" should be limited to, at most, the first two cases.

The housing crisis -- or rather, the house financing crisis -- will have, not one, but many endings, covering a range of possibilities:

• Borrowers paying reliably on mortgages "under water"
• Borrowers with payment difficulties who renegotiate their loans (lower interest rate)
• Borrowers in default who might renegotiate or just move out, to rentals
• Borrowers in foreclosure who must move out, or stay and rent with option to buy

At this point, the last category is small, a little over a percent of mortgages. The range of options is enough to make simply throwing people out on the street an unnecessarily harsh choice. Banks and lenders will not want to sit on unoccupied, non-income-generating property in any case. Both lenders and borrowers will unavoidably take some losses along the way.

Except for directly intervening with borrowers with Fannie and Freddie loans, it's hard to see what role government should take here, except to act as a catalyst. Government should certainly not be engaged in perpetuating the housing bubble; for example, in trying to prop up house prices or encouraging any more subprime lending. If it does anything for the housing market, it should be terminating the ingredients that went into the bubble in the first place.

---

The general financial crisis, centered in the credit markets and impacting others (like the stock market), was certainly triggered by the weakness in the subprime mortgage market and exacerbated by falling house prices across the board. But the financial system, as evolved over the last thirty years, has developed intrinsic weaknesses of its own that falling house prices merely exposed. Those dangers are embodied in excessive debt and rationalized in turn by faulty theories about controlling risk.

Many of the supposed culprits -- mortgage bonds and "derivative" securities (essentially, complex, composite repackagings of existing securities); the non-existent "deregulation" of Wall Street; and the alleged merging of investment and commercial banking -- are bogus. These supposed factors are either not real or not capable of producing an unforeseeable credit crisis of this magnitude.

Over the last generation or so, the financial world, American and non-American, the regulated and the regulators, has developed an unhealthy and misplaced confidence in its ability to quantify and manage risk. The crisis we see unfolding now has nothing in the slightest to do with "fraud" or malfeasance on any individual's part.* Traditional regulation is designed to deter and punish such misbehavior, which is multiply times over illegal anyway. A crisis of this type is a result of collective misjudgment and collectively-held false ideas about risk, mixed with a certain level of hubris.

Viewed this way, our present financial troubles start to look less like a crime caper and more like the failure of a complex technological system, like the explosion of the space shuttle Challenger or the sinking of the Titanic. Megan McArdle had an interesting post on this point a while back.

To follow Megan, it's especially enlightening to compare the failure of financial risk management with the Challenger explosion, on which topic she recounts the story in Richard Feynman's famous What Do You Care What Other People Think? and captured in detail in Feynman's appendix to the Rogers Commission report. The key comparison: the different ways that different people interpreted "small" risks. Based on decades of prior experience with rockets, the engineers knew in their bones that the "small" risk of a fatal shuttle accident was about one in a 100. (And we know now, with over 25 years of shuttle experience, that they were right.) But they couldn't articulate and defend their point of view in the face of managerial and political figures, whose notion of "small" was more like one in a 100,000 or one in a 1,000,000. Each near-fatal incident, instead of being interpreted correctly as a warning, was instead rosily misinterpreted as "great, we survived another close one" and falsely built up NASA's confidence.

That difference -- "small" as one in a 100 or one in a 1000, versus "small" as one in a million or ten million -- is precisely the difference between the "wild" and the "mild" in risk, "Extremistan" versus "Medocristan." Readers of previous posts on finance and statistics will know of Nassim Nicholas Taleb's The Black Swan and all about such misperception of risk. A one-in-a-hundred incident is something likely to happen more than once in a person's lifetime. A one-in-a-million or ten-million incident is unlikely to happen in anyone's.

It makes the crucial difference to social systems created and run by humans. All of us, especially the college-trained, are prone to the Tyranny of the Cookbook, falsely believing that some answer is better than no answer, even if that answer is wrong. Much of the financial world still wrongly assumes the mild risk of Medocristan and rationalizes the powerful evidence to the contrary by handwaving.
---
* It has even less to do with "corruption," something outside of Wall Street's power, since that requires the granting of political favors. You have to look to K Street (in Washington) for that.

Climate science after Kyoto

With the Official Science monkey gone from our backs, what can we do with climate and climate science?

Over the last year, I've outlined on this blog a set of open questions that are frequently ignored and often not even seen correctly as questions, but that need to be answered if we're going to talk sense about "climate" and "climate change." Not able to predict weather beyond about two weeks ahead, we need a simplified and abstracted definition of "climate" whose "state" can be defined, analyzed, and predicted with some confidence. The fallacies of temperature averaging and "climate parameterizations" were a failed attempt to do this. Whatever notions of "climate," "climate state," and "change of climate state" we end up with will have to withstand - as temperature averages and the "hockey stick" cannot - probing criticism and emerge free of the hefty list of fallacies associated with the "global warming" hysteria.

In retrospect, the generational dead-end of climate and allied sciences with "global warming" and general circulation models (GCMs) can be viewed as an attempt to force a premature integration of theory and observation. It would be best for all involved if theory and observation were to remain aware of one another, but go their separate ways until such time as they have sufficient means to meet each other honestly. Until such a synthesis is possible, it's best to maintain a pluralistic and agnostic attitude about the Big Picture, resisting the forces bent on our "salvation" from the "evils" of industrial civilization or plying us with supposed alternatives to "knowing" nature.

What follows is a personal and partial view, informed by this failure to get a grip on these issues and by my own scientific experience along the edges of the problem and in related fields. It should not be taken as definitive or complete.

Theory: It's a really hard problem. The Earth's climate is the most complex scientific problem ever posed and almost certainly unsolvable in its full generality. Any progress we make with this problem will therefore necessarily involve approximations. The essential point is that, if we're going to attempt actual predictions, we need better and controlled approximations at every step. These are currently lacking.

Theory: Science is hypothesis and deduction. That is, it's not just a piling up of facts. Certainty of conclusions requires control of assumptions and reasoning.

There is thus an important role for mathematical deductive modeling, with simplified and controlled approximations applied at every step. The ideal should be to make these mathematically simplified and controlled models closer and closer, with each step, to the real climate. The failure of the "parameterized" GCM approach underscores the need to keep the modeling within controlled approximations at each step and not jump into the deep end of the pool right away.

Theory: Don't BS - simplify and smooth. A revealing way to look at the "climate state" problem is to grasp the motive behind "climate parameterizations": it was to "force closure" on the dynamical-structural equations of climate. In general, there are never enough equations to match the number of unknown variables. "Forcing closure" on the system means guessing or making up extra equations to close the gap.

But the gap could equally well be closed the other way: reduce the number of variables. A simplified "climate state," less complex than "the exact, instantaneous state of the whole atmosphere," is just such a proposal. It's also likely that such a state will not only involve flows and topology in space, but time and space integrals of the basic variables (equivalent to what statisticians call cumulants). Such integrals are usually better-behaved than the original variables.

Theory: Boil, mist, and trouble. Climate is chaotic, in the technical sense: exponential sensitivity to errors in initial conditions. Alternatively, climate is essentially nonperiodic, and not all climate disturbances die away. The atmosphere is a fluid, in the physicist's sense; its chaos is turbulence. Turbulence is the largest unsolved problem in physics. A partial or complete solution would have immediate impact on many areas of science and engineering, pure and applied, theoretical and practical - everything from understanding convection in planetary atmospheres and stars to improving your airplane or boat ride to reducing turbulence losses in your car engine.*

Climate needs new and better techniques for coping with chaos. Many such techniques have been developed in the last 25 years in various areas of science, but they haven't penetrated far into the climate world, partly because of the paralysis induced by Official Science. They include exceptionally relevant techniques like the following.

• Renormalization. This technique is a powerful generalization of the dimensional analysis we learned in school. (It's sometimes goes under the guise of "homology" or "rescaling.") It relates one mathematical problem posed at one set of space and time scales to a different problem at a different set of scales. Sometimes, impossible problems posed at one scales can be recast into other problems at different scales, and those different problems are solvable, either exactly or by controlled approximation.
  
  Renormalization for climate means imagining a scale at which decades, centuries, or even millennia seem modest and slow cycles like El Niño, say, wink by in rapid succession. On those scales, we can see more clearly the invariant and almost-invariant structure that must define, at a deeper level than everyday weather, what "climate" is.
  
  
• Dynamical reconstruction of phase space. This requires some contact with observed climate (see below), but the essential technique amounts to isolating the relevant degrees of freedom in the very complex climate system, the ones that operate on scales of tens to thousands of miles. Only a small subset of the possible changes in the climate system are actually important. Isolating them is a big step toward defining "climate" in a simplified sense. It will undoubtedly involve flows of heat, water, etc. (not local temperatures or humidities) and how they're connected in space (their topology).
  
  
• Non-Gaussian statistics, for extreme weather analysis. This is an application of the great progress that has been made in understanding how energy and other conserved physical quantities move through "open" systems, like the climate. Again, the issue straddles both theory and observation. People just have to stop assuming Gaussian (classical central limit or bell-curve) conditions in analyzing weather "events." There's never been any reason to do so.
  
  
• Pattern formation. This is an intersection of renormalization, non-Gaussian statistics, and "complexity," as an earlier posting discussed. The locus classicus for these techniques is understanding the perpetually landsliding sand pile. (There's even a cute book on the subject by Per Bak.) In complex, open systems with "flow-through" of air, water, and heat (or sand grains for that matter), long-range patterns with "almost" (but never quite!) repetitive behavior form and dissipate over and over - just like the weather: cyclones, storms, fronts, and so on.
  
  Pattern formation is especially germane to understanding clouds - their nature and lifecycle - better. Clouds are the most important feature of climate not easily captured by simplified models; convective turbulence is actually secondary in importance, at least for heat flow, although it's still crucial for the complete picture. And the big, difficult pieces of climate - clouds, turbulence, water transformations - are all linked together. Convection doesn't just transport heat; it lifts water vapor to higher altitudes than it would otherwise go, making clouds form more often and last longer than they would otherwise.
  
  The presence of clouds in turn transforms the climate by changing how radiation flows into and out of the atmosphere and providing a greatly enhanced form of upward heat convection. The main source of IR-active gas in the clear air is not CO2 or CH4, but the feedback effect of enhanced, clear-air water vapor. But even limited condensation of the enhanced water vapor into clouds changes the radiation flow drastically.

Observation: Go forth and squint hard. In the end, a chaotic system is its own best computer: no model we devise or limited set of observations we make can ever capture every aspect of its behavior. But observation nonetheless remains essential for understanding climate. Modern scientific observation of the atmosphere, increasingly detailed since the 18th century, has a lot to tell us about the repertory of possibilities.

In understanding actual climate, we must always keep in mind the proviso that chaotic systems feature an unending stream of unique events. We also have to face repetitive trends that repeat on time scales longer than the modern scientific record captures. Climate is, in this sense, a unique problem, in that we're inside the system being studied, and we're myopic observers with only hints and partial clues about the long term. Although laboratory experiments are essential for isolating general physical laws, the actual conditions of climate do not constitute a laboratory experiment. It's not controlled, and we're not outside the system in a position to aspire to know and control everything about it.

Observation: The Sun will have its say. It always does. It's the ultimate factor in charge of Earth's climate. Like other stars, the Sun is variable, at a small but measurable level. What limited observations have been made of our Sun already strongly hint at important solar modulations of Earth climate. The more basic solar physics in control here, and how the Earth responds to solar changes, are still poorly understood, and the whole problem remains at the frontier of research. But the base of raw data needed is now available in a way not true 20 or 30 years ago. Studying other planets' response to the Sun's variability will help.

Observation: All things green and blue. Plants and oceans need to be understood better as well. Over scales of decades and longer, they play a critical role in absorbing and recycling carbon dioxide. Current climate models capture the ocean part only imperfectly and plants barely at all. Yet there's a 0.2/0.00038 = 530 ratio of diatomic oxygen (O2) to CO2 in the air, which large ratio is made entirely possible by plant metabolism.** The annual plant-driven variations in atmospheric CO2 concentrations are about eight percent of the total. Since 100/(8/year) ~ 12 years, every CO2 molecule in the atmosphere gets captured by a plant in a little more than a decade.

What's not understood is how plants are responding in their annual cycle of growth and decay to increasing CO2 in the atmosphere. With more "food," there will be more and bigger plants. How much is unknown, although the Ice Age results give a very rough idea. What little research here has been done so far has been strongly tainted by people out to "prove" that plants aren't important - even though they clearly are. It's an obvious place for Gaia-philes to speak up. One of the few geoengineering ideas with any merit involves humans enhancing an already old and thoroughly proven means for removing CO2 from the atmosphere: more plants, bigger plants, maybe even über-plants. More on that next.
---
* Indeed, the wild claims made by the IPCC for GCMs and "climate parametrizations" can be put into sharp relief when we consider that, were these results actually definitive answers for climate, we would also have a solution to fluid turbulence.

In fact, we don't. Over at the Clay Institute web site, you'll see there's a Clay Millennium prize for solving turbulence (Navier-Stokes equations) - and it remains unclaimed. Given the true state of affairs (turbulence remains an unsolved problem in physics and engineering), we can then rightly reason backward and conclude that the climate problem remains unsolved as well, since the turbulence problem is embedded within it.

** Without constant plant replenishment, the O2 would rapidly disappear from the atmosphere by oxidation weathering and water absorption. Animal metabolism would be impossible without plants, although plants can and, long ago, did do fine without us and our animal relatives.

A portfolio for Extremistan

So friends ask me, what do you do for your portfoilio? Are there rules of the road in Extemistan?

It's more of an art than a science, but I have some intuitions that I follow and that have served me well. One is good return, and another is don't get cocky. Don't worry about betas, volatilities, and correlations. That stuff is highly questionable at best and outright BS at worst. Everything known for sure about price behavior strongly hints that their statistical distributions are "fat-tailed," subject to "large fluctuations." Moments like the variance are probably not even defined, being either infinite or at least very and fuzzily large - so much so, they might as well be infinite.

Average return over some fixed time period is a stand-in for cumulative return, which is the right thing to look at to see how an asset has performed.* The problem is that, cumulative return isn't enough. You're not going to buy an asset ten years ago, or whenever the cumulative return period started. You're going to buy it now. So is it worth it now? All you can do is figure out if the asset is now overvalued. "Value" investing amounts to no more than paying a good price or less for something. To check valuation, people use all sorts of numbers, such as the price-to-earnings ratio. My favorite is the price-to-book ratio. Ratios below two are very favorable. From two to four is okay. Above four or five, you're getting into overpriced territory.**

The final principle is diversification may be hairy, but it's worth it. But you have think more diverse than many brokers have traditionally, beyond the old trinity of stocks, bonds, and money market. Think of real estate (yes, it's still a good investment, if you pick the right type), and commodities and raw materials (yes, they're volatile, but diversify within the class).

And that's it. I'm with Schwab and settled on three of their funds with the best relative rankings by these criteria: a small- to mid-cap value fund, real estate (with holdings more in commercial than residential real estate, and a lot of international coverage), and an international fund (valuation a little questionable, but needed for diversification's sake).

POSTSCRIPT: Back to another recent excursion in Extremistan for a minute, the water crisis. A friend pointed out that just the assumption of stationarity (underlying probability distribution being unchanging in time) is an assumption, just like the Gaussianity (bell-curve-ness) assumption. And that's true. If there were definitive evidence of non-stationarity, by all means let's drop it.

But stationarity is a simpler and more primitive assumption than Gaussianity. We have very good reasons, based on everything known about "open" systems with "flow-through" (rather than "closed" systems with fixed totals), to drop the Gaussianity assumption. It's a more specialized assumption than stationarity and thus more likely to be wrong: so says Occam.

Even if stationarity happens to be wrong in the end, there's no reason to assume that the time variation of the statistical distribution is due to human activity. There are all sorts of more likely possibilities. There's a pernicious assumption that "open" systems should be stationary, and, if they're not, it's humans' fault, dammit. In the case of climate, we already know of decadal to millennial timescale changes due to solar variability; on longer timescales, due to the Ice Ages, continental movements and changes in the Earth's orbit and orientation in space. No need to invoke human involvement unless there's some other "smoking gun" that can't be explained better some other way.
---
* Cumulative return = (present value/initial value) = (1 + inferred average annual return)^years.

** There are more sophisticated approaches, a particular favorite of mine being the flow ratio. But such detailed analysis of an asset is worth it only if you're investing in a single stock or other asset at a time.

Fat tails and outliers: The water crisis

Recently, the occasional severe droughts that afflict the drier parts of the US have given rise to a predictable whine of "correct" journalism that wrongly attributes the "water crisis" to human-caused changes in water supplies, instead of correctly to a combination of rising demand and misguided scientific and policy assumptions. Like the fallacies of extreme weather and quantitative finance, the water panic is ultimately rooted in an essentially wrong picture of statistical fluctuations in open systems. Amplified by "correct" herdthink, it has turned into a perfect storm of bad science and bad journalism.

Statistics and stationarity. A statistical distribution is a substitute for full knowledge of a collection (possibly infinite in size) of particular instances. It's a way of stating limited knowledge. Stationarity means that, while individual instances can differ and limited samples might show time dependence as the distribution is sampled, the statistical distribution itself doesn't change. It's not always right, but for most cases, it is a good starting assumption in the absence of compelling evidence to the contrary.

The problem with stationarity is that it is often combined with another, much more questionable assumption, Gaussianity of the distribution: the distribution is assumed to be a bell curve, arising by the classical Central Limit Theorem from more primitive proto-distributions with finite moments. For open systems not closed with respect to exchange of flows with the outside world, it's well-established that Gaussianity is usually wrong. Better to assume the more general case, a Lévy distribution with some of its moments infinite. Recall such distributions have "fat tails" and support large deviations from the mean ("black swans").

Wrong conclusions prompted by wrong assumptions. For example, a stationary but non-Gaussian distribution, if it is sampled periodically, will produce moments that look as if they're increasing in time, perhaps implying a time-dependent distribution. In fact, this is a well-known fallacy in statistical reasoning. The moments are simply diverging. The distribution is stationary, just not bell-curve. There are other, better ways of analyzing samples from statistical distributions in this case.*

It's been known for over a century that rainfall and other water-related flows in our environment do not follow Gaussian random-walk behavior. Their fluctuations exhibit violations of this assumption, of broadly two types: memory or time correlations (so that each step in the walk is not independent of previous ones); and divergent moments (infinite variance or standard deviation, say). Either one of these should be enough to signal that Gaussian assumptions should be loosened. Unfortunately, a combination of theoretical prejudice and convenience has led policy-makers to stick to wrong assumptions anyway, just as financial traders and regulators (until recently) have been under the spell of the classical Gaussian random walk to explain price changes.

And now to bad journalism. There has been a recent water crisis in the American West. Actually, it happens roughly once a decade - the current one is in the midst of disappearing after an exceptionally snowy winter. Such crises are increasingly attributed to humans changing the available water supply and destroying "stationarity." In fact, it's the accompanying Gaussian or bell-curve assumption that's wrong.** The "throughput" of annual water flow available for human use changes year to year. But the underlying statistical distribution doesn't have to change.

And then to bad policies. Policies built on wrong Gaussian assumptions will lead to the same characteristic mistake over and over: the conclusion that fluctuations should be frequent but small deviations from a well-defined mean. In reality, the distribution has much larger fluctuations, which hit regulators, policy-makers, and ordinary folks again and again as "surprises" or "crises." But there's no crisis, just wrong assumptions. Gaussian-stationarity was never a reality, only just an assumption, and a bogus one at that.

In the case of water flow, increasing demand raises the chance that, in any given year, the the water system will be flowing below the threshold needed to meet that demand. There are three solutions.

Lower demand. Much water use in the American West is heavily subsidized by state and federal governments. Much of it goes into marginal and inefficient agriculture (for example, growing alfalfa and rice - both monsoon crops - in the California Central Valley). Reduce those subsidies, and you'll reduce demand.

Boost reserves. Just as banks and other financial institutions should be required to hold on to larger reserves to meet "black swan" crises, so a hedge against a large downward fluctuation in water flow is to build more and larger reservoirs, and to manage them more conservatively.

Change expectations. If scientists, policy makers, politicians, and voters carry around in their heads a false concept that water flow is basically steady and predictable, they'll treat the inevitably different reality as a "crisis." If everyone involved understands that water flow is subject to large changes year to year and not predictable beyond outlining a rough range, the policies will be different and more oriented around hedging better against drought by storing water during good years.

The Earth's water flow is deterministic, like all aspects of climate. But it's also chaotic. Fat-tailed statistical distributions are characteristic residues of chaos, as is the typical pattern ("intermittency") of clusters of good years and bad. Caveat emptor.
---
* Use the absolute linear range, for instance.

** More technically, it's the assumption that the distribution can be estimated by estimating moments from past water flow data. If some or all of the distribution moments are infinite, this technique doesn't work and never worked. Scientists, engineers, and policy-makers who thought otherwise were simply fooling themselves.

Fat tails and outliers: Extreme weather

It's often claimed, on little physical basis, that "global warming" with rising concentrations of infrared-active gases will lead to more extreme weather. We need to define more clearly what "extreme weather" is and apply some of principles learned from other open, "driven" systems, with non-conserved totals of energy, matter, and so on. The weather lives in Extremistan, at least to an extent.

Concentrate on just the atmosphere. The amount of dry air is essentially fixed, but not so the water in the air (both vapor and condensed), which is highly variable. The amount of heat (which is also rather variable) is less important than the heat flow. While the heat flow is split among radiation, evaporation, and turbulent convection, the total flow is relatively fixed. The division among the three mechanisms of flow is less stable.

Start with the total available. Instead of taking each "extreme" weather event one at a time, let's reverse the logic, with the same reasoning used by geophysicists when they consider earthquakes.* Imagine you're a storm god with a more or less fixed annual budget of heat and water. Taking the totals as the staring point, how do you divide them up every year? All at once? Blow your budget on a few big events? Or on many, many little events?** The "storm-generating system" is an open, not a closed, system; one with stuff flowing through, not isolated and in thermodynamic equilibrium. Extremistan statistics (fat tails) should be the default way to analyze it. It's natural then to assume that "storm budget" for a year will be dissipated by the cumulative effect of a range of events, some small, some medium, some extreme, all together making a distribution of event frequency versus event size.

Unfortunately, a wholly bogus case for "increasing extreme weather" has been built on using misguided Mediocristan (Gaussian) statistics to analyze extreme events. The fundamental problem is the same as in the other misapplications of the classical Central Limit Theorem to such events: the distribution moments are supposed to finite, but in fact, are not. So the sampling of the distribution by observations is misinterpreted to infer that the total of such events is increasing in time (non-stationary), when in fact the moments (like the variance) are simply diverging.

A subtle point: The number versus the energy release of extreme events. A "global warming" world is one where many differences in climate fade, leading to a fall in certain kinds of "weather events," like fronts and tropical storms. In such a world, it's reasonable to conjecture that both the number of extreme events and the cumulative energy dissipated by them go down.

But that reasoning alone is not precise enough to tell us how much each will go down. Periods of warming (like 1910-40 and late 70s to mid-90s) had fewer tropical storms. But a few of them were nonetheless memorable for their size; for example, the unnamed 1938 hurricane that hit Long Island and New England, or Hurricane Gilbert, which hit the Caribbean, Central America, and Mexico in 1988.

It could be that both the number and cumulative energy of extreme weather events goes down under "global warming," but that the energy per storm or event can go up. The number of events can go down by more than the amount of energy dissipated goes down. Thus the ratio (energy/event) can go up. A possibility to think about.
---
* This is the Gutenberg-Richter law, one of a large set of modified power-law distributions with "fat tails" (slowly falling probability for larger and larger events, instead of a sharp fall-off). The upper cut-off is not literally infinite, but very large. In practice, it's set by the total amount of energy available - in the earthquake case, the total amount stored as a potential energy in the Earth's crust under tension. Conceivably, all that stored energy could be released in one giant earthquake.

** Don't laugh. If the Greeks had known about Extremistan, they would have deified it. Actually, they did: Tyche, or Fortuna.

Pop! Is the debt party over?

Some are predicting the end of the US's 30-year party with debt. The debt fling was in part a result of foreigners' willingness to lend us extraordinary amounts of money. This willingness has kept the dollar artificially high relative to other currencies, especially the Asian ones. The US has an almost unique combination of strong growth, political stability, and lack of corruption. So it's the perfect place for foreigners to park their money.

But in the last year, the Fed has lowered short-term interest rates so low, well below European rates (which are kept high by the European Central Bank's fear of inflation), that US credit markets are no longer so attractive to foreigner investors. Easy credit is getting harder. Of course, the crazy real estate bubble of 2002-2007 also featured reckless lending to subpar borrowers, and banks have now become skittish about new lending. Foreigners might want Euros instead, not because the Euro zone has great growth prospects (it doesn't), but for simple safety. If that became a persistent and widespread trend, it would spell the end of the US dollar's 60-plus-year long role as the world's reserve currency.

It's not clear if that will happen in the near future. More likely is a split world, where dollars, euros, yen, and yuan more or less share the space as reserve currencies. A nearly-as-dramatic change almost happened in the late 70s, during the Great Inflation and rapid decline of the dollar's value. The taming of inflation, begun in 1979 by then-Fed chairman Paul Volcker and backed by both presidents Carter and Reagan, turned that situation around. The price was that the US federal government went from printing money to borrowing it. But because the dollar's value held relative to other currencies (also rapidly depreciating with inflation in the same period), the dollar's role as reserve currency also held.*

But the larger debt party - which began in the 1970s with the first great postwar real estate bubble and the first wave of Silent Generation/Boomer home buyers, then spread to investment markets and governments in the 80s and 90s - is probably coming to an end.** Each wave of bubbles - smaller but still significant in the 70s and 80s, larger in the 90s stock bubble, and larger still with the housing bubble of the 00s - has imposed larger and larger costs of post-bubble clean-up. To keep the post-bubble bursts from being too painful, the Fed has three times in the last 15 years (1995-1999, 2002-2005, and now) opened the cheap credit spigot, keeping interest rates low and putting a large strain on the dollar. Fundamental (microeconomic) reasons for the bubbles are left unaddressed: herd psychology, grandiose expectations, bad lending practices, and the federal government's hard-sell (via Fannie, Freddie, and so on) of house-owning to people who can't afford it. Thus the feds end up becoming, more than ever, guarantors and rescuers of bad financial decisions, encouraging more of the same in the future.

Sebastian Mallaby of the Washington Post has a recent column on just this topic. Along the way, he brings up the disappointment of investors in markets that don't behave as the standard financial theory says they should. The "Gaussian random walk, efficient market" theory is dazzling, but it's also dead.

Debt also seemed not to be too risky because financiers had excessive faith in their statistical models. These suggested that their investments would never lose more than a small percentage of their value: Hence a thin capital cushion would suffice. But since its invention in the 1960s, financial economics has overestimated the efficiency of markets and underestimated their tendency to swing viciously. Along with the authorities' successive rescues and savers' confidence in American assets, this error kept the debt party going.

Here is another major reason investors and lenders are souring on risk: they're starting to realize that they don't understand it as well as they thought. It's time to go back to the drawing board, and also to impose tougher capital reserve requirements on banks and other lending institutions. The financial black swans are always out there, waiting.
---
* The relative value is all that matters in the short term. In the long term, the mix of stability, non-corruption, and decent growth is more important.

** You sometimes hear it falsely claimed that the debt party started in the 80s. It didn't; it started in the 70s. The 80s did see much more American borrowing from foreigners. The fact that the debt party started in the 70s is linked to the Great Inflation: inflation makes debt attractive to borrowers, because they pay back their debt in dollars worth less and less each year.

The serious inflation that ran from the late 60s to the early 80s was the single most important reason Americans, Boomers especially, became addicted to debt in the first place. It also pushed investors into far riskier investments (savings and loan institutions with overpriced real estate; major bank loans to Latin America, for example) in order to get higher returns that would beat the inflation.

Chaos and markets II

For those who teach finance, a number seems better than no number — even if it’s wrong.

- Mandelbrot and Taleb

It's the Tyranny of the Cookbook, to which we can reply: No number is better than some number - especially when it's wrong.

Much financial advice is common-sensical, but in recent decades has incorporated misguided notions of implicitly Gaussian or bell-curve statistics in analysis of price movements, as well as false concepts of "efficient markets." You often hear the jargon of means, variances, and betas. (A beta is just price volatility defined as a second moment, or a variance. The square root of the variance is the standard deviation.) We've already seen a truckload of examples of where and why such concepts break down and why methods based on such assumptions are wrong. The fact that this approach to finance has a Nobel prize is irrelevant.* The methods and concepts have spread from academic finance and economics departments to the desktops and minds of investment specialists in the last 30 years and done significant damage: the Long Term Capital Management crisis in 1998 and the mortgage crisis of 2007-08 were both made possible, in part, by such "professional consensus" malpractice. Here we have legendary cases of Platonified false expertise and the "empty suit" syndrome. The price change distributions are fractal-driven power laws, not bell curves, a fact first presented to the economics world almost 50 years ago by Mandelbrot - and then rejected because it didn't fit convenient, if unempirical, Mediocristan assumptions. The missing practical key is the widely unrecognized enhanced risk of large fluctuations, especially downward moves. Individuals and institutions adopting wrong rules expose themselves unwittingly to much larger risks than they realize.

If we drop the assumptions of bell-curve price fluctuations and efficient markets, where do we stand?

The first is basic math and science: get your units straight. People who practice finance usually get this right, but it's amazing to see ignorance even in the business pages about this. Economics, like mechanics, has three basic types of units: money (a universal store of value and medium of exchange), things or activities (count them distinctly and don't commit the Fallacy of Aggregation, lumping bananas and pork bellies, say), and time. The essential point is that wealth is an accumulation of flows. The flows are prices (measured in money) times things or activities (quantified somehow) divided by increments of time. Interest rates are prices divided by prices divided by time, or just 1/time. Wages are money per unit of labor (an activity) per time. And so on.

The principle of diversification remains, but its rationale changes. It's not "everything will even out" (it doesn't always), but "we don't know very well how individual investments and investment classes will perform - sample all of them." Diversification, not only within investment classes, but especially across classes, is even more important in Extremistan than in Mediocristan.

More basic to the uncorrelated, Gaussian price movement picture is the efficient market hypothesis, which has failed in a number of crucial respects. Market timing matters, especially if you're making large moves (investing or liquidating). The market analysis based on this wisdom is called "technical analysis" or "charting," and its advocates are called "chartists." They stare at price chart patterns. In the "uncorrelated random walk" picture, these patterns mean nothing. But in fact they do mean something. Market moves are indeed correlated across time. Only after three to five years do they start to lose their memory, and it's not clear that they ever entirely do.

Furthermore, there are investment classes that consistently under- and overperform the whole market average. The best-known underperformer is the class of "growth stocks," because they're hyped by the media and analysts to the point where buyers demand them strongly - they're consistently overpriced relative to their long-term performance. OTOH, there are underpriced investments: so-called "value" stocks, for example. Warren Buffet and others have made a fortune hunting for undervalued but worthy investments. It's all boils down to not paying more for an investment than it's worth.

Finally, the "fat tail" phenomenon should make everyone suspicious of probability distribution moments (means and variances). If misanalyzed using Gaussian assumptions, fat-tailed distributions appear to be non-stationary: if you keep sampling such distributions to estimate moments, your results will not, in general, converge as you add more data points. The estimated moments will just keep growing. After an infinite amount of sampling, they diverge to infinity. While means and variances are measures of performance, they're not good measures.

The devil's staircase. A better approach than looking at daily movements is to look at cumulants (integrals) and at absolute linear ranges (price highs - price lows). The cumulant is more stable than the daily changes in value, and sudden jumps in the total value of an asset or flow of goods and services show up clearly. (The fact that such sudden jumps often dominate the total or cumulative history of an asset or market also stands out clearly.) The absolute linear range grows with time, but gives you some sense of the best and worst the market can do. These are the rules of the road in Extremistan. "Mild" variables change by a large number of small increments. "Wild" variables change by a small number of large increments, and "really wild" variables change mainly by a handful of very large increments.

Markets with an incomplete cookbook. The investment community at large still has not fully absorbed Mandelbrot's message about fractals and the uselessness of Gaussian, bell-curve statistics in understanding and prospering in markets. The normal and the Levy-type distributions look similar when you compare them for small deviations from the mean.** It's the large deviations that constitute the acid test, and it is here where investment professionals often start waving their hands.† In a Gaussian world, such large changes shouldn't occur almost ever, and the history of Gaussian markets would be dominated by many, many small changes. But real markets are strongly shaped by a limited set of rare, large, and consequential events. A new investment science to replace the rigorous, Platonified irrelevancies of contemporary financial theory is badly needed.

POSTSCRIPT: Here's a short note on market risk by Mandelbrot and Taleb from a few years ago.

References

= B. Malkiel, A Random Walk Down Wall Street, rev. ed. Classic presentation of efficient-market, Gaussian random walk theory to the masses. Much of the technical side is wrong as a picture of markets, but the basic investment advice (the trade-off between active and passive investment, diversification) is sound.

This posting is a sketch of what's needed to replace the bell-curve price movement framework. Just noted today: the embarrassing underperformance of stock index funds since the 2000 market peak, compared with even lowly bonds, not to speak of value stocks.

= R. Haugen, The Inefficient Stock Market. Nice short, if technical, study of systematic inefficiencies (over- and underpricings) in markets.
---
* Black and Scholes won it in 1997, and Taleb and others have railed against this as a perfect example of rewarding Platonified bullshit with its origins in academic circles, with highly restrictive assumptions, applied to real life where those assumptions don't hold. The LTCM crises occurred less than a year after the award - again suggesting a just G-d, or perhaps one with a refined sense of humor.

A larger objection can be made against the economics Nobel prize altogether, and Taleb and others argue that as well. It's actually a Nobel foundation prize paid for by the Royal Bank of Sweden, not specified in Nobel's will. Although some great and deserving economists have won it (Hayek and Friedman among them), in general, it's difficult to argue with the reality that economics has often been subject to both fads and conveniently cookbook pseudoknowledge. The standards for the Nobel prizes in the natural sciences are much stricter, and I hope they remain thus, so that at least those Nobel prizes mean something.

** Actually, the log-normal. The Gaussian bell-curve is applied, not to prices, but to the logarithms of prices. Small changes in prices are then translated into small percentage changes. (For price P, the differential dP is replaced by dP/P.) For small ΔP's, the log-normal and Lévy-type distributions look almost identical - it is here that the theorists of the Gaussian random walk go astray.

The "random walk" idea can be taken beyond the Gaussian or normal type and recast into a more general form of Lévy flights, dropping the requirement of finite distribution moments. To handle correlations over time between events, it can also be generalized in another way, to have memory: fractal random walks. Such erratic "random" or "drunkard's walks" are an important tool for applying statistical methods to dynamics under conditions of limited knowledge. The random walk is also central to analyzing diffusion (both standard Gaussian and "anomalous" fractal types). In chemistry and biology, the random walk is sometimes called Brownian motion.

† In the last generation, improvisations have grown up around the failure of Gaussian methods, but this series of ad hoc patches and fixes doesn't get to the root of the problem. Some analysts still just take out large deviations ("outliers") by hand, a kind of data denial. Others appeal to the notion of "exogenous" (outside-the-system) shocks, which destroys the method's predictive (if not its retrospective) powers.

The most sophisticated patch is to make the Gaussian parameters depend on time, the common version being GARCH. This is the best you can do within the misguided Gaussian framework; in that wrong framework, the actual (and probably stationary) distribution of price movements looks non-stationary. The time-dependent parameters are supposed to mimic this, but at the cost of largely destroying the method's predictive power.

Hysteria, regression, and amnesia

A politically- and journalistically-induced craze like "global warming" has its costs. The most obvious is a poisoned politics overburdened with pseudocrises. The potential economic cost is also clear: implementing the Kyoto Accord reductions of CO2 emissions would mean dismantling a significant chunk of modern civilization.

A spiritual cost. But there's another cost as well, more elusive but very real, imposed on our minds, both the general public and the scientists who've been arm-twisted and panicked by the alliance of politicians, Official Science, environmentalists, and the media - not scientists now or ever - that drives this fake crisis. For the general public, the price is the decline of scientific literacy - the "global warming" hysteria is both of result and a further cause of this decline. As we live in a civilization based at every turn on advanced technology and scientific discoveries, this is worrisome. Another, related worrisome trend is the decaying influence of education, which more and more is challenged by the "junk-food" alternative: information in place of knowledge, tidy and conveniently televised moralistic and narrative fallacies left and right replacing cause-and-effect thinking, journalism in place of books, and misleading fabrications and half-truths in place of first-hand knowledge and personal experience.

In the scientific world, the price of "consensus" science officially imposed is, self-censorship among scientists who don't want the hassle and personal vilification that result from defying the false official thinking. Add to that the shifting of research funding and acceptance by scientific journals away from scientifically sound work toward "para-science" - politically-motivated hack work that doesn't ask questions and look for answers, but takes predetermined answers and policies - "something must be done!" - for granted, then fills in the "back story" to rationalize whatever policies have already been chosen anyway. This sort of thing is something, but it's not science or research.

The price extends into technical details as well, as the previous posting demonstrated. It's the use of bad techniques - methods inappropriate to the task, methods disproved, inadequate methods superseded by better ones - in a rush to a predetermined conclusion in support of "narratives" and policies already chosen by politicians and the media. From the point of view of a scientist, this is especially depressing; it puts the cart before the horse. It would be like doctors abandoning modern antisepsis and surgical techniques and taking up bloodletting again, if some fringe environmentalist group decided that rubber gloves and surgical scalpels were evil talismans of modern civilization and the media then decided, in unison, to scare everyone into accepting it.

In the case of the study of Earth's climate, the large looming discoveries of the last 50 years - nonlinearity, chaos, the non-Gaussian (non-normal or non-bell curve) statistics of events, the limits of prediction, the difficult of defining what we even mean by words like "climate," the fragmentary but still impressive advances made in filling in the details of paleoclimate and the Ice Ages - have been largely ignored, because (a) they're often hard to fully grasp, at least at first; and (b) they uniformly conflict with the "global warming" agenda, either by directly contradicting it or by clarifying how hard it is to predict the weather in the future. The "global warming" agenda can only be accepted by "unknowing" what modern science has learned about climate and dynamical systems and regressing to more naive and poorly informed positions.

We forget and forget and ... huh? So finally, we end with amnesia - a weird disconnect between what scientists and educated laypeople already know (the long-term weather can't be predicted, say) and the unfounded but aggressive confidence that everyone has been bullied into on this subject. But it also shows up in the purely political aspects of the situation. The Kyoto Accord on reducing CO2 emissions was agreed to in 1997 and initialed by President Clinton. The U.S. Senate then rejected the accord in a virtually unanimous vote. Their motives were largely about the economic cost, a perfectly legitimate concern, and one that Clinton himself was fully aware of - the White House expected the Senate to reject it. It was never formally submitted for ratification, in fact.

Ah, but not to listen to our allegedly omniscient, but in reality, amnesia-inducing news media. In their frequent but largely false retelling of the story, the Kyoto Accord was agreed to by "everyone" (that is, by themselves), until the evil W. undermined it by withdrawing from it. Except the U.S. was never in it. All W. did was to revoke the presidential initialing of it. Furthermore, no country that signed the Accord has come anywhere close to meeting its reduction targets, and, unless it wants to dismantled modern civilization within its borders, no country ever will. The Kyoto Treaty is an example of the corruption of our politics by manufactured crisis, accompanied by the 24/7 drumbeat of media hysteria. It is to the great credit of the U.S. Senate (a body whose seriousness I frequently wonder about) that they rejected it - they took it seriously enough to see how fantastically absurd the Accord was and rejected it. They could have, after all, cynically accepted it anyway, knowing full well that the U.S. would never meet the targets, but indulging in a moment of feel-good self-righteousness. That is evidently what happened in many other countries.

Recently, Glenn Reynolds of Instapundit laid out another case of journalistic malfeasance committed in connection with Kyoto by the Associated Press: read his comparison of history with the AP's garbled pseudo-history here. As Reynolds rightly says, "You have to wonder ... why people bother to listen to the Associated Press when it can't get basic bits of recent history right."

Fat tails and outliers: A closer look

No, it's not about the Fat Tonys of the world, Taleb's proverbial cabdrivers who know at least as much about events as so-called experts, not because they're so smart, but because the so-called experts know far less than they think. But Fat Tony might appreciate the world of "fat-tailed" probability distributions, since they provide the mathematical way of capturing, in part, the phenomenon of the black swan: why large deviations from the mean ("outliers") are less common than small ones, but still much more common than expected on the basis of the normal or Gaussian bell-curve distribution.

The Gaussian distribution is used so much because of an important mathematical result, the Central Limit Theorem (CLT). It states that, if we consider a large number of instances of a random process, the collective "distribution of distributions" is Gaussian, if certain conditions hold. These conditions are that:

• The individual instances making up the distribution must be independent of one another.
• The moments, or weighted averages, of the original probability distribution must be finite.

What happens in the "large numbers" limit, if these conditions hold, is that, of all the moments of the original distribution, only three matter after the dust settles - the total population size, the mean, and the variance (the zeroth, first, and second moments - see below). All the other moments either vanish or are controlled by the first three. These three are exactly the ones needed to define a Gaussian bell curve.

A simple example. Let's consider a population of particular instances of some property or attribute, quantified by a random variable x, allowed to range from -∞ to +∞. Its probability density is f(x); within an infinitesimal range dx, the total number of instances between x and x+dx is f(x) dx. The cumulative number of all instances of x < X is the integral of f(x) from -∞ to X. Define the nth moment (or weighted area under the curve) as M(n) = ∫ xⁿ f(x) dx. The non-negative integer n = 0, 1, 2, ... ∞.

The Gaussian with zero mean and variance of one is f(x) = exp(-x²/2)/√(2π). (The normalization is chosen such that M(0) = 1.) It is strongly peaked at x = 0 (the mean) and falls off rapidly for deviations from the mean.

The "fat tail" case occurs when, whatever f(x) is doing for small x, it decreases for large x as |x|^-a, a > 0, apart from overall multiplicative constants. f(x) falls off for large x, but far more slowly than the Gaussian does. Then M(n) ~ ∫ |x|^n-a dx. Replace the upper (lower) limit of the integral with +X (-X), X → +∞. Then M(n) ~ X^n-a+1. There are three possibilities:

• n - a + 1 < 0. The moment M(n) is defined (convergent or finite).
• n - a + 1 = 0. The moment M(n) is infinite, diverging logarithmically.
• n - a + 1 > 0. The moment M(n) is infinite, diverging as a positive power.

For a "fat-tailed" distribution behaving this way, while some moments (for lower n) might be defined, the remaining moments n > a - 1 are undefined. Therefore the CLT does not hold, and it is not correct to use Gaussian-based statistical methods for such populations.*

Long before Fat Tony.... Such distributions are called, in the mathematical literature, Lévy flights, after the French mathematician Paul Lévy, who first worked with them in the decade prior to the Second World War. Mandelbrot, the geometer of fractals, was a student of Lévy. Both Lévy and Mandelbrot went into hiding after the French defeat in 1940, avoiding the Nazi and Vichy dragnet of French Jews.

After the war, they were also intellectual refugees from a certain style of mathematics that swept over the French academic world and had a strong influence elsewhere. Collectively named the Bourbaki school, it drove applied and "heuristic" mathematics to the margins of the field and favored a lean, abstract approach of theorem-proof, with no pictures, diagrams, or applications. (It was the same period that the artistic avant-garde moved strongly in the same direction: away from sense perception, toward "pure" abstraction.) The situation relaxed in the 1970s and 1980s, followed by a strong revival of interest in applied mathematics both among mathematicians and scientists and engineers who use mathematics. While rigor and precision are essential to mathematics, it can't survive or even make sense without contact with applied problems and the world of the senses, and the Bourbaki revolution petered out.

Using Lévy's results, Russian mathematicians Gnedenko and Kolmogorov proved a generalization of the Central Limit Theorem that allows for systematic statistical methods to be applied even in such Extremistan cases. But the resulting "distribution of distributions" is not Gaussian. If we want to study the statistics of events in a chaotic system, like the climate or financial markets, say, we must use these generalized methods pioneered by Lévy, not the 19th-century methods of binomials, Poisson, and Gauss. Like 20th-century artistic palettes and musical styles, it's a 20th-century statistics cookbook of expanded possibilities and greater generality. In the next posting, we'll meet a recent climate case where appropriate statistical methods were applied, with striking results, to a situation where wrong methods were long used.
---
* Usually, a > 1 in practice. If 0 < a < 1, then even the zeroth moment M(0), the total number in the population, is infinite. (The mean and variance are undefined as well.) Mathematicians can still cope with cases where some or all the moments diverge, by using something called the generating function of the probability distribution.

Meet the thinkers: The curious aviary of Dr. Taleb

We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns - the ones we don't know we don't know.

- Donald Rumsfeld

Some of us been waiting for something like this book for a long time, and its Levantine author has come a long way - all the way from the hills of northern Lebanon and the Syro-Greek Orthodox town of Amyoun. The book is The Black Swan: The Impact of the Highly Improbable, and the author, Nassim Nicholas Taleb, former financial trader and now extraordinary professor of the inexact sciences at the University of Massachusetts, Amherst, etc., etc. - essentially, the Dean's pet, and they don't know where to put him. The Black Swan is one of the most important science books for a non-science audience in many years. Like the best chaos and complexity books of a decade or two ago, The Black Swan deals with scientific questions arising from the stuff of everyday life, not far-off galaxies and times long ago.

The core of Taleb's point is the impact of what we don't know, the improbable, and how "randomness" is really another name for our ignorance. But Taleb has a larger target: a whole book was needed to attack and dismantle the legitimacy of bell curve statistics, "Mediocristan" methods wrongly applied to "Extremistan," and explain why so much of the world doesn't follow the "middle of the road" behavior prescribed by the Gaussian-normal distribution and its cousins, such as the binomial or Poisson distributions.

The book is rich with fallacies exploded:

• The Ludic Fallacy. This is the fallacy we pick up when we learn probability based on tightly constrained assumptions, "rule of the game," that make understanding statistical methods based on them as easy as an elementary cookbook. (Ludus is Latin for "game" or "fun.") Real life often presents us with situations of limited knowledge, where probabilistic thinking is appropriate, but where we don't know the "rules of the game," at least not all of them. Many trained in probability and statistics apply the cookbook methods anyway, for the lack of anything better. They capture risk - the known unknowns - but not true uncertainty - the unknown unknowns.
  
  
• The Narrative Fallacy. This is a biggie, practiced on an industrial scale by the news media, every day. We draw connections between dots where the real connections are different, or don't exist, or there are no dots to be found. The news media does it to keep our attention with frequently made-up stories, or "narratives," to use the post-modern jargon, that seem better than no story, or a different one.
  
  
• The Narrative Fallacy supports a related fallacy, one of Misattributed or Reified Intentionality, the fallacy that human society is collectively a result of human intentions or consciousness. In fact, most of it is not, and attempts to force it to be so have led to one disaster after another. Our minds are too limited and possess too narrow a scope of awareness to make this possible. Human society is mostly made behind our backs, so to speak. Taleb's developed views on this question end up very close to the views of the famous Austrian school of economics and sociology.

Taleb has an outrageously funny time explaining what went wrong with statistics and the social sciences in the 19th century, when they were invaded by the concept of the Average Man, and everything was reduced to bell curves, means, and small variations.* All this would hold if our world were Mediocristan. But much of our world is not.

Who's Stan, and what's the difference? Mediocristan is tightly constrained by "fixed totals," or what physicists call "conservation laws." We've already met these and seen what they do. They force the collective behavior into highly restricted patterns, with "equipartitions" of energy, or number, or volume. This certainly is an aspect of our world, and not just in thermodynamics. Heights and weight, for example, both of them strongly limited by gravity and metabolic limits, are distributed in a way close to the bell curve. But then again, consider the distribution of weights in aquatic animals, and you can already see: without gravity, the maximum size is much bigger (think of whales and octopi).

The key to Mediocristan is the Central Limit Theorem. If a population's distribution (of whatever attribute) is made up of independent instances and has well-defined moments (weightings), then the distribution approaches the bell curve in the limit of "large numbers." The presence of "fixed totals" guarantees well-defined distribution weights (moments).

But in many, perhaps the majority of, cases, it fails. The instances are not independent of one another, not distributed with well-defined weights, or neither. The distribution then has much less reason to clump near the mean. In fact, in such cases, many of our usual statistical clichés (means, variances, medians, etc.) fail to capture what's going on.

This is the world Taleb calls Extremistan.** If there's no "fixed total" of something being distributed (like economic wealth, or the total number of books sold by a single author, say), there's no reason to think that the total will be broken up in a roughly even way among instances. Here is the key to understanding much of our world - economic markets, wealth, and income in particular. Many days on markets are boring. Some are interesting. A few are extraordinary - and it these days, the black swans of the financial world, that end up dominating the cumulative history of the market. Just look at the last few months' newspapers.

We encounter similar truths in biological evolution, in contrast to the anodyne but wrong gradualism still dominantly taught. Most of the cumulative change in biological evolution is due to a small number of extraordinary turns of events that have outsized impacts echoing through the millennia. Ditto for human history.

And of course, on Taleb's home ground of financial markets, the reality of black swans, and fractal or fat-tailed distributions, is of intense interest. The disastrous application of bell curve-based statistical methods to quantitative finance in the last generation has not made markets better-behaved or investment strategies sounder. On the contrary: the 1998 Long Term Capital Management and 2008 mortgage crises make clear just how wrong these methods are. They're "state-of-the-art" in some sociological sense, but it's a mistake to call them an art, much less a science.

We've met these strange birds already: Taleb's black swans are the stream of unique events of chaos. His grey swans are those occasional, semi-tamable events at the low frequency end of the spectrum.

Plato in Nerdistan. As the book develops in its middle, Taleb wanders through the thickets of epistemology, how we know what we know. This part is somewhat weaker than the book's earlier and last parts, because the argument goes too far afield and loses a bit of focus. Taleb over-blurs the distinction between event (his specialty) and entity. Before European explorers reached Australia, they believed that all swans are white. The whiteness was not an essential part of the definition of "swan," nor was the belief obviously false. It was a contingent statement about two different properties of things: "swanness" and "whiteness." This supposed connection met its end when the explorers encountered the black swans of Australia. A deeper lesson took a longer to sink in, and some still resist it: disproving something is much easier than proving it. Proving something requires understanding its nature more deeply and thoroughly than our knowledge often runs.

Even this middle part is rich with deserving targets. Taleb calls them "Platonified abstractions," the stuff of academic knowledge. They're thrown around confidently by people who don't know what they don't know. This might almost be a definition of nerdity: what you know fits into cut-and-dried abstractions, and you confuse these with the actual world only known to us very imperfectly. Nerds stand in counterpoise to Taleb's foil, the Fat Tonys, the proverbial cabdrivers of the world who know better and who understand that when it comes to Platonicity, you can take it or leave it.

What do you know, and how do you know it? Exact human knowledge is coined under laboratory control or by precise logic. Most of the knowledge we use in everyday life is approximate knowledge in well-defined, if not controlled, conditions. At the edges of what we know is amorphous knowledge, often mixed in with a lot of prejudice and guessing. And if we want more and better knowledge, we face the reality of trade-offs. I can be sure something will happen today, but I don't know its significance. I can also be sure something significant will happen in the next year, but I don't know when.

Modern science is not based on induction, contrary to common belief. It's based on a mixture of hypothesis, deduction, controlled experiment, and controlled mathematics. It's not because scientists are dogmatists that they live by deduction. It's because deduction allows one's reasoning to be kept under precise control, with all the assumptions on the table and the steps clear. Induction (like statistical correlation) can certainly be strongly suggestive of hypotheses, and it's essential for developing logical definitions. But you can't prove anything with it. One counterexample - the black swan - destroys it. Silent evidence is always lurking to upset the induction cart.

The essence of probability. Coping with limited knowledge means falling back on probabilistic arguments, and this is in fact the origin of statistics. Its modern founders (Pascal, Bayes, Laplace, Gauss) all identified probability with a greater or lesser sense of certainty about something, not its frequency. This distinction fueled a great 19th-century debate between Bayesians and frequentists. Until the 1920s, the frequentists had the upper hand. But modern mathematics has abandoned frequentism, except as an approximation in carefully circumscribed situations where the Ludus isn't a Fallacy (like sports or gambling, for example). With frequentism came many long-unexamined false assumptions; for example, that "noise" and "randomness" are "theory-free" concepts. In fact, few things are more loaded down with theoretical assumptions than "randomness," if taken as a metaphysical category. Taking it as a statement about the limits of human knowledge, OTOH, makes it almost a truism. In most cases, the Ludic Fallacy will come back to bite us: we often don't know all of the rules of the game.

Unfortunately, the frequentist approach to statistics is still taught because it's cookbook. Even in situations where a canned approach is not appropriate, a recipe feels comforting, relieving people of having to think. I might even call this the Cookbook Fallacy: having a wrong recipe is better than no recipe. Actually, no recipe is better than a bad one - at least it's honest and doesn't force us into wrong assumptions.

Taleb in his garden. Along with his skeptical empiricism, Taleb exhibits other exquisitely refined scientific tastes, paralleling his capacious gourmand tastes in literature and food. This might seem an affectation, but it points to an important truth.

Richard Feyman, another man of powerful scientific intuition, said: to do good science, you gotta have taste! Science, like the arts, has its forms of kitsch: rules mechanically applied without the imagination and drive for the fully worked-out development, but without indulging in useless repetition. Science is still and will always remain partly an art. To have taste is to avoid weak arguments and rationalizing, and to avoid applying methods and concepts where and when they are not valid. It is to think that, if there's no deep fundamental principle that prevents something, then why not? What would the world look like if it were so? Maybe you lack the imagination to see that that is our world. Taste is seeing that not taking obvious things for granted is a true royal road to discovery. It is paying attention to the silent evidence, to the dog that didn't bark, and to the pious who prayed and drowned anyway: survivor bias.

Taste in science also requires revisiting fundamental issues, ones never completely resolved. The progress of science has solved many problems defined more narrowly. But deep issues remain, even if transformed. Science has its classics and its literature, history, and philosophy; progress doesn't erase their importance. Read them and avoid being a cultural philistine.†

Taleb reminds us that what we don't know can hurt us, and that what we don't know is often more important than what we do. The Black Swan is a fine book. Buy, read, and enjoy it, patiently and slowly. And if nothing else, be charmed by the bittersweet tale of Yevgenia and her unknown masterpiece.
---
* Hayek attacked much the same in his Counterrevolution of Science, laying out the 19th-century origins of Platonified pseudo-knowledge in the social sciences and the pretensions of social planning that often went with it. Plenty of perceptive people, like our old friend Poincaré, resisted this development, this misapplication of inappropriate mathematical methods to society. But the Tyranny of the Cookbook is an unrelenting one.

** Not to be confused with Wackistan. That's where Ahmadinejad lives.

† Taleb uses the German term, Bildungsphilister, just to show, I suppose, that he isn't one.

Be alert to a real affectation, indulging in philosophical problems isolated from anything real. As Taleb points out, most philosophical issues worth bothering with are suggested by something outside philosophy.