Chaos, Ignorance and Newton’s Great Puzzle

This is chaos:

Chaos theory is an investigation into mathematical and physical systems that are highly sensitive to initial conditions. The simulated three-pendulum systems above only differ in the tiniest way from each other, yet, after a few seconds, they are all wildly divergent in their movements.

I’ve grown to prefer the word “chaos” to the more commonly used term “luck”. People talk about luck a lot. We argue about whether someone’s success was an inevitable consequence of their skill and decisions, or whether they were just lucky. But we also talk about luck in superstitious terms, as if it was something someone possessed (“He’s just lucky.”) or that could ebb and flow with the tides of fortunes (“He’s just had a streak of bad luck.”).

Chaos, on the other hand, is a more mathematically precise concept. A system is more chaotic if small changes to its state can create wildly different outcomes. Compare the above system to the solar system. The fact that the sun rises each morning from the horizon may seem a banal fact, but it depends on the lack of chaos in our solar system. Adding a second sun creates a three-body system which, to this day, physicists do not have a way of precisely predicting the orbits.

Why Chaos Matters

Just as you can consider the amount of chaos in physical system, you can also think of the amount of chaos in different pursuits in life. The more chaotic the pursuit, the more likely small changes can result in completely different outcomes years later.

Although an absolute measurement of chaos is probably impossible, you can compare different pursuits by how chaotic they might be.

Consider two professions: becoming an actor and becoming a doctor. Both are prestigious professions. Both require a lot of hard work without much reward in the beginning, but have the potential for big payoffs.

However, acting is a much more chaotic profession than medicine. Landing the right audition can give you a name, which puts you up for bigger and bigger parts. But just as you rise, you can also fall. A bad movie or changing public tastes have also made many famous stars disappear in an equally short time.

Medicine is not chaotic. Although there will be marginal doctors who just get over the cutoff point for getting into medical school or landing a top residency, most are firmly within whatever category they eventually end up.

If you considered only a cruder metric, such as success rates, however, you might totally miss this picture. Most would-be actors and doctors fail to eventually reach success in their profession. But they may fail for different reasons. Would-be doctors fail because the profession is difficult and long, and so creates a high-dropout rate, not because it is inherently chaotic. Would-be actors may fail or succeed because small events create feedback loops causing wildly different fortunes.

The Problem with Mercury

For two hundred years since Newton, physicists had a problem: Mercury. The closest planet to the sun had a strange orbit that precessed. According to Newtonian physics, this wasn’t possible. The object would always have a fixed orbit around the sun, not one that wobbled around.

Physicists struggled against the problem of Mercury for years. Some suggested there must be a phantom planet, Vulcan, even closer to the sun, which was kicking Mercury’s orbit around.

Eventually, however, the true answer emerged: Newtonian physics is wrong. Albert Einstein introduced general relativity and its warped spacetime curvature. The new equations correctly predicted the precession and Mercury’s orbit was finally understood.

In Mercury’s case, physicists could safely rule out chaos because its precession was so orderly. However, in life, we often don’t get to observe the exact same conditions again and again to see the patterns. We only live once, so everything we experience is, in a certain sense, experienced for the first time.

As a result, the randomness we perceive in life always has a mixture of two possible components. The first is chaos. This is the amount that the system would be unpredictable, even if we had a near perfect understanding. The second is based on our own ignorance of the system, the amount it defies our expectations because we don’t really understand how it works.

How to Use Chaos in Your Thinking

In practice, it’s not possible to cleanly separate which systems are chaotic and which are merely poorly understood. However, it’s still useful to understand the existence of these two different sources of randomness.

When approaching any problem in life where outcomes are highly divergent, there’s always two possible mistakes that can be made.

The first is to overestimate the possibility of knowledge. By assuming that the system can be understood, but it is actually dominated by chaos, you may be gambling more than you realize. People who make this mistake may be convinced they’ve found the secret of success, but they’re really fooling themselves.

(Comic by XKCD)

The second error is to overestimate the role of chaos. By assuming that the system is inherently unpredictable, you may forego the possibility of mastery.

In my own life, I’ve seen this error with people new to blogging. Many people believe the field is inherently chaotic, depending on “going viral” and getting a lucky bit of publicity. The truth, however, is far more mundane. Most new bloggers I’ve met have obviously different levels of quality, consistency and work-ethic, which leads predictably to different outcomes, given enough time. Blogging success may be both rare and difficult, but I believe it’s less chaotic than many pundits would suggest.

Calibrating Between Assumptions of Chaos and Ignorance

One of my goals is to try to calibrate my expectations of chaos versus ignorance in different domains. Is my failure to predict because of inherent unpredictability or a lack of understanding?

This is difficult to do, but I’ve found there’s a few heuristics that can make the process easier:

  1. Theoretical justifications. I bias towards believing success in picking individual stocks is mostly chaos. I say this, not out of any first-hand experience, but because the efficient market hypothesis strikes me as being a reasonable theory. As such, I’m not interested in learning the skill of picking stocks, and have opted to invest in index funds instead.
  2. Existence of expertise. Look at the predictions of the best people. How often are they correct? If even the best don’t predict well, that doesn’t bode well for me. However, if successful predictions are common, it might demonstrate that there’s an understanding that exists which I don’t possess.
  3. Diminishing randomness. Another heuristic is to see how your experience of randomness is decreasing over time. If I’ve gone from 0/10 to 2/10 on a level of mastery from newbie to expert, and my ease at predicting success has gone up substantially, I’m more inclined to believe this trend will continue.

Randomness, whether from ignorance or chaos, is not ultimately avoidable. But by better calibrating your understanding between the two, you can see where investments in learning more are worthwhile and where it is probably a waste of time.

  • Michael Thiessen

    As a long time reader, I would like to point out that the addition of the ads seems to have caused quite a bit of jankiness on the page. Most noticeable when scrolling or typing.

    I personally don’t mind the placement of the ads, you have done it in a unobtrusive way. Perhaps there is an ad network/system that would work better (I am a web developer but have no experience with online ad networks).

  • Hi Scott,

    Great article! You’ve connect a very interesting theory to the investment world that make total SENSE.

    To add to your great article, technical analysis is chaotic. You can predict the direction of a trend with a high certainty, but it isn’t possible to formulate a Newton type of formula with 100% accuracy.

    Would be interesting to know how you invest in index funds?

  • Deepti Km

    Yes! Great article about my favorite topic! One more factor in your analysis of how the dynamics of different fields affect success rates: what is the ‘size’ of the solution set?

    If your target is very constrained (say, ‘win X place in Y competition in Z year’ or ‘get P position at Q company in R year with S salary’) then the odds of hitting it under chaotic dynamics are infinitesimal. If P position at Q company doesn’t open up in R year of if they go under, then you have no levers to pull in order to hit your target.

    If your target is less constrained (‘get any job in L field that is at least M fun and pays at least N’), then you still have control actions you can take when unpredictable factors blow you off target. For example, you can trade off points on the fun scale and the salary scale or switch companies. You can visualize this bigger target as a curve or surface in state space, whereas the constrained target was a point. So if you get buffeted around by random factors, you can still aim for the closest point on the target surface.

    Some fields like acting or professional sports have highly constrained targets (you have to get one of the finite existing positions that pay at all) with few levers you can pull to adapt around random factors. In other fields, you don’t have to be able to hit a specific point or points, but just control some characteristics of your outcome (and so there are infinitely many acceptable outcomes).

    It’s like in the three-pendulum: you can’t guarantee having the joint angles be (20 degrees, 21 degrees, 4 degrees) at time 5 s (if that’s what you wanted), but you can guarantee having the energy eventually be below a certain threshold (if there’s friction in the system). There’s tons of ways the latter could happen, and we may not care which one does.

  • mbw

    I like idea of using “chaos” as a drop in replacement for “luck” as it is used in this context.

    I don’t agree on the stock-picking, though. Efficient market hypothesis is kind of like spherical cows—it makes the math easier to model but it’s not very close to reality. You’d have to spin up countless alternate earths just to get a single stock picker as “lucky” as Peter Thiel and in order to have that same planet also contain a Warren Buffet…

  • horse woth no name

    hi,,,, knowing everything is a bad goal……. changing your weltanschhauung is a better one,,,,,,,, cordially,,,,,,,

  • Scott Young

    Is Peter Thiel such an investor though? My understanding is that he made his riches mostly through investing in companies before they were in public stock markets (Facebook, PayPal, his other companies, etc.)

    It’s also debatable whether Warren Buffet himself qualifies, although he is often trotted out as proof that the EMH is false:

    I’d rather think instead of the EMH being axiomatically true, but that, for the typical outsider investor it might as well be true. If there were systematic gains to be had through the application of a particular investing strategy, why would anyone expect those gains to be gobbled up by part-time stock pickers and not the quants and insiders on Wall Street who have the money, computing power, intelligence, research and (quite often) inside knowledge that normal people don’t?

    Calling the EMH a spherical-cow theory is a bit ironic, given the general accuracy of physics. I’d say that the EMH is more like a spherical-Earth theory. True, the Earth is slightly oblate, but it’s not flat…

  • Scott Young

    Right now my portfolio is pretty simple. 100% equity, mixed USD and CAD currency with some geographic diversification (although it’s weighted towards North America). I wouldn’t recommend that equity ratio for most people though, I keep it because I already keep a lot in cash for business working capital.

    I might pick some individual stocks or investments for fun. But I honestly already have most of my financial success and failure entwined with one company (mine) so I’m happy to have my fun there instead of on the markets.

  • Scott Young

    Yeah Disqus decided to run ads on my site without telling me (or paying me). We paid them $10/month and turned them off.

  • mbw

    EMH assumes perfect information, self-interested rational actors, no collusion or manipulation and roughly equivalent analytic capabilities. Reality doesn’t match any of those assumptions.

    Especially when it comes to asset bubbles and other forms of mass euphoria (or disphoria), it breaks down pretty badly. Volker specifically highlighted the role faith in efficient markets played in the 2008 crash.

    >”If there were systematic gains to be had through the application of a
    particular investing strategy, why would anyone expect those gains to be
    gobbled up by part-time stock pickers and not the quants and insiders
    on Wall Street who have the money, computing power, intelligence,
    research and (quite often) inside knowledge that normal people don’t?”

    This is an excellent question and one I asked myself before I started my last portfolio. I think the answer is clear. If you can’t compete against Wall Street fund managers directly, and you almost certainly can’t, then compete on a dimension they can’t. Here are two—longer time horizons and very small companies. Very small companies aren’t an option for most funds simply because even buying the entire company outright isn’t enough to move the needle on the performance of the fund. Long time horizons are also out because fund managers face intense scrutiny quarterly. Unlike VC money, their investors can sell at any time.

    So the tldr; is: invest in things too small to be worth Wall Street’s time instead of famous companies and invest on fundamental business value over a multi-year time horizon instead of any sort of short term trading.

  • Kevin Bourque

    “[M]ost [doctors] are firmly within whatever category they eventually end up.”

    I’ve been thinking about this for a few hours now and can’t decide if this is objectively true or if it seems that way because in general people get better (or get a result) after visiting a doctor.

    I’d love to hear your thoughts on this and/or expand on this in general with non-chaotic professions. For example, actors, small business startups, artists & musicians would be chaotic. But could we apply the statement above, i.e. “firmly within their category”, to lawyers, engineers or physical therapists? Food for thought.

  • Scott Young

    I think LT and small cap investing are possibly EMH weak points.

    But I think there’s two kinds of EMH here. One is that the market is always correct and rational. That could be called the strong EMH. The weak EMH is less confident. It simply suggests that there isn’t usable information lying around that allows for arbitrage opportunities. The latter expectation seems far more reasonable to me than the former.

    I think there are possible cracks in the system, but I think that as advice for a marginal, small investor, it’s probably a safe assumption to assume there aren’t.

    Another side-effect I’d add is that by doing research and buying individual stocks, your increase in expected return has to exceed that of a diversified position. That’s a lot harder to do when you don’t have a lot of money under management or aren’t investing in funds.

  • mbw

    An even more extreme example large scale madness in the public market was the 1989 Japanese asset bubble.

    You might be too young to have much impression of it, but at the time the Japanese stock market capitalization was 90% of all of Asia’s (despite the rapid development Korea, Taiwan, Singapore and Hong Kong). Even more shockingly, the Japanese stock market capitalization was more than the sum of all stock markets in the US and Europe combined.

    Simultaneously, there was a real estate bubble beyond anything the world has seen since. At its peak in 1989, the emperor’s palace in Tokyo was assessed at a higher value than the entire state of California!

    Interestingly, at the time Japan lagged considerably in living standards with only 46% of homes having flush toilets compared to over 97% in the US, UK or West Germany, under half its roads were paved vs 82% for the US and 99% for West Germany and there was only about 1/10 the park space per person in Tokyo as in New York, London or Bonn.

    Such is the madness of crowds. Memisis is powerful and on its own intelligence offers surprisingly little protection from it. Psychological distance from the turbulence of the news cycle and social trends is probably the greatest edge.

  • Scott Young

    Yeah, in hindsight a lot of bubbles seem obvious. But it’s really hard to time stock market collapses and many times what looks like a bubble may in fact be a correct evaluation, only to continue to grow later.

    The test for refuting some kind of weak EMH (such that it would actually make sense for small investors to engage in research and trading), is that one would have to predict those bubbles in advance and make a profit from taking a short position. Given that not nearly enough people did do this to deflate the bubble, that necessarily implies that it’s difficult to do.


    I’ll accept the idea that the market only imperfectly aggregates information, and that irrational bubbles and crashes can occur. The idea is simply that: are the existence of these phenomenon something you can reliably earn a higher rate of return from? The answer to that question seems to me to be probably not, or only in rare cases.

    Bubbles are probably a good example. Holding a stock short has unlimited downside risk, so while you might think a certain market sentiment is overrated, you might go broke before the bubble pops.

    So, on a theoretical level I believe the EMH may be flawed. But on a practical level, I think it’s case is much stronger for why it’s very difficult to reliably beat the market.

  • mbw

    To be clear I do NOT recommend short selling to a typical investor. The market can stay irrational longer than you can stay solvent. The correct move is to simply not buy any equity without a compelling case that’s significantly undervalued.

    I’d recommend value investing and only investing in companies with the plan of holding for 5 or more years. I’d read the quarterly reports (but not much else), keep an eye out for material changes in the underlying business. I’d mostly ignore the price of the stock after buying. In the case of literally every single value investor I’ve met, it was easier to get out-sized returns with small amounts of capital than with large amounts. It’s also worth pointing out that value investors with large amounts of capital sometimes struggle during bubbles because it’s difficult for them to find bargains.

  • Scott Young

    True, but I guess my point is that my feeling is that if the market is irrational it’s probably more irrational in an over-estimation than under-estimation. That is, because it’s a lot easier (and safer) to buy long than short, investors will gobble up perceived undervalued assets proportionally to the available information that they’re undervalued. However, assets may stay irrationally “stuck” higher because in order to benefit from that knowledge you either need to have a short position or sell a previous long position. The latter, requires that you be from the more limited class of investors who previously invested in that asset.

    The whole tl;dr of my point is simply that bubbles are not a good example of EMH failure because they’re probably harder to exploit.

  • Bo Yu

    Very great question. You are actually asking how we can measure the randomness in every profession. In science and engineering experiments, if you want to do the job, you do many experiements and calculations to get a final results. To put it a simple way, you actually need data to see how people start up with a profession end up with other professions, and you also need to know the distribution. Scott get the conclusion from his intuitive or life experience. But be careful, sometimes you could be fooled by your intuitives.