I love numbers. I’d much rather have data than opinions. Is exercising once a day better than twice a week? What page layout will get the most conversions? Does GTD actually help you get more done?
You could consult “experts” for answers to these questions. But I’m going to start off saying something that might sound hypocritical: there is no such thing as an expert opinion.
Why would someone who basically writes his opinions claim there is no expertise? Because it’s true and I’d rather be honest than successful. None of my opinions are expertise. But the same is true of every person who claims to be an expert and gives opinions without supporting evidence.
Finding Numbers and Making Yourself an Expert
My ideas are merely speculation. I’d like to think that they can guide you to running your own experiments. But if you confuse my speculation with hard facts, you’ve made your first mistake.
The solution is to find your own data and make yourself an expert. When you can know quantitatively which method works better, you can make sound decisions. And unfortunately, sound decision making is a difficult skill to obtain.
The branch of cognitive psychology that deals with biases studies our errors in reasoning. Here’s just a brief list of them [1]. With dozens of different ways our brain warps reality and misleads us, we can be blissfully ignorant about our own weaknesses. By using data and your own expertise, you can avoid many of the pitfalls of opinion.
When Not to Look for Data
Experiments work whenever there is a clear, quantitative notion of “better.” If your experiment results in answers based entirely on personal tastes, the data isn’t important. Spiritual beliefs or relationship preferences are good examples of where trying to do a scientific experiment is pointless.
This doesn’t mean you shouldn’t try out different options. Just be careful trying to analyze areas that have no objective component.
How to Run a Life Experiment
Here are some tips for getting you started running life experiments:
- Find Metrics – Find one or two measurements you can find easily to track your progress. For health this might be weight and number of consecutive push-ups you can do. For a website this might be page views or conversion rates.
- Metrics Should Be Easy to Measure – If your metrics are difficult to measure accurately, try picking something else. The hassle of measuring can stop otherwise useful experiments.
- One Test at a Time – Don’t test areas that might overlap. If you want to test out a new exercise strategy, you can’t simultaneously test a new diet. Unless you can separate the effects of each, your data is meaningless.
- Duration – The costs of running an experiment have to be weighed against the accuracy of the data. Producing perfect results is useless if it takes three years to get findings. I’m a fan of a one-month trial for most life experiments. But some people prefer year long experiments for bigger tests.
- Split Tests – Split tests occur when you test two different options at the same time. These are less prone to errors and can give you accurate results. They are easy to do with online sales pages or independent activities. But they are hard to pull off when you can’t go back and forth (i.e. measuring weight loss with different diets).
- All Results are Good – The only bad result is when the test was run poorly so data becomes useless. Your own expectations can distort experiments if you aren’t careful. Any result is an increase in your knowledge, even if it doesn’t confirm what you wanted to believe.
- Isolate Assumptions – Anytime you do something consistently without a good reason you have an assumption. Isolate these because they are perfect candidates for experiments. As a writer, I’m constantly varying how I write articles to see which formats generate more comments, links, traffic or follow-up.
- No Impact is a Great Result – You might become disappointed if you learn that you can’t detect a difference between two strategies. This means one of two things: either you didn’t give the experiment enough time or control to demonstrate the difference, or there really isn’t a big difference. If the last case is correct, you can stop wasting time debating about which of the two methods to pick.
- Data First, Theory Later – Whenever you start debating two options, the first step is to run an experiment. Only after an experiment does it make sense to start theorizing why a difference might occur. If you spend to much time theorizing beforehand, you might corrupt the results of an experiment.
- When in Doubt, Go With the Numbers – When your opinions contradict the data, my advice is to side with the data. As long as your experiments are run well, then numbers are far more useful than an uninformed opinion.