How Successful Was the MIT Challenge?

By far the most popular project I’ve done publicly has been the MIT Challenge. Starting nearly seven years ago, in the fall of 2011, the idea was to learn the curriculum of MIT’s four year computer science undergrad, evaluate myself by trying to pass the final exams, complete the programming projects and finish in one year.

In simple terms, the project was a success. I did what I set out to do, under the constraints I had imposed upon myself, within my original time limit.

However, behind those simple terms rested a lot of implicit assumptions. Now that I’ve had more time (and hopefully wisdom) between me and the project’s completion, I’d like to dig into some of those assumptions and see how well they’ve held up over time.

Whenever you set a goal in life, there’s a certain simplification of reality. Say you set a goal to lose 15lbs. Implicit, although maybe not articulated, behind that goal are a whole host of assumptions: that you’ll be healthier, look better, that the lost weight will be fat. Even behind those are deeper assumptions you may not even be aware of, such as that being fit is an important value in life.

Therefore, I think any project can be simultaneously evaluated at multiple levels. There is the explicit level of what you stated you were trying to achieve. But beneath that, are many other ideas of success and achievement, some of which you may only realize long after the project has finished.

I believe doing this kind of postmortem (however belatedly) is useful because, while highlighting one’s successes is fun and easy, it’s by looking at how our ideas differed from reality that we get true growth. I also believe that by digging deep and looking at some of my mistakes, I might be able to offer a more comprehensive picture for those who want to do better than I did in the future.

Side note: This is an incredibly self-indulgent post. If you’re not that interested in hearing me rant about myself and my past projects for 2000+ words, you should definitely skip it.

How to Evaluate?

I can think of a couple ways to evaluate the project, beyond simply whether or not I matched my explicit standards and constraints:

  1. Knowledge and skill. How did the knowledge and skill I acquired during my project compare with an MIT student. Did my choice of constraints/standards lead to missing things that were important?
  2. Rigor and difficulty. How well did my constraints match those of an MIT student? In particular, did my choice of constraints/standards make some things much easier than they should have been, or some of the accomplishments less comparable to the “official” setting?
  3. Outside impact. Did my ideas about what the MIT Challenge would mean for other students hold up?

On these three bases, I think the first holds up best, the second is debatable and the third was probably the biggest failure of the project.

How Successful Was The MIT Challenge for Learning Stuff?

The most obvious basis for evaluating the Challenge was to look at how it worked for actually learning the material, and perhaps secondly, how those skills extended beyond the scope of final exams and programming projects of the coursework.

In this sense, I believe the project was mostly a success. There were departures from the normal computer science curriculum. However, these were normally done because to follow the original program strictly would have been impossible. This included things like not having a thesis programming project, doing economics instead of humanities classes and substituting classes which depended on lab work for more theory-based classes.

These deviations likely made my experience different in some ways from an actual MIT student. However, I think for the most part, they didn’t substantially detract from the experience. I still learned roughly the same volume of material, it was just shifted in some ways to make the project practical.

More critically, however, I think are how ways I did the challenge might have impacted my learning of computer science outside of what could easily be measured via my system of doing final exams and programming projects.

I can see a few places where I think my approach to this issue may have created problems:

  1. Spacing. By compressing the challenge into one year, it’s likely that I did some damage to my ability to retain the material long term, versus, say, a more traditional four-year schedule. I did my best to mitigate this during the challenge by taking multiple classes simultaneously. However, I think, knowing what I now know about the typical half-life of knowledge learned in formal schooling, I would have been more serious about planning a long-term retention plan, the way I did with my language learning project which had a similar weakness.
  2. Lack of peer interaction. During the challenge, I was mostly dismissive of the importance of peers/teachers. I felt this way, rightly or wrongly, because I didn’t find those resources very helpful when I did my actual undergrad. However, I think, owing to the extremely theoretical nature of MIT’s CS program, the peer effects likely are an important part of the education process at a top school, since they allow students to find out about practical projects, tools and opportunities that aren’t taught in classes.
  3. Insufficient overlearning of core calculations. I believe my ability to do advanced mathematical calculation and symbol manipulation (algebra, calculus, etc.) was probably weaker than it should have been. Some of this showed up on the exams, with such classes being my worst performing. However, some of it was hidden from that perspective because I was aiming for moderate performance, not overlearning beyond sufficient performance. An actual MIT student, by means of doing assignments, past what is necessary, would likely have better abilities and better long-term retention of those types of abilities than I did.

These critiques don’t mean that I think I could have necessarily done much better, had I to do it all over again. Rather they’re useful to think about in case you wanted to attempt something similar. In my own situation, I think these trade-offs were mostly warranted, however someone learning for a different purpose might disagree.

Trent Fowler recently did his own year-long engineer/computer science program. While it differed from mine in many regards, an interesting way it did so was a much larger emphasis on practical projects. I’m not quite sure how to compare the two, but it does indicate that the space of possibilities for learning is vast.

How Successful Was the MIT Challenge for Matching the Rigor and Difficulty of the Real Thing?

Here my project and the actual degree start to differ substantially. I used a much more limited set of evaluation criteria. I focused on getting a pass (50%+) rather than getting as high a GPA as possible. I graded my own exams, using a solution key, rather than wait for an independent grader to go over them.

If the goal of the Challenge had been to replicate the difficulty of an MIT degree, or to perfectly match the standards of evaluation MIT used, then the project was doomed from the start. This was clearly not my intention, so evaluating it on this basis seems unfair (although, perhaps, probably worth pointing out to many people who casually observed my project and might have been misled to think that I did everything an MIT student would do, and got a degree to go with it!).

However, even within the intention of the project, my goal wasn’t to substantially reduce the difficulty where that was intended to measure my performance on the stuff I had learned. My original idea was that, should I clearly fail an exam according to MIT’s standards, I should have to retake it. I shouldn’t be giving myself an easier time, just because the material was hard.

From this basis, I think I did fairly well, although there are some points of contention. The most major one is how I did grading. While, in almost all cases, I only wrote exams with solution keys I could use to grade later, there often wasn’t clear instructions for how to handle grading. This, like many aspects of the challenge, required assumptions to be made about the proper way to do something.

Generally, my approach with exams was that any instructions provided (say explaining the amount of marks provided for a particular answer, or allowable materials, etc.) must be followed exactly. If information wasn’t provided, I did my best to devise what I thought was an acceptable approach (say docking a certain amount of marks for making a mistake in algebra, let’s say).

For most of the classes, this judgement call wouldn’t have made a difference. However, in some of the classes, particularly those which required long, drawn-out calculations, it could pose a challenge. If an exam had a question worth ten marks, how many points should be deducted because a minus sign wasn’t carried over?

Here opinions vary. In my own undergraduate experience, partial marks were common. Others, however, prefer stricter grading. In that view, any mistake, how matter slight, and all marks should be forfeit (including any follow-up questions that rely on the original result). This was not how I graded myself during the MIT Challenge, nor the basis I used for deciding whether I passed.

In the interest of curiosity, however, I decided recently to go back and see what the outcome would be if I applied the extremely strict grading. No part marks, no carry-forward of mistakes from multi-part questions from one part to the other. Under this new, ultra-strict, grading scheme, my exams passed went from 33/33, to 27/33.

I don’t think that the “correct” answer is the ultra-strict approach, and I still think the original way I evaluated the exams was fair (even if it may not have been exactly the same as another grader would have marked it). However, I think it’s worth noting the extent to which these assumptions might change the outcome of certain things.

An interesting, if perhaps unfortunate, consequence of the MIT Challenge was that many people focused on the presumed “impressiveness” of it, rather than (as my original hope) the idea of learning a lot of material, in a short time, with low cost and some benchmark to university standards. If that impressiveness is grounded in the idea that I did something equivalently difficult to a “real” MIT degree, irrespective of the knowledge acquired, I think the project doesn’t really work. This isn’t so much a critique of how I did the project, but perhaps what other people make of it.

How Successful Was the Idea of the MIT Challenge?

My original hope with the Challenge wasn’t to gain fame as the guy who did this strange project. Rather, I sincerely believed that it was only a matter of time until there were many people doing projects like this and I really wanted to be the first.

In retrospect, this is the biggest area where my thinking was at odds with reality. In the time since the MIT Challenge, I’ve met a handful of people who have done vaguely similar projects in terms of scope and ambition. However, this is a far cry from being a stable and recognized alternative to the formal education system.

Technology isn’t the problem. When I did the Challenge, that was before MOOCs, before the wave of online courseware and certificates from real institutions. Indeed, many of the deficiencies of my project were a result of the patchwork way I had to do it. Most classes had only a fraction of the full material available to an MIT student, so I had to make do. Today, doing something like the MIT Challenge is even easier.

I’ll admit, I was an education optimist. I believed that the purpose of school was learning, and if one could get the learning for a fraction of the cost, it would only be a matter of time for institutions and students to follow.

I’m now much more of an education skeptic. I largely agree with Bryan Caplan’s view that formal education is mostly signalling. This is the view that school doesn’t so much teach you valuable skills, but helps filter society by which people are smart, conscientious and conformist enough to put up with it. This filtering can explain why schools seem to teach so much useless stuff, yet are nonetheless a requirement for almost any good job.

That view took a long time for me to come around to, mostly because I really love learning the kinds of things they teach in school. This was particularly true during the MIT Challenge itself. Although it may sound strange to say it, I genuinely enjoyed most of the project. It was an intellectual puzzle talking about big ideas I could do at my own pace and with my own process. My actual university classes I did during my undergrad, I enjoyed a lot less.

However, this signalling explanation seems to fit the data, and my post-experiment experience, much better than any other I’ve heard.

This doesn’t mean I think learning isn’t valuable. It just means that a lot of education isn’t really about learning. The kind of learning which is valuable, both professionally and personally, is often at odds with how school curricula are designed. This is a bittersweet realization because it both affirms the idea of the Challenge (that big gains in potential learning are possible, in principle) and undermines it (because such gains rarely happen because, institutionally, we mostly pay lip-service to the ideals of learning while we really care mostly about filtering people).

As I wrote, shortly after the Challenge, if my goal had solely to acquire computer science knowledge and become a better programmer, I might have taken a radically different approach. Much more programming. Way less theory. Especially less of the abstruse mathematical sub-fields of which I did more than a few classes on.

However, if I had taken the Challenge and optimized it solely for acquiring computer science knowledge, it might not have attracted as much attention as it did. In that sense, it may be that elements that caused it to fail at my stated purpose, ended up allowing it to succeed in my career as a writer. Projects which succeed, not in spite of their flaws, but because of them, are a feature I’ve found in many of the things I’ve done in life, even though their paradoxical nature makes them impossible to plan in advance.

What Should You Think About the Challenge?

One thing I learned early about writing is that people rarely read the words you write, as you intended them. They read themselves in between the lines, and fill it in with their own imagined realities. This can often be frustrating, because people want you to be accountable, not only to what you said, but also to the hidden ideas they had of you.

In that sense, there’s nothing to be done. For some people, they’ll like the idea of learning without going to school. Others see a possibility of doing a lot more by rethinking their methods and constraints for taking on a project. Maybe other people just admire the boldness and strangeness of someone who would push himself to grind through classwork and exams full-time for a year without any credentials at the end.

For myself, I think the Challenge offered me many opportunities, both in my profession as a writer, and personally, in changing how I think about accomplishing big goals and learning new things. However, despite the unambiguously good things that came from the Challenge, I think there’s a lot of things that I was wrong about, or could have done better.

In that sense, I hope not that people get inspired by a project I’ve done or article I’ve written and try to do something similar, but that they can also learn from my mistakes and strive to do better than I could have done.

Book Club: The Elephant In The Brain (February 2018)

This month we read The Elephant In The Brain by Kevin Simler and Robin Hanson.

About the book: Human beings are primates, and primates are political animals. Our brains, therefore, are designed not just to hunt and gather, but also to help us get ahead socially, often via deception and self-deception. But while we may be self-interested schemers, we benefit by pretending otherwise.

In fact, the less we know about our own ugly motives, the better – and thus we don’t like to talk or even think about the extent of our selfishness. This is “the elephant in the brain.”

Such an introspective taboo makes it hard for us to think clearly about our nature and the explanations for our behavior. The aim of this book, then, is to confront our hidden motives directly – to track down the darker, unexamined corners of our psyches and blast them with floodlights. Then, once everything is clearly visible, we can work to better understand ourselves: Why do we laugh? Why are artists sexy? Why do we brag about travel? Why do we prefer to speak rather than listen?

This month, I sat down with one of the co-authors of the book, Robin Hanson, to discuss the main ideas of the book.

If you would like to stream audio on your browser, click here listen on Soundcloud.

Broadly, one the takeaways from this book is that politics isn’t about policy, healthcare isn’t about wellness, and education isn’t about learning. One of the first ideas that Robin explains is that:

Most areas in life aren’t central to you and your identity. So in these areas outside of your center, you are much more willing to believe that what goes on there is less than it seems and perhaps even a little fake. But if we get to some that is really close to the center of your identity, something that’s really precious, you’ll find it that much hard to believe. So if you’re an atheist, you’ll find it easy to believe that religion isn’t about God.

When I asked Robin what he thought about how to find the balance between accepting the tension between his own beliefs and what works for getting along well in society, he had this to say:

In all of these areas, the reason that people behave the way they do is because the social incentives push them that way–even if you don’t believe that medicine is very effective for health, you still have to buy it for your family. Similarly, even if you don’t think you’ll have much influence on politics and the outcomes, you still need to act as if you care and act as if you thought you had an influence otherwise people around you might think you are uncaring about your society and the consequences for it. For the most part if you want to get along, you should probably do what everybody does. If you want to be different, you need to pick your battles. I still send my kids to school even if education is not about learning because it is a route to success in our society.

I ask Robin to delineate between the tension of higher, more sincere motives that might be undercut by lesser, faker motives and the idea that there really are selfish motives recast in an altruistic life.

The most high-minded altruistic best motives do exist to some percentage. At the conscious level of our mind we are sincerely trying to help and trying to do good, and trying to learn and trying to get healthy etc. and it’s these underlying motives that systematically shift our behavior to something more effective at other goals and we often don’t even notice the conflicts and if we do we are puzzled and confused and we struggle to overcome it and if we fail we wonder why we fail so much to do this thing we thought we wanted to do when something inside us keeps pushing us the other way.

What is the main takeaway?

I think the mistake we’re making is that we’re taking people at their word for the main goal. We’re designing reforms that help them better achieve that thing they say they want but at some level, we all know it’s not what they want… we need to come up with ways to reorganize systems.

Special thanks to Robin Hanson for being my guest this month. Feel free to join in on our Facebook Group Discussion I’d love to discuss this book with you there. For March, we’ll be reading Dao De Jing by Laozi.