Lots of people such as slatestarcodex are posting the intellectual progress they have made in the past decade, and I thought about doing a similar thing. However, the truth is that in 2010 I was a 15 year old moron who didn’t do anything except play Starcraft. Unlike a lot of people here, I wasn’t a child prodigy and had effectively zero intellectual development at age 15. A decades-long progress review would be a funny exercise since it would include essentially all my intellectual development to the present. Instead I have decided to take this opportunity to start on a habit of yearly intellectual progress reviews. So here is the review for just 2019.
Overall, my estimation of this past year had been somewhat disappointing. I have certainly learnt and developed a lot but I haven’t been developing at a faster rate than last year. My gains this year have primarily been a result of velocity, not acceleration. This is a major problem, especially as there are still low hanging fruit, such as Anki, which I have still failed to implement. My primary goal this year is to implement these capacity raising innovations and thus increase acceleration in development.
One success this past year has been focus on a specific topic in my PhD. This has earned significant dividends and is the primary life skill I have developed. For the first time in my life I have actually focused on a project long enough to ship it, and that had paid off. I am still really bad at this and have way more open projects than completed ones. My aim, at least in the remainder of my PHD, must be to resist the temptation to start a new bunch of projects but rather finish off and ship currently open projects. One of which is this blog, where I have a whole bunch of partially written posts and very few posted. So expect more on this front too.
Anyhow, I see my principal intellectual developments as follows:
I finally understand Active Inference and Predictive Processing at an expert’s level of detail. I have contributed original work to this field which, while not groundbreaking, I believe is valuable. Hopefully next year I will continue my contributions here.
I now understand reinforcement learning and deep reinforcement learning at a deep level. I can read papers at the forefront of the field and understand and critically evaluate them. I hope to contribute original work to this field in the coming year.
I have gotten into the Julia programming language. I had nibbled around the edges in it up since 2017 but never fully got into it In 2019 I made a significant amount of progress and while definitely not an expert, I did complete a major project with it and plan to use it on other things as well.
I have made significant progress mathematically understanding robotics and control theory. Before this year I didn’t even have a really good idea of what these fields were like but now I feel like I have a good basic understanding, and it does inform my work. This was a result of competing several lecture courses in this field.
A second avenue of progress has been building a foundation of understanding of statistics. Previously I had only ever done a (pretty bad) undergrad psychology statistics course before and though I had some scattered understanding as a result of machine learning, I lacked a firm foundation in the basics. I feel that I started to build that this year some basic foundations in this subject and I have made a fair bit of progress in it. I understand the basics more deeply than before.
Beyond this relatively little progress was made in computer science or other related subjects. My maths remains pretty shaky and suspect. My key goals for next year are primarily to focus and complete my PhD which will likely mean reduced investment in extracurricular learning, to the long term detriment of my intellectual development, but hopefully long term gains in potential due to the signalling benefits of having a PhD. Other than that since I think I know enough in my subject to try to make original contributions, my primary extracurricular goal is to develop my understanding of the mathematical foundations. Ideally, I would achieve basic maths proficiency through courses such as real analysis, complex analysis, topology, and abstract algebra to deepen my understanding of the fundamentals before deep diving into ever more abstruse regions. This will take a lot of time, however, and is almost certainly a multi-year project.
In term of additional focuses to learn this year mathematically, I have information theory and then ultimately information geometry seem important. Nonlinear filtering theory and SDEs would be a good thing to start getting some traction on. I also want to start branching out a little more into other subjects. Specifically, I’d like to look more deeply into economics, to help me understand the world more, and biology because it’s super interesting and biotechnology will likely become massively important by the end of this decade.
My reading progress was very poor and last year I read very few books, probably only about 10. I need to improve this dramatically this year. I did well with video lectures and most of my gains of understanding come from this. I think I do well with this format.
Below is a summary of key lecture courses completed per month in 2019.
January: – University of Edinburgh Natural Computing – genetic algorithms, ant colony optimization. Not especially useful except knowing that this field exists. Lots of fairly useless details about algorithms that work terribly now compared to deep learning but still useful for broadening knowledge.
– Human Behavioral Biology –. Really good lectures… incredibly fascinating. Not at all relevant to my studies but really cool. I Need to pick up his book to relearn and review the material.
– Dayan Computational Neuroscience –. Really interesting. Lots of detail I didn’t retain. Since I had no real excuse to use it a lot of it faded. Oh well.
Feb: Bartosz Milewski Category Theory – Really interesting but a lot of it faded and I lost interest towards the end. I know a fair bit about functions and categories although I don’t think it imparted the ‘categorical way of thinking’ very deeply. I think this is one of the things where you have to already be in a specific intellectual place to truly get and I’m not there yet.
March: None.
April: Deep Reinforcement Learning Sergey Levine – This was really great. I’d already gone through David Silver’s course but this really also improved my understanding and brought me up to modern state of the art.
Control Bootcamp, Steve Brunton – This was a really great intro to key parts of control theory in linear time invariant systems. Super cool and interesting. Very different seeing the engineering side of things.
Graphical Models and Variational Inference – A good course but I didn’t finish it. An intro to continuous time and winer processes etc.
May: None.
June: Statistics 100 Harvard – a great course covering the fundamentals of statistics. A lot of this was review of things I had already half-learned. But it was nice to get it all in one coherent place.
July: All of Statistcs, Larry Wasserman – ANother great statistics course which was very useful for solidifying the basics. I now understand moment generating functions (!)
August: MIT Signals and Systems. – Very fascinating and interesting. Essentially a beginning engineering concept and helped solidify a lot of intuition about basic ODEs. Also a lot of digital signal processing basics which is useful to know vaguely what’s going on in that field.
Numerical computation – A very helpful course for understanding how to actually do linear algebra on a computer. Lots of core information on things like the Cholesky Decomposition which I had vaguely been taught in my masters, but forgot. Working through the error bounds on floating point computations was also enlightening.
Sept: Control theory – this was a very useful lecture series that I think went above and beyond the earlier control bootcamp.
October: Underactuated Robotics, Russ Tedrake – This was an absolutely fantastic lecture series which solified a lot of concepts in control theory and robotics and intorduced me to some more state of the art methods. The course notes are fantastic. I would reccommend tihs course highly.
November: None
December: None.