My 2021 Forecasts, and Why 2022 is Slow

I can't say for sure that my interest in predictions originated from the Astral Codex Ten blog, because I was already an avid media consumer of people like Phil Tetlock (author of Superforecasting) and Nate Silver (Founder of Five Thirty Eight). But this format for structuring and presenting predictions is definitely an ACX rip-off. I want to do similar things for 2022, but I'm slowing down for two reasons.

1. I noticed that the number of predictions for 2021 was far too low to get any kind of calibration on my predictions. You can't tell if you are overconfident or underconfident when you only predict two things at 60%. Unfortunately, increasing the number of predictions has a very obvious trade-off, it takes time! I had originally hoped to do my 2021 predictions with my friend Aaron Kirkey, but he balked when he saw the list of questions, and procrastinated until at least half of the predictions had resolved themselves. It takes time to predict, and it can take at least as long to create a long list of interesting questions that will be easy to resolve. I had originally hoped to make 2022 predictions over the winter break, but life is busy, so here we are.

2. I wanted to focus on scientific predictions. And I wanted to make scientific predictions in ways that could influence a future career path, and where I could point to them and say "look at me doing smart things over here". I kind of wanted a separate forum for those predictions, so that they didn't have to be on the same blog as my weird poetry thingy or some edgy atheism nonsense. Since I'm actively looking for careers after I finish my PhD, I wanted to do that first. But then we run into the time problem again. I want to write professional posts about smart prediction questions, all on a different platform? It hasn't happened yet.

The solution, I think, is to throw up last years results here, add in some ugly graphs, and check that off my list. I'll probably focus on making general 2022 predictions next (I've got a headstart between some ideas that Aaron sent me and some NHL predictions, and I'm going to count both). Then finally, I'll focus on the professional quality chemistry predictions once I've got the easy stuff out of the way and I can focus. Hopefully all will happen in short succession.

---

2021 Predictions. Bolded predictions were correct.

The following were all made February 21, 2021

1. Jasper, Aaron, and Ellen will be fully vaccinated against Covid-19 (2 shots if needed) - 90%

2. Health care workers in Canada will have received "booster shot" vaccines for new variants. - 30%

3. Jasper and Aaron will have seen each other in person at least once. - 70%

4. Canada will have held a federal election. - 90%

5. CBC or CTV will cover a federal politician travelling to Mexico or Hawaii, not for business. (after February 21st) - 60%

6. The Calgary Flames will play an NHL playoff game - 50%

7. The San Jose Sharks will play an NHL playoff game - 10%

8. The Utah Jazz will win the NBA championship - 20%

9. A team will be awarded the Grey Cup - 70%

10. Fans will attend an indoor NHL game in Canada - 80%

11. A city in Canada will hold a marathon event with >500 contestants - 40%

12. Facebook stock price will be above $400 (on opening January 1st) - 10%

13. The price of (Brent Crude) oil will be greater that $64 (price on February 21st) - 50%

14. Joe Biden will be president of the USA - 95%

15. Boris Johnson will be Prime Minister of Great Britain - 80%

16. Justin Trudeau will be Prime Minister of Canada - 70%

The following predictions were made on November 21, 2021. They are a good example of why I still want to keep these private predictions at least nominally separate from professional-looking science predictions.

17. Kyle Rittenhouse will be found not guilty on all counts - 80%

18. Conditional on "not guilty" there will be no 'serious' rioting in USA one week afterwards (I defined serious elsewhere) - 80%

---

By treating a 20% probability as an 80% probability of the opposite, I can get:

95%: 1 correct, 0 incorrect

90%: 4 correct, 0 incorrect

80%: 5 correct, 0 incorrect

70%: 2 correct, 1 incorrect

60%: 0 correct, 2 incorrect

50%: 1 correct, 1 incorrect

And I can graph it as follows:

---

Takeaways:

• You need a lot of predictions before you can judge calibration at all levels. Two predictions at 60% tell me nothing about my ability to predict things that occur 60% of the time.

• I'm probably underconfident where it matters. The chance that an 80% probability happens 5 times without fail is roughly 1/3, so certainly not impossible, but suggestive that I should round most of my confident predictions higher.

• I got a similar result from an online calibration test. I didn't save my results, but I think that everything that I gave at least an 80% probability to was correct, and it contained a fairly large sample size. It's a fun test to do if you have the time.

• Despite saying that I was"underconfident" it really sucks to be wrong. I predicted even Canadian health care workers wouldn't have a booster shot by the end of the year, when even I had a booster shot by then. If I had only given it a 20% chance rather than 30%, I would have felt even worse. Nevertheless it would have meant I was much better calibrated on my 80% guesses.