You know, I'm something of a modeller myself
I recently read a blog post written by Joel Hellewell, who worked on the COVID-19 response team at LSHTM. The post is here https://jhellewell14.github.io/2021/11/16/forecasting-projecting.html. I found it particularly interesting to hear the perspective of someone who had worked on (mathematical) modelling infectious diseases for a longer time, and how the response to the pandemic compared to these activities.
I wanted to write a reply which turned out to be a bit too long for a tweet, so here it is. (background: I spent Apr-Dec 2020 working part-time with one of the COVID-19 modelling groups at Imperial’s department of infectious disease epidemiology; my contribution was pretty much entirely to the software/programming side, not the modelling.)
Have forecasts been useful?
While scenarios were used for policy planning in England (e.g. to guide the scheduled easing of NPIs in 2021), inference from models was used to understand the transmission process, I’m not clear whether short- or medium-term forecasts were ever used/useful for this. I would be interested to know how useful predictions for number of cases, hospital beds, and deaths have been in England. It appears to me that policy decisions happen on much slower timescales than ~weekly, but maybe the NHS is able to more quickly respond to forecast demand?
I’d also agree with the post that any longer term forecasting for most diseases is very difficult, but especially one where unexpected far-reaching policy decisions and significant adaptive mutations come into play during the forecast period.
Time pressures
Joel’s post argues that there is little motivation to assess accuracy of forecasting results, as this kind of activity doesn’t directly feed into career success (broadly equal to publishing ‘high-impact’ papers). At least in the team I was in, my impressions was that the modellers spent a huge amount of time scrutinising their own work, comparing it to others, and generally being very thorough in their process.
[Joel responds that he specifically meant retrospectively checking predictive accuracy of forecasts against what actually happened; all SPI-M groups follow good practices when creating models, but none do these checks routinely.]
For us (me?), the pressure to publish wasn’t a dark spectre looming behind every decision, which was good, but I think it’s fair to say there was a collective sense of frustration that by late 2020 we had yet to publish our work, whereas other groups had produced multiple career-defining papers. Instead, there was a constant demand for production of further scenarios and forecasts (not generally formulated by the group themselves) which to me felt more and more like it got in the way of the science-driven side of the work.
Capture by philanthrocapitalists and friends
Just an observation that this seems to have really ramped up in genetics too. I feel like I read a very big tech-type post at least once a week now touting the achievements of their funding and (often internal) fundees, but with very little actual scientific substance.
Why be a scientist?
I want to be able to have at least some time to pursue things I enjoy out of interest rather than following a rigidly-defined project plan to a set time (which for me usually means doing something technical, or making a visualisation). So, something I think I personally learnt from COVID-19 was that I don’t really want to be in a public health role, or one that feeds directly into policy making.
But will this even be possible post-pandemic if you want to have a career in infectious disease research? I hope so.
Generally, I’m more optimistic that science can be what we (scientists) make of it, and we do have the ability to change this trajectory towards something kinder and more enjoyable. Particularly, younger scientists I work with almost always have positive attitudes towards issues such work-life balance, inclusivity, for-profit publishers and open science, and the means by which we quantify research output and impact.