Universities are faced with a problem when hiring new staff: academics are the lifeblood of universities, but are expensive and almost impossible to remove once in place. This means that there is an enormous amount of pressure on hiring committees to identify those researchers who will prove fruitful once they are in post, but those predictions rely on extrapolation from a relatively short track record. Specific metrics that UK universities want to maximise are:

1. Research income (keeps the lights on, maintains league table positions)
2. “Impact” beyond the academic sphere (in Research Excellence Framework jargon: “Impact Case Studies” – you can see a few of my colleagues Case Studies here)
3. High quality publications (number isn’t an issue, but the REF demands four publications from each academic in each 5-6 year reporting cycle)

Obviously the dilemma is this: when hiring a junior academic, that candidate will not have had an opportunity to apply for funding (often because they simply weren’t eligible without a permanent position). They will also not have had a sufficiently long career to be able to demonstrate the impact of their work. Most Impact Case Studies rely on 10-year timescales, from inception of a project, funding for that project, completion, dissemination, and implementation of findings. The only thing that academics can be judged on as they apply for jobs, then, is their publication list (which will also be short, because they’ve only just started!).

This is where the Sports Illustrated curse comes in. For those of you not familiar with the publication, Sports Illustrated always features a leading sportsperson (or sometimes, perhaps more famously, semi-naked women) on its cover. This person is usually at the top of their game and has accomplished something impressive in the period leading up to the publication of that issue. However, readers and pundits began to notice that following the publication of issues on which they were the cover star, those athletes would suddenly experience a drop in performance or a run of “bad luck” – a phenomenon which was described as the “curse” of the Sports Illustrated cover stars. Various examples are cited including…

• 1963: Jack Nicklaus is featured as the 1962 US Open golf champion in the run up to the tournament. He missed the cut in 1963.
• 1988: Michael Spinks appeared on the cover just before his fight with Mike Tyson. Tyson knocked him out in 91 seconds, ending Spinks’ career.
• 2015: Serena Williams stars on the cover (see above) touting a potential Grand Slam (all major titles in a single calendar year) before going on to lose in the US Open to the unseeded Roberta Vinci.

There is a simple, statistical explanation for this, and that comes in the form of “regression to the mean”. Almost any phenomenon you can think of that can be quantified can be described using two variables: a measure of central tendency (think “average”), and a measure of error (think “variability around that average”). Nicklaus, Spinks, and Williams were all at a high point in their careers, but that was a short-term spike and not a long-term trend. When they “suffer from the curse” that is simply their performance returning to the background average, or “regressing to the mean”.

The SI curse has two major implications for academic hiring. The first is that most hiring is done on the basis of top ranking (in my field, Nature and Science) publications over short careers. Since those publications are (arguably) not really indicative of the true ability of those academics, hiring committees are making decisions based on unreasonable criteria.In addition, those academics who have obtained a top-ranking paper often do so as a junior part of a larger team (e.g. under their PhD or postdoctoral supervisors). Inevitably there are those academics who have far greater potential but who have not had the lucky break to get a top-ranking paper, and are effectively excluded from tenured positions as a result. Both the academics and the wider community suffer as a result. The second implication is that the drive for REF-returnable papers in the UK emphasises the use of these high ranking papers despite the fact that the young academics might not be able to sustain that level of productivity. The result then is that the young academics become valued only for that paper, potentially finding themselves resented or burdened with undue amounts of admin or teaching when colleagues realise that they are not “geese who lay high impact factor eggs”.

This post was inspired by a few posts on Twitter from Dennis Eckmeier who cited a couple of papers that have attempted to establish what it is that predicts future performance in academics. The first of these by von Bartheld et al. (2015), demonstrated that past citations and high impact factor papers had very little predictive power. Instead, it is the number of first-author papers that an academic produces that best-predicts future success (defined as external grant income from any source, and at least one paper as senior author in each year). Bizarrely, there seems to be a clear break-point at around six first-author publications over a 10-year period:

Furthermore, the number of papers that an academic publishes full-stop has a strong effect on the probability of success:

The second paper that Dennis cites was by van Dijk et al. (2014), which also showed that the number of publications was important, but added journal impact factor and a measure of article citation rate. Those authors go so far as to provide a calculator that will allow you to predict your probability of becoming a “PI” (“principal investigator”) based on their statistical models: http://www.pipredictor.com/. Sadly, gender is also a factor even when all of these factors are controlled.

So what is a hiring committee to do? In my department, we advocate for those academics who have not yet had the opportunity to demonstrate their ability to obtain grant funding, look at quality rather than impact factor of work that has been produced, and place great stock in research plans submitted alongside CVs as an indicator of potential future performance. However, this can be difficult when hiring committees are drawn from a broad range of disciplines (and especially when the management are not familiar with the subject area). In those cases, impact factor and journal reputation become one of the few common languages. I don’t know what the solution is, but an emphasis on REF and impact factors certainly isn’t helping.

References

van Dijk D, Manor O, Carey LB (2014) Publication metrics and success on the academic job market. Current Biology 24: R516 – R517

von Bartheld CS, Houmanfar R, Candido A. (2015) Prediction of junior faculty success in biomedical research: comparison of metrics and effects of mentoring programs. PeerJ 3:e1262 https://doi.org/10.7717/peerj.1262