Rankings, Goodhart's law and the REF

There was an old lady who swallowed a fly

I don't know why she swallowed a fly - perhaps she'll die!

I really dislike rankings. I feel they are a way to make subjective data seem objective and hide a lot of information that we care about. There is a whole chapter dedicated to why they are bad in my book.

One such ranking, the Research Excellence Framework AKA REF2021, has just been released (it was meant to come out last year but got delayed because of the pandemic, hence 2021). It basically assesses the “quality” of university research. To do this, academic panels assess research papers submitted and are awarded a score of 4* (quality that is “world-leading”) to 1* (quality that is “recognised nationally”). Quite what this means is anyone’s guess (especially as the papers being assessed have already been published in academic journals who already have a sort of quality ranking). But once you have done all that you can calculate a mean score for each subject across universities.

Quite often the results cluster around 4* and 3*, so the mean average you get usually sit around the decimal places, 3.12, 3.5 and so on. And as some departments are so small, a marginal decision (say someone judging a paper as a 4 rather than a 3) can affect your average by a few decimal points. This might not seem like much, but when it comes to rankings where everyone average is basically the same, you can easily jump up 10 places. I think due to shear amount of work involved in this, people are probably quite reluctant to admit that a lot of the variation in these rankings are random.

But I think the biggest issue with the REF is that it is a perfect example of Goodhart’s Law (another chapter in my book). It states, that when a measure becomes a target, it ceases to be a good measure. And universities certainly try and target the REF - it is government policy and in the job description of all VCs across the land. 

What this means, is that there is a large incentive to “game” the system, to try and get as highest REF score as possible. How can you game the system? Well you could potentially not include some individuals who you don’t think will score so high, or perhaps hire a few big hitters on a temporary basis. All of these things were noted when the last REF was done in 2014, so how did we respond? Well, we created more rules. 

Football is a simple game, two teams run around for a bit and try and put the ball in the oppositions net. But when the game was first invented, the problem of goal hanging made for poor games. So the offside rule came into being. In 1863, a player was considered offside if there were 3 of the opposing players in front of him. In 1925 this evolved to two players and in the 1990s this changed again, to being level with the 2^nd to last player. There have been multiple changes since and the reason is that each time a new rule is made, players try and find a way to game it. Whether it is the offside trap or interfering with the goal keepers view, no matter what rule you create people will have an incentive to try and game it.

To that end we now have VAR to enforce the offside rule. But to many people, VAR really takes the joy out of a last minute winner. Which is kind of ironic as the whole point of introducing the offside rule back in 1863, was to make the game more enjoyable to watch!   

This is perhaps the biggest problem with the REF. No matter what rules we impose, if we incentives trying to do well in it, people will try and game it. It is like the old lady who swallowed a fly. Rather than taking the L she decided that in order to fix this problem she would swallow a spider, which created a further problem because now she has to find something that will deal with the spider - so she swallows a bird. You end up getting so lost in trying to fix problems that you lose sight of the original problem you were trying to fix. So what is the REF for anyway?

Society doesn’t particularly care about relative rankings of universities or the REF per se. What it does care about, if at all, is getting universities to produce research like creating new vaccines, understanding the universe and whatever I am currently working on.  If university increased its research "quality" by 1000% it would have a much better outcome for society (although you wouldn't be able to tell from looking at rankings, another reason why they suck).

What the REF has certainly done, is increase the amount of research output by incentivising it. But does that result in scientific progress? I am very sceptical that focusing on research outputs is a good way of going about it. Spending a lot of time and recourses to create something that isn't a particularly good measure of research quality doesn't seem like a good deal to me.

What causes red tape?

Government plans to cut red tape, again. I mean, getting rid of excess bureaucracy seems like a no-brainer as by definition excess bureaucracy is well, excess. The more interesting question is why was red tape put there in the first place?

Having to wear a hard hat on a building site, even if there is clear sky above, infuriates people. It lacks “common sense”. But given the large number of people on building sites, a discretionary policy would most likely result in a tragic accident. People are more likely to forget to put them on a hardhat if it isn’t mandatory at all times. Just as in the Covid era we traded off rights for lives, there will inevitable be some debate about what the level of regulation there should be. What I find quite difficult, however, is finding examples regulations where there would be no negative consequences of getting rid of them.

It was easy under the EU to get annoyed about excess regulation, as the government can always blame someone else for its imposition and say there is nothing that can be done. But now there is no one else blame it is going to be quite difficult for the government to change anything. If they try and change any regulation there will be a backlash because some people will be unhappy (e.g. US trade deal chlorinated chicken saga).

Saying this, we often don’t introduce regulations until something bad has happened. If you ever see a sign saying “slow” on a road, it is because someone has tragically died there. Whether or not putting a sign saying “slow” actually helps save future lives, we must be seen to be doing something in response to a bad outcome. Reactionary bureaucracy is more likely to be excess as a result.

This sort of bureaucracy is especially prevalent in management. If pens go missing in the stock cupboard at work, the management invariable needs to be seen to do something about it. If not, they have to explain to their bosses why they haven’t done anything about it.

So perhaps the manager increases the amount of stock takes or makes workers fill out a detailed forms to use a pen. Regardless of whether this improves the situation, at least the manager’s arse is covered. What the company have not done, however, is think about the opportunity cost.

As I pointed out in my anti-waste blog, a lot of the red tape created by governments in the public sector is its own attitude to waste. I have even resorted to buying my own whiteboard markers as it just isn’t worth the effort trying to source them (if you are as outraged as I am you can rectify this injustice by reimbursing me on my gofundme page here).

When a new minister comes in, they invariably want to make a mark on things. For some reason, just keeping things chugging a long doesn’t seem to cut it with voters. And this is why you get policies like changing GCSEs from being graded alphabetically (A-F) to numbers (1-9). A simple enough idea to announce, but when you consider the amount of forms and systems that need changing, it just becomes a massive pain.

So rather than the government always trying to cut tape, they should reflect upon the type of policy making that creates red tape in the first place. 

Nimbyism: who has the right to decide what happens in a local area?

A Nimby is a person who opposes new developments in their local area, an acronym of Not In My Back Yard. The term has certainly taken off in recent years with debates about the amount of housing being built in Western countries.

What I find interesting about Nimbyism is that a Nimby doesn’t have to be against housing per se, they may actually think it is a good idea for more housing to be built in society in general. What they are against, however, is housing being built near where they live. You can see where this situation ends up, a sort of tragedy of the commons. If everyone is a Nimby, then there are no back yards to build in, anywhere.

Nimbyism may come from a self-interested perspective: if you own a house in an area and more housing gets built, it could potentially lower your house price. And adding more people will mean more congestion on local roads and longer waiting times at the hospital, if no more infrastructure gets built. It could even be you just don’t like the look of new developments. The fact of the matter is, a lot of these new developments benefit other people and so it doesn’t take much to tip you over the edge into Nimbyism.

What would a Steelman case for Nimbyism be? I think there are actual reasons why you may want to be against certain things being built in your locality if you think it would have a negative impact - not just for those currently living in the area, but for future generations as well.

As a relatively recent resident of Durham I would like to say I now feel part of the city. But I am still aware that I haven’t been here all that long and I don’t know how long I will stay. Do I have the same vested interest in the city as someone whose family as lived here generations? I would like to think so, but it is not a straight-forward question to answer.

One of the most interesting documentaries I have seen in a while is Wild Wild Country on Netflix. The documentary is about an Indian Guru, Bhagwan Shree Rajneesh. He moves to the US and buys some land for him and his many followers near the town of Antelope, Oregon, which has a population of about 50 people. Tensions rise between the locals and the new settlers. Eventually Rajnessh has enough followers in the area to outvote the locals to the extent of changing the name of the town from Antelope to Rajneesh. All sorts of weird things happen after that and its certainly worth a watch (if you have seen the “joy of sect” episode of the Simpsons, Rajnessh is who the cult leader was based on). But I think this documentary highlights an interesting facet about the morality of Nimbyism.

Does the fact that the local residents of Antelope, having been there for a long time, mean they have a greater right to have say in their locality? Or does Rajnessh and his followers have the right to be there and make changes to the town through democratic means?

Although this is an extreme example, I do think we have something to learn from thinking about who has the right to decide what happens in a local area. Taking a strong position either way can get you into difficulty. For example, some people are concerned that building new houses will create gentrification: rich people moving into poor neighbourhoods. But if you give more powers to local areas, it may lead to rich neighbourhoods preventing poor people from moving there. Gentrifstation, if you will.

So I think if we are going to solve a problem like, Nimbyism it will need to involve a larger discussion about the politics of the local.

The problem with being anti-waste

Finish your dinner, some people are starving in the world. Although this phrase is often used by parents to try and emotionally manipulate their children into eating vegetables, it reflects a deeper societal concern about our attitudes to waste.

If you do not have enough of something you need then waste is a serious problem, a matter of survival. Perhaps this is why we react so negatively to waste when we have abundance, to waste is to forget how privileged we are.

But defining waste isn’t as straightforward as it seems.

How do you peel a potato? Well, arguably the fastest way is to simply turn the potato into a cube as you only have to make 6 cuts with a knife. Granted you will lose a lot of perfectly edible potato this way, but it is a lot faster than the standard way of peeling which involves lots of movements to remove the peel.

The thought of doing this to a potato has probably filled you with deep sense of moral outrage. But I think it is important to reflect on why the potato is a valuable resource and not your time? Put another way, would you spend 10p to save you a few mins of extra peeling time? I think I probably would, but I would hesitate if it were 10 pence worth of perfectly edible potato (incidentally many people throw a way potato peels but if you put them in the oven with a bit of olive oil and they make a great snack!). I think we probably have an in-built aversion to this kind of waste for all sorts of evolutionary reasons. But what actually is waste?

Waste is an useable by-product of the production process, what’s left over. It is intrinsically tied to efficiency: the more efficient the process the less waste that occurs. The problem here is that not everything that minimises waste, leads to efficiency gains. After all, the easiest way to minimise waste when peeling potatoes, is to simply not eat potatoes in the first place.

The problem with focusing on waste is that it usually just concentrates on minimising waste in one resource, and not thinking about the process as a whole. The public sector is often viewed as wasteful as it doesn’t have the profit motive to keep costs in check. If you want to spend any money in the public sector it invariably involves filling out lots of forms to justify that this spending is absolutely necessary. This will obviously reduce the waste in the sense you will have fewer gold-plated toilet seats, but in reality it means that you are paying someone to fill out forms rather than doing their job.

This notorious example of this is the NHS getting rid of managers and getting doctors to do administrative tasks instead. You brings down the overall wage costs but in the end you are paying doctors, doctor’s salaries, to not spend time with their patients. And because quality of care is lot harder to measure, you end up seeing “waste” in all things that are easier to measure like wages. It is management by spreadsheet (but not in a good way).

Minimising waste leads you to get rid of any slack in the system – ignoring the fact that this is often the most efficient way of running things. Negative shocks happen all the time, people get ill or leave. And then more people get ill or leave because they are trying to do the jobs of people who are ill or on leave. If at work you are constantly putting out fires at work, then the system is not running as efficiently as it could be.

Now the elephant in the room when it comes to waste is climate change. Energy is something we do not want to waste as the by-product is carbon emissions. And it is because it is so important we need to be very careful when assuming something is wasteful. For example, people often think that food grown locally is more environmentally friendly because it cuts out transport costs. But transport costs are such a small component of the energy use that goes into food production that we can end up using more energy eating locally if we don’t grow things in the most productive areas.

So I guess my argument is that to be truly anti-waste, you need to focus on long-term efficiency and not make me fill out loads of forms.