You are at a stag do and in charge of paying the bill and you need to do long division which literally no one knows how to do by hand. So, you attempt to divide £2344 by 24 (lads) into your calculator. But because the multiple rounds of black sambuca adds to your already poor levels of coordination you drunkenly mash the last digit and end up typing in 2344/26. Inevitable you get the wrong answer and argue with the waiter for half an hour.
Who is at fault here? Could it possibly be you? No, it is the calculator’s fault. The calculator, being a sentient and telekinetic being should have known you meant to type 4 and not 6, like you definitely did do with your actual physical finger.
The problem of designing algorithms is similar to that of designing laws. After all, laws are basically simply algorithms without numbers.
Writing laws are particularly difficult because of this. We
need caveats and exceptions, we need careful definitions. But even this process
isn’t perfect. We still need judges and juries to interpret the law on a case
by case basis.
The problem with laws and algorithms is that they can have
unintended consequences. The "do not steal" law did not intend for a person to bleed out on the street. The question which we should then ask is who is to blame for these unintended consequences? Can we really blame someone who did not intend for these things to happen when they were acting in good faith? In the A-levels case the answer is yes
Firstly, I do not think we should blame the designers of the
algorithm for having unintended consequences. It is not like they didn’t think
things through or create an overly simple law. They did carefully check for certain
things such as if their algorithm was favouring certain groups.
But when we make algorithms, we need to be sufficiently prepared
to deal with unintended consequences.