Algorithmic accountability, NYC edition


New York by Daniel Schwen (Own work) [CC BY-SA 2.5], via Wikimedia Commons

There is an interesting attempt to create algorithmic accountability in New York.  It seems that they will attempt to pass this into law:


If this was applied in the private sector it would probably fail horribly. But in the public sector, what are its strengths and limitations?


As I’m going to spend a lot of time looking at difficulties, it’s only fair to start with positives:

  • It’s simple – it doesn’t attempt to be unnecessarily complex
  • It introduces the idea that people can look at impacts, rather than just code

The second of these looks like an interesting way forward. I might suggest an alternative, which would be a ability to evaluate the overall impact of full result sets – thus enabling you to investigate biases in the round, rather than focusing on individual results, some of which will always be wrong!

Definition of an algorithm and data

This is probably the first challenge. When is something an algorithm? When does it use data? If I have a set of actions that I always take in a certain order (for example, I open the city’s parks in a specific order) is that an algorithm? Even that simple example impacts people, as park A is open 5 minutes before park B…

And what is data? Does it have to be in a computer? How about if it’s written down?

Generally I’m in favour of opening decisions to scrutiny, but I firmly believe it should be all decisions, not just computer decisions!

What is source code?

A naive reading of this would suggest that source code could be as simple as pointing at the R code behind an approach. Or it could mean publishing the actual model.

The first is easy to do, the second isn’t. Trained models aren’t necessarily designed to be published in an interpretable way (a bit like the difference between compiled and uncompiled code) – so should we limit approaches to ones where a ‘raw’ or interpretable model could be generated? Even if we could, it might not mean much without the software used to run it. In essence it might be worse than useless.

Another challenge is where does a model begin? A lot of time is spent preparing data for a model. Up to 80% of the time when generating a model. If you just show the model, without describing all of the steps that are needed to transform the data, then you are going to severely mislead.

Data submission

But what about allowing users to submit data and see the impact? This is an interesting idea. But it too has some interesting consequences.

What would you do if someone actively used this to game the system? Because they will. Yes, you could monitor use, but then you end up in another area of potential moral difficulty. And it’s one thing if someone is using it to understand how to maximise the benefits they receive (actually I kind of approve of this), but what if they are using it to understand how to avoid predictive policing? And can the police use this data to change their predictive policing approach?

Another interesting problem is that often a single result doesn’t tell you much. Yes I can see that my zip code has a predictive policing score of 10. But unless I know the predictive scores of all other zip codes, plus a huge range of other things, that doesn’t tell me much.

And how do you stop people from using it to spy on their neighbours? Entering in other people’s data to find out things about them?

Unintended consequences

Finally, some thoughts about unintended consequences. Will this discourage effectiveness and efficiency? After all, if I use a dumb human to make a bad decision, then I won’t be held accountable in the same way as if I use a smart algorithm to make a good decision. And this is important because there will always be mistakes.

Will there be any attempt to compare the effectiveness against other approaches, or will there be an assumption that you have to beat perfect scores in order to avoid legal consequences?

Will vendors still be willing (or even able) to sell to NYC on this basis?

Final thoughts

I think this is an interesting approach, and certainly I’m not as negative about it as I was originally (sorry, shouldn’t tweet straight off a plane!). Thanks to@s010n and@ellgood for pointing me in this direction…


How in love with AI are you?

AI is a problematic term at the moment. There is an awful lot of conflation between true existential/ubiquitous computing/end of the world AI on the one hand, and a nerd in a basement programming a decision tree in R on the other.

Which makes for some nice headlines. But isn’t really helpful to people who are trying to work out how to make the most (and best) of the new world of data opportunities.

So to help me, at least, I have devised something I call the LaundryCulture Continuum. It helps me to understand how comfortable you are with data and analytics.

(Because I realise that the hashtag #LaundryCulture might confuse, I’ve also coined the alternative #StrossMBanks Spectrum).

So here are the ends of the Continuum, and a great opportunity to promote two of my favourite writers.

In the beautiful, elegant and restrained corner, sits The Culture. Truly super-intelligent AI minds look on benevolently at us lesser mortals, in a post-scarcity economy. This is the corner of the AI zealots.


In the squamish corner sits The Laundry, protecting us from eldricht horrors that can be summoned by any incompetent with a maths degree and a laptop. This is the home of the AI haters.


Where do people sit? Well it’s pretty clear that Elon Musk sits towards The Culture end of the Continuum. Naming his SpaceX landing barges Just Read The Instructions and Of Course I Still Love You is a pretty big clue.

The Guardian/Observer nexus is hovering nearer The Laundry, judging by most of its recent output.

Others are more difficult… But if I’m talking to you about AI, or even humble data mining, I would like to know where you stand…

In defence of algorithms

I was going to write a blog about how algorithms* can be fair. But if 2016 was the year in which politics went crazy and decided that foreigners were the source of all problems, it looks like 2017 has already decided that the real problem is that foreigners are being assisted by evil algorithms.

So let’s be clear. In the current climate people who believe that data can make the world a better place need to stand up and say so. We can’t let misinformation and ludditism wreck the opportunities for the world going forwards.

And there is a world of misinformation!

For example, there is currently a huge amount of noise about algorithmic fairness (Nesta here , The Guardian here et al). I’ve already blogged a number of times about this (1, 2, 3), but decided (given the noise) that it was time to gather my thoughts together.


(Most of) Robocop’s prime directives (Image from Robocop 1987)

tldr: Don’t believe the hype, and don’t rule out things that are fairer than what happens at the moment.

Three key concepts

So here are some concepts that I would suggest we bear in mind:

  1. The real world is mainly made up of non-algorithmic decisions, and we know that these are frequently racist, sexist, and generally unfair.
  2. Intersectionality is rife, and in data terms this means multicolinearity. All over the place.
  3. No one has a particularly good definition of what fairness might look like. Even lawyers (although there are a number of laws about disproportionate impact even then it gets tricky).

On the other side, what are the campaigners for algorithmic fairness demanding? And what are their claims?

Claim 1: if you feed an algorithm racist data it will become racist.

At the most simple level yes. But (unlike in at least one claim) it takes more than a single racist image for this to happen. In fact I would suggest that generally speaking machine learning is not good at spotting weak cases: this is the challenge of the ‘black swan’. If you present a single racist example then ML will almost certainly ignore it. In fact, if racism is in the minority in your examples, then it will probably be downplayed further by the algorithm: the algorithm will be less racist than reality.

If there are more racist cases than non-racist cases then either you have made a terrible data selection decision (possible), or the real problem is with society, not with the algorithm. Focus on fixing society first.

Claim 2: algorithmic unfairness is worse/more prevalent than human unfairness

Algorithmic unfairness is a first world problem. It’s even smaller scale than that. It’s primarily a minority concern even in the first world. Yes, there are examples in the courts in the USA, and in policing. But if you think that the problems of algorithms are the most challenging ones that face the poor and BAME in the judicial system then you haven’t been paying attention.

Claim 3: to solve the problem people should disclose the algorithm that is used

Um, this gets technical. Firstly, what do you mean by the algorithm? I can easily show you the code used to build a model. It’s probably taken from CRAN or Github anyway. But the actual model? Well if I’ve used a sophisticated technique, a neural network or random forrest etc, it’s probably not going to be sensibly interpretable.

So what do you mean? Share the data? For people data you are going to run headlong into data protection issues. For other data you are going to hit the fact that it will probably be a trade secret.

So why not just do what we do with human decisions? We examine the actual effect. At this point learned judges (and juries, but bear in mind Bayes) can determine if the outcome was illegal.

And in terms of creation? Can we stop bad algorithms from being created? Probably not. But we can do what we do with humans: make sure that the people teaching them are qualified and understand how to make sensible decisions. That’s where people like the Royal Statistical Society can come in…

Final thoughts

People will say “you’re ignoring real world examples of racism/sexism in algorithms”. Yes, I am. Plenty of people are commenting on those, and yes, they need fixing. But very very rarely do people compare the algorithm with the pre-existing approach. That is terrible practice. Don’t give human bias a free pass.

And most of those examples have been because of (frankly) beginners mistakes. Or misuse. None of which are especially unique to the world of ML.

So let’s stand up for algorithms, but at the same time remember that we need to do our best to make them fair when we deploy them, so that they can go on beating humans.


* no, I really can’t be bothered to get into an argument about what is, and what is not an algorithm. Let’s just accept this as shorthand for anything like predictive analytics, stats, AI etc…