The worst big data article ever?

There are many, many bad articles on big data. It’s almost impossible to move without tripping over another pundit trying to rubbish the topic. Partly this is just the inevitable sound of people trying to make a name for themselves by being counter-factual. It’s far easier to stand out when you’re fighting against the tide. Even if you end up getting very wet…

Fortunately St Nate of Sliver has actually analysed the data and it’s clear that pundits are fundamentally useless.

But occasionally you come across something so egregiously crap that you have to comment.

This week’s crap-du-jour comes courtesy of Tom Leberecht and Fortune.

In it he decides to lump almost every woe in the world and pile them at the feet of big data. So here are my rebuttals:

Big Data = Big Brother?

This hasn’t been a good couple of weeks for the field of data mining. The NSA scandal has  caused sales of Nineteen Eighty-Four to rise, unfortunately not quite at the speed that the use of the phrase “Big Brother” by journalists has risen.

In his article Leberecht oddly passes this over, and instead mentions the evil of passing on data to private companies in a sideways swipe at quantified self.

Perhaps he forgets to mention it because it appears that people can also see the positive side? There are real issues around privacy, anonymity and data security, but pretending that the age of big data is the cause is rather odd.

Big data is not social

Well firstly, hasn’t he heard of Social Network Analysis? But secondly he seems to be advancing the argument that the status of X (relationships) is threatened by allowing Y (data analysis).  Sound familiar?  Yes that’s the argument against gay marriage. Somehow if my gay friends get married, my marriage will be threatened.

Well for the record analysing data doesn’t mean that humanity will be diminished. Welcome to the world of science! Were we more human in the 17th century? Or the 13th? Because there was a hell of a lot less analysis then than in the 1950s…

Finally on this topic, what about the growing Data Philanthropy movement? Every week we see new initiatives where people want to apply big data to address real social issues in ways that couldn’t happen before.

Big data creates smaller worlds

Apparently big data filters our perception, and limits our openness to new ideas and cultures. Really? To go back to gay marriage – can we imagine this being a possibility 20 years ago? The ability to interact and identify unusual events and groups menas that there is far more diversity than there ever was. A goal of marketers is to open people’s eyes to new things (and to get them to buy them). Leberecht seems to think that the collaborative filtering that Amazon famously use would only ever return you to the same book.

Big data – and opening yourself up to ideas that aren’t part of you narrow ‘intuition’ will surely make your world bigger and more diverse…

Big data is smarter, not wiser

The article makes it clear that wisdom has a twofold definition in Leberecht’s world: it is based on intuition (guesswork) and it is slow. Oh, and it also rejects feedback. Well firstly, big data isn’t always fast. Believe me, Hadoop isn’t a solution suited to the rigours of rapid operationalisation. There are other things for that. But as a definition of wisdom this seems to be a disaster.

Not only should you take the risk of making the wrong decision (intuition is guesswork), but you should do it slowly, and without paying attention to any feedback you get.  Truly this is fail slow writ large.

Big data is too obvious

I think this is the heart of Leberecht’s argument. He didn’t think of big data.

His example (that the financial collapse was caused by measurement) is patently wrong. The problem with mortgage backed securities was exactly the opposite: people failed to measure the risk and relied on intuition that the boom was going to continue indefinitely.

Big data doesn’t give

And then finally we hit the sleeping policeman of CP Snow. You are either an artist or a scientist. A businessman or a data scientist. Creativity belongs to the former, sterile analysis to the latter.

I’ll let you guess what I think of that!