Mark Huberty

Commoditizable data science is a fallacy. Is avoiding the commodity enough? No. Even with a cadre of data scientists and their esoteric, self-critical training, organizations still face a more fundamental risk at the hands of big data. Unless we see significant organization change, big data threatens to make organizational decision-making worse, not better.

We should know by now–a good 40 years into the IT revolution–that technology is nothing without organizations adapted to use it effectively. I remember hearing, way back in 1996, a major recording industry firm talking about all the new services they could offer while selling music over this new “Internet” thingy. Seven years later, they’d done nothing, and a combination of Napster and Apple ate their lunch. They failed to adapt their internal organization–how they conceived of themselves, what markets they served, how they added value–in light of the internet. These sorts of problems help explain why consulting firms make small (and not-so-small) fortunes delivering “change management” services alongside enterprise software. Otherwise that shiny and expensive Peoplesoft installation tends to become merely a glorified Excel spreadsheet. In another life, I was on the inside of a few of those jobs. Cultural change is not easy.

Why this tangent? Because the people thinking “I know, tools, not people!” aren’t merely seeking ways around the expense of building a data science organization. Commoditizing data science also helps them ignore the organizational change that big data and data science implies. They don’t want some recently (or perhaps not-so-recently) minted PhD standing in their office saying “actually, all this data we collected is useless” or “no, I won’t give you the answers you want, because I don’t think they’re defensible”. They haven’t had the heavily ingrained biases systematically drummed out of them by sadistic dissertation advisers. Indeed, some of them may have succeeded because of those biases: they just stuck to their guns no matter what, and eventually made out. The world takes all kinds; but some of those kinds are more useful in building a data-driven organization than others.

For such organizations, big data might as well be a cudgel. “Big data” is a big deal, they want in on it, and they want in on it so that their organizations (or, perhaps more importantly, their fiefdom within the organization) can hopefully function better doing what they already want it to do. They want tools and people with which to confirm their own way of thinking about the world. They want tools to help them win bureaucratic battles. They want answers that they can use to justify what they were going to do anyway. A good data science organization won’t let them do this. Mere tools will.

Am I being unfair? Perhaps. But there’s a second-order version of this problem, one more lazy than pernicious. The people with this problem don’t want a cudgel. They are perfectly happy having someone with those “big data” tools in their organization. But they still haven’t through about the organizational problem–about the sociology of the tools, rather than the tools themselves. So they’re still stuck when that person (now not a “data scientist”) tells them they are in trouble. That recording industry firm didn’t have its head in the sand–it knew what the internet was! It was going to sell music on it! But the firm hadn’t internalized just how radical a transformation that meant, and couldn’t quite embrace it when it came.

These problems threaten to be particularly bad because “big data” will disrupt industries that the IT revolution has to date passed by. All sorts of things formerly off-limits–medical diagnosis, language translation, personal transport–are now squarely in the crosshairs of big data and algorithms.¹ Hence the organizations where big data presents the biggest opportunities are also those with the least experience in managing the consequences of radical technological change.