Foreign Affairs magazine is not usually a tome of project management knowledge, but an essay on 'big data' by Cukier and Mayer-Schoenberger caught my eye.
They write:
Using great volumes of information .... requires three profound changes ..... The first is to collect and use a lot of data rather than settle for small amounts or samples, as statisticians have done for well over a century. The second is to shed our preference for highly curated and pristine data and instead accept messiness:..... Third, in many instances, we will need to give up our quest to discover the cause of things, in return for accepting correlations.
In internet years, I think 'big data' is somewhere around 1994. So, there's a bit to go yet. But, forgive me -- did they say: "forget economically viable samples, accept messiness over accuracy, and be happy not understanding causation from correlation"?
One wonders if this pair have ever had to define "DONE" to the project sponsor?
I've done some pseudo-big data projects: data warehouse for a multi-billion $(revenue) company and multiple ERPs with zillions of millions of records. I never got away with 'messiness'. Somehow, the CFO, CEO, and all the other C's and VPs wanted to have assurance that business data was accurate... how quaint! When I was a business unit vice-president, I wanted accurate reports also. Who knew this wasn't needed?
However, I get it. Business is changing so project metrics may have to change as well. There may be a challenge fitting 'messiness' into the "technical performance measures" (TPM) or taking earned value on it.
And, certainly correlations instead of causality will start many arguments about "who shot John" and whether an improvement or a disaster can be laid at the foot of the project.
And, the economics of ever larger samples -- perhaps the whole population -- is going to come out of somebody's budget: there's no free lunch (data), even if user generated (See: Facebook)
KENNETH CUKIER is Data Editor of The Economist. VIKTOR MAYER-SCHOENBERGER is Professor of Internet Governance and Regulation at the Oxford Internet Institute. They are the authors of Big Data: A Revolution That Will Transform How We Live, Work, and Think
Check out these books I've written in the library at Square Peg Consulting
