eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
While big data is getting all the headlines, small data is the next big thing. Big data and the value associated with combining and culling vast structured and unstructured data sets for business insights is a worthwhile but, as the name implies, big undertaking.
It may be too big for all but the largest enterprises that have the time, money and expertise to build a big data platform.
Small data is something else again. Think of all the digital tidbits consumers leave in their paths as they go through the day. Credit card payments, location fixes, newsletter signups, Facebook likes, tweets and Web searches. As Deborah Estrin stated at TEDMED 2013, “Small data are derived from our individual digital traces. We generate these data because most of us mediate or at least accompany our lives with mobile technologies. As a result, we all leave a ‘trail of breadcrumbs’ behind us with our digital service providers, which together create our digital traces.”
Marketing and sales executives are developing methods to use small data as an end run around their company’s big data initiatives, which, like those massive ERP systems of yore, can often be long on promise, riddled with cost overruns and always just beyond the horizon in delivering value.
Former McKinsey consultant Allen Bonde blogged about the use of small data in marketing recently on his appropriately named Small Data Group site when he highlighted “the growing shift from big budget campaigns, long-form video, and big data, to word of mouth viral campaigns, short-form content like vine and snapchat, and small data.”
In a telephone interview, Bonde said big data is about machines and small data is about people. He further contended that big data often feels like a “concoction” by system vendors to sell a new platform to enterprise executives. He is currently in the midst of completing a study on the small data phenomenon, but early results show those marketers adopting small data are able to gain control and immediate feedback from marketing and sales campaigns that big data systems—still largely residing in the IT departments—are unable to duplicate.
Examples include the recently introduced Hubspot Signals product, which gives sales personnel immediate feedback to client activities such as opening newsletters, email queries or Facebook likes. In another small data example, Bonde pointed to the travel site Kayak, which can offer users an immediate estimate if the price of the trip they are considering will increase or decrease based on the booking experience of other travelers.
Rufus Pollock, the founder of the London-based Open Knowledge Foundation, is one of the more articulate writers regarding small data. Big data to Pollock has all the hallmarks of the era of big computing, centralization and big budget dollars sunk into massive projects. “But the discussions around big data miss a much bigger and more important picture: the real opportunity is not big data, but small data. Not centralized ‘big iron,’ but decentralized data wrangling. Not ‘one ring to rule them all’ but ‘small pieces loosely joined,'” Pollock recently blogged.
Perhaps the greatest value of small data will not be for marketers or businesses trying to gain insights to existing or potential customers, but for customers themselves. Big data systems are currently structured as one-way pipelines. A company collects structured and unstructured data about its target segments and starts working away at converting potential customers into product buyers and advocates. Whether it is in business or politics, the consumer never gets to see his profile or understand why he is getting certain offers.
But, returning to that TEDMED presentation, Estrin advocates reversing the data profiling process. “But none of these services currently think about the value of providing these personal traces back to the person who generated them,” Estrin said. “And consequently, they do not yet have a ready-made vehicle to repackage their data about me in a useful format for me and provide it to me.” She goes on to conclude, “Let’s get our search engines, social networks and mobile carriers to start packaging our small data for us.”
The possibility of individuals owning their small data would mark a startling change from the world of big data profiles and marketing. I’m not sure the business world is ready, but I’m guessing we’d rather own our information and decide which bits to share than have big data systems that can scoop up those bits without our knowledge.
Eric Lundquist is a technology analyst at Ziff Brothers Investments, a private investment firm. Lundquist, who was editor-in-chief at eWEEK (previously PC WEEK) from 1996-2008, authored this article for eWEEK to share his thoughts on technology, products and services. No investment advice is offered in this article. All duties are disclaimed. Lundquist works separately for a private investment firm, which may at any time invest in companies whose products are discussed in this article and no disclosure of securities transactions will be made.