Big Data? No. Big Signal!

One of the best ways to understand VRM (Vendor Relationship Management) is to look at it from a more familiar perspective.  When it comes to consumer data, one of the most familiar perspectives is that of Big Data so naturally many questions about VRM are couched in Big Data terms:

  • How big is VRM data anyway?
  • How much data is (or will be) in the personal cloud?
  • Who crunches VRM data to come up with something useful?

The answers to these questions lead to one inescapable conclusion: VRM isn’t a difference in scale.  It is a difference in kind.  This isn’t Big Data.  It’s Big Signal.

When you need to resolve faint signals, collect a lot of data.

When you need to resolve faint signals, collect a lot of data.

To illustrate, let’s imagine I want to figure out my gas mileage.  I top off my tank, drive a mile down the road, drive back and top the tank off again, then divide 2 miles by some portion of a gallon.  The biggest margin of error here, inferring the amount of gas consumed based on the amount replaced, makes any result completely useless.  Other margins of error include the odometer which only measures in tenths, the amount of idle time waiting to get out of the gas station and onto the road, etc.  Even the few teaspoons of gas in the hose that I don’t get charged for throws off the measurement with a sample size of only 2 miles. I am limited by the resolution with which I can measure the important data.

Knowing this, what most people actually do is generate much larger sample sizes, perhaps exhausting an entire tank of gas before calculating the mileage.  If we want even higher accuracy, we top the tank, mark down a starting mileage, track purchases over several fill-ups, top the tank one final time and then calculate the number.

To get good numbers, we need data big enough to squelch the noise in the signal.  We need a sample that is a few orders of magnitude greater than the limit of resolution that we can measure.  That’s the function of Big Data.  It lets us tease out signal that is inaccessible in less-than-big data.

Or we could buy a modern car that actually measures gas consumption directly, calculates a number in real time and posts it on the dashboard.  In this case, increasing the signal-to-noise ratio gives much better information, and in real time.

That’s Big Signal.


When you have a source of really good data, you don’t need as much of it.

Big Data is about getting a sample size sufficiently large to tease useful signal out of the noise of the various margins of error.  It’s about getting past the limits of resolution of the tools we are using to measure some phenomenon.

By comparison, VRM is about Big Signal.  The VRM owner/operator gets the ability to slice and dice personal data in a way that up to now was only available to enterprises.  Not only don’t I need data from millions of cars to calculate my gas mileage, it would actually get in my way to have it.  For the owner/operator, big signal is better access to their own data, however meager that may be in comparison to Enterprise databases.

For merchants and vendors, Big Signal is about vastly improving the signal to noise ratio to provide better data and in near-real time.

For example, you don’t need to analyze my music purchases to realize I heavily favor non-DRM formats.  I’m happy to tell you because it’s in my interest to do so if it means more DRM-free selection.  And if you provide something of value to me, such as the ability to export my data, an ass-kicking UI, or some great analysis tools, I’m happy to give you data on the things I didn’t purchase from you – signal that up to now has either been unavailable or scarce and relatively expensive.  Then there’s also the ability to correlate my data across many silos.  Our current Big Data silos can’t talk to one another well.  Where there have been attempts to do so, privacy issues arise quickly.  But I can correlate my own data across many different markets, verticals, vendors, niches and more, and do so without privacy issues.  As the vendor who provides to me an app that does something useful with the data, I am willing to let it collect some level of anonymized summary data.  I will insist on independent verification and/or completely open source code so that I have some confidence as to the selection of data being as advertised, but meet my terms and you can have access to new sources of clear data, correlated across many verticals.  And you can have it in near-real time.

So to me, VRM is equivalent to giving up calculating your gas mileage using log books and receipts and instead using tools that measure the desired number directly, in real time, using much better quality source data.  In data terms it is about providing vastly stronger signal, greatly reduced noise, or both, thus enabling useful outcomes from sample sizes of as little as one person.  Thanks to Personal Clouds, VRM breaks the privacy barrier to allow correlation across data categories that have traditionally been isolated silos, and this will usher in a new era of smart commerce on the consumer side of the economy.  Vendors and merchants who respect their customers and have earned our trust can participate in that new data economy.  Not by surreptitiously gathering up all our data and correlating it, but rather by supplying tools that run in personal clouds and that provide compelling functionality, an opt-in ability to feed some of that data back to you, and transparency about the whole arrangement.  For that you get much better quality of data in near real time.  For that you get Big Signal.

Leave a Reply