These kinds of stories always fascinate me. In this case, it’s the latest results from efforts to determine whether gathering TV audience data via the set-top boxes provided by cable, satellite, and telecommunications companies might be a viable alternative to the established People Meter, audimeter, and paper diary methods employed by Nielsen.
I get into this ongoing issue in a fair bit of detail in Audience Evolution. What’s particularly interesting about this recent story though is the extent to which we’re now getting some details on the nature of the inevitable differences that arise when set-top box audience data are compared to the audience estimates produced by traditional audience measurement systems.
Needless to say, different methodological approaches generally produce very different portraits of the media audience, and the emergence of the set-top box method is proving no different in this regard. For example, in terms of household ratings for newscasts, results indicate set-top box ratings that are as much as 79% higher than the audience estimates produced by Nielsen’s methods (obviously, if you’re a TV news director, you know which measurement system you’d prefer).
You might ask, how can two presumably sophisticated and professionally designed and implemented audience measurement systems produce such vastly different portraits of the audience? The answer lies in the wealth of methodological differences between the Nielsen and Rentrak (the set-top box measurement firm) approaches, ranging from different sample sizes to different sampling universes (the Rentrak sample, for instance, excludes all cable and over-the-air-only households), to the different ways in which the technologies operate. For instance, the Rentrak system can’t tell if the TV is on or off. It only knows if the set-top box is on or off. This of course leaves ample opportunity for viewing numbers to be inflated (I, for one, leave my set-top box on virtually all the time. I guess that would mean that my household might be counted as watching TV 24/7).
But in any case, as this issue moves forward, the key point will be who the winners and losers appear to be (as reflected in the new ratings estimates) in any transition from People Meters and diaries to set-top boxes. As we saw in the cases of the Nielsen People Meter and the Arbitron Portable People Meter, those content providers whose offerings don’t fare as well under a new measurement regime will scratch and claw to keep that regime from becoming the currency, and will likely employ virtually any vaguely viable argument on their behalf. And of course those whose performance improves under the new system will fight equally as hard for its adoption.
In these disputes, it’s never really about accuracy, because there really aren’t any indisputably accurate figures against which the data from the competing systems can be compared. Instead, the services represent alternative versions of reality. The one that you’ll embrace is the one in which you appear to perform the best.
So, as set-top box initiatives continue to move forward, brace yourself for the same kind of heated stakeholder battles that characterized the introduction of the People Meter, the Local People Meter, and the Portable People Meter.