The Testing Threat
In the 1980s, Coca-Cola executives were shocked to learn that what Pepsi advertisements said was true — in a random taste test, people preferred Pepsi over Coke. Coca-Cola executives responded with a massive retooling effort, resulting in a product dubbed “New Coke”.
“New Coke” turned out to be a major flop. What we know in hindsight is that the way taste-tests are done is biased — in small amounts (sips) people prefer the sweeter drink, but in larger amounts (a 12 ounce can), people preferred the original Coca-Cola formula.
The Imperfect Metric
This is a phenomenon that happens all the time — An effort is made to quantify success. The metric chosen is imperfect; yet people exert a lot of effort to maximize or minimize the metric, even if flaws in the metric are known. I’ve talked to students who don’t understand the concepts they are studying, but simply memorize the “correct answers” because they know that is how they will be evaluated. Teachers teach to the test; and students study to the test.
Music: Data Driven by Shazam
In a similar way, the author of an Atlantic article describes a smartphone app called Shazam. A “Shazam” is equivalent to a google search for music; but the music industry treats search traffic for a song as if it were the same as a Facebook “like”.
So, what meaning does the Shazam metric really convey? Quality? Novelty? Attention?
The music industry has made “Shazam” the new “test”, and by “teaching to the test” the direction of the music industry has shifted. The industry is now more data-driven, but the result is more repetitious music with predictable chord progressions — a sort of “comfort food” (6 min).
So like “New Coke”, does our our crude big-data analysis result in better music, or are we just making it simpler and “sweeter”?