How Bad Data-Driven Decision-Making Led to the Mistake of “New Coke”

new_coke

The Testing Threat

In the 1980s, Coca-Cola executives were shocked to learn that what Pepsi advertisements said was true — in a random taste test, people preferred Pepsi over Coke.  Coca-Cola executives responded with a massive retooling effort, resulting in a product dubbed “New Coke”.

“New Coke” turned out to be a major flop.  What we know in hindsight is that the way taste-tests are done is biased — in small amounts (sips) people prefer the sweeter drink, but in larger amounts (a 12 ounce can), people preferred the original Coca-Cola formula.

The Imperfect Metric

This is a phenomenon that happens all the time — An effort is made to quantify success.  The metric chosen is imperfect; yet people exert a lot of effort to maximize or minimize the metric, even if flaws in the metric are known.  I’ve talked to students who don’t understand the concepts they are studying, but simply memorize the “correct answers” because they know that is how they will be evaluated.  Teachers teach to the test; and students study to the test.

Music: Data Driven by Shazam

In a similar way, the author of an Atlantic article describes a smartphone app called Shazam.  A “Shazam” is equivalent to a google search for music; but the music industry treats search traffic for a song as if it were the same as a Facebook “like”.

So, what meaning does the Shazam metric really convey?  Quality? Novelty?  Attention?

The music industry has made “Shazam” the new “test”, and by “teaching to the test” the direction of the music industry has shifted.  The industry is now more data-driven, but the result is more repetitious music with predictable chord progressions — a sort of “comfort food” (6 min).

So like “New Coke”, does our our crude big-data analysis result in better music, or are we just making it simpler and “sweeter”?

Open Source Media

A lot of people think about software when they hear the words “open source,” but I’d like to extend the concept to “media”.  By that I mean books, tv, magazines, radio, etc.

Example

The basic idea is simple — suppose you’re reading a book about Jack Kennedy that makes an interesting claim and then cites its source with a footnote to an “NBC Interview with Jack Kennedy: Chet Huntley and David Brinkley in the Oval Office in the White House, Sept 9, 1963.”

One of my first questions would be: “Can I get a transcript of the interview?”  A second would be: “Can a get a recording of the whole interview?”  Without the first, I can’t verify what the president said.  Without the second, I can’t contextualize what was said.

Two related questions this raises are: “What are the ground rules for the interview;” and “How much editing was done to produce the final product?”

It’s interesting that in Brinkley’s interview, the president was given a number of “mulligans,” although he appears not to have seen the questions ahead of time.

One of the commenters noted:

The media and politicos are in cahoots, rehearsing the interview.

Ground Rules for Interviewing

So I’ve been thinking: “What are fair ground rules for an interview?”  Here’s a few ideas:

  1. The full recording, including out-takes, should be available for the historical record.  (How soon is that?)
  2. Should anything be left out of the transcript?  Inevitably I think the answer will have to be yes, unless you get rid of all “off-the-record” interviews.  I also think the appropriateness of off-the-record remarks varies according to the degree of power that the interviewee has.  The secrets of the powerful often warrant less protection than the secrets of the weak.
  3. It may take time to gain the trust of the interviewee; and in real-life, the interviewer only begins recording when trust has been established and the interviewee is ready.

The Complete Record

I’ve sometimes wondered, what would happen if journalists tried to put everything on the record.  They would record their telephone calls asking for the interview. They would share all their email correspondence.  They would begin recording as they approached the office or home of the interviewee and then just keep filming until after they left.  And they would publish the entire contents of this “record” with every interview they did.  This is now feasible on the web, whereas it was impractical in the television or print-only world.

Now of course most people wouldn’t care to watch the whole thing; but a few would; and they might post notable things for the inspection of a wider group.  Is this what we want?

Paris Review Style Interview

An alternate model is employed by the literary journal “The Paris Review.”  It’s editors like to select their favorite authors to interview; and they give the authors full license to edit their answers.1

Naturally, the authors are used to choosing their words carefully; and this approach allows them to extend such care to the interview.  It allows the author to say exactly what they want, potentially resulting in more clarity, or alternatively less accountably.

Speaking about interviewing authors, David Fenza says:

A good literary interview is not faithful to the actual spoken event.  The transcript of the actual spoken interview should only serve as a draft of a dialogue that will, eventually, present the writer as completely and succinctly as possible.  A good literary interview is improvisational, but it’s also revisionary.  Writers are creatures who succeed through revision; they are most themselves when they revise; and this should carry over into the interview.2

When to allow a “Paris-Review” style interview depends on the type of interview desired. In any case, the ground rules should be disclosed.

If a President is given chances to “edit” their answers, there should be some indication of this when the interview is published.  But no matter how the interview is edited or revised, can the full historical record be preserved?

It is common to see something like “This is an edited and condensed version of the interview.” It would be interesting to see some sort of statistical disclosure about how much of the included text was changed; and how much was excluded.

Death Therapy

erasing death

Often when I hear of someone given multiple life sentences as punishment for a crime, I ask my self what the difference is between one life sentence and five life sentences.  I understand there is symbolism in the additional sentences; and in some cases people with a “life sentence” may be released early.3 In any case, I ask myself is whether there would actually be a way for multiple life sentences to be carried out.  Perhaps the sentence could mandate killing the criminal and then resuscitating them again, only to kill them again and then resuscitate them.  The process could be repeated enough times that the criminal could serve 5 life sentences in the course of a month.  (No, I’m not a lawyer.)

A new book by a doctor who specializes in resuscitation suggests that there is a common experience of dying that is consistent across cultures.  To be considered “dead”, one’s heart has to stop beating.  This stops brain activity, but it does not mean the the brain cells have died.  In fact, it is possible for a body to be chilled, and the person to be resuscitated several hours later.

Upon regaining consciousness, many patients have no memory; but others report seeing a bright light and feeling a very loving presence.  They recall having their life reviewed with them and feeling pain as they recall times when they caused others pain.  Some patients report being inspired to try to do better with their new lives.

So perhaps instead of giving our prisoners a lethal injection, we could give them “death thearapy”.

I can imagine that were this resuscitation perfected, so that the risk that patients stay dead is reduced, many wealthy people would pay for such an experience.

The Temptation of Jesus, Mercedes edition


Superbowl Ad, 2012: Mercedes-Benz 4 , 5

Matthew 4:1-11

Then Jesus was led by the Spirit into the Desert to be tempted by the devil.  After fasting forty days and forty nights, he was hungry.  The tempter came to him and said, “If you are the Son of God, tell the stones to become bread.”

Jesus answered, “It is written: ‘Man does not live on bread alone, but on every word that comes from the mouth of God.” 6

Then the devil took him to the holy city and had him stand on the highest point of the temple.  “If you are the Son of God, he said. “throw yourself down.  For it is written:

“He will comand his angles concerning you, and they will lift you up in their hands, so that you will not strike your foot against a stone.” 7

Jesus answered him, “It is also written: ‘Do not put the Lord your God to the test.'” 8

Then the devil showed him the new Mercedes CLA, and all the fame and glamour that it would bring, saying, “Sign your soul over to me and all this is yours.”

As Jesus was carefully considering the devil’s offer, he saw the CLA’s price of only $29,900 and said, “Thanks, I think I got this.”