How Bad Data-Driven Decision-Making Led to the Mistake of “New Coke”

new coke

The Testing Threat

In the 1980s, Coca-Cola executives were shocked to learn that what Pepsi advertisements said was true — in a random taste test, people preferred Pepsi over Coke.  Coca-Cola executives responded with a massive retooling effort, resulting in a product dubbed “New Coke”.

“New Coke” turned out to be a major flop.  What we know in hindsight is that the way taste-tests are done is biased — in small amounts (sips) people prefer the sweeter drink, but in larger amounts (a 12 ounce can), people preferred the original Coca-Cola formula.

The Imperfect Metric

This is a phenomenon that happens all the time — An effort is made to quantify success.  The metric chosen is imperfect; yet people exert a lot of effort to maximize or minimize the metric, even if flaws in the metric are known.  I’ve talked to students who don’t understand the concepts they are studying, but simply memorize the “correct answers” because they know that is how they will be evaluated.  Teachers teach to the test; and students study to the test.

Music: Data Driven by Shazam

In a similar way, the author of an Atlantic article describes a smartphone app called Shazam.  A “Shazam” is equivalent to a google search for music; but the music industry treats search traffic for a song as if it were the same as a Facebook “like”.

So, what meaning does the Shazam metric really convey?  Quality? Novelty?  Attention?

The music industry has made “Shazam” the new “test”, and by “teaching to the test” the direction of the music industry has shifted.  The industry is now more data-driven, but the result is more repetitious music with predictable chord progressions — a sort of “comfort food” (6 min).

So like “New Coke”, does our our crude big-data analysis result in better music, or are we just making it simpler and “sweeter”?

Open Source Media

A lot of people think about software when they hear the words “open source,” but I’d like to extend the concept to “media”.  By that I mean books, tv, magazines, radio, etc.

Example

The basic idea is simple — suppose you’re reading a book about Jack Kennedy that makes an interesting claim and then cites its source with a footnote to an “NBC Interview with Jack Kennedy: Chet Huntley and David Brinkley in the Oval Office in the White House, Sept 9, 1963.”

One of my first questions would be: “Can I get a transcript of the interview?”  A second would be: “Can a get a recording of the whole interview?”  Without the first, I can’t verify what the president said.  Without the second, I can’t get the context.

Two related questions this raises are: “What are the ground rules for the interview;” and “How much editing was done to produce the final product?”

It’s interesting that in Brinkley’s interview, the president was given a number of “mulligans,” although he appears not to have seen the questions ahead of time.

One of the commenters noted:

The media and politicos are in cahoots, rehearsing the interview.

Ground Rules for Interviewing

So I’ve been thinking: “What are fair ground rules for an interview?”  Here’s a few ideas:

  1. The full recording, including out-takes, should be available for the historical record. [ref]How soon is that?[/ref]
  2. Should anything be left out of the transcript?  Inevitably I think the answer will have to be yes, unless you get rid of all “off-the-record” interviews.  I also think the appropriateness of off-the-record remarks varies according to the degree of power that the interviewee has.  The secrets of the powerful often warrant less protection than the secrets of the weak.
  3. It may take time to gain the trust of the interviewee; and in real-life, the interviewer only begins recording when trust has been established and the interviewee is ready.

The Complete Record

I’ve sometimes wondered, what would happen if journalists tried to put everything on the record.  They would record their telephone calls asking for the interview. They would share all their email correspondence.  They would begin recording as they approached the office or home of the interviewee and then just keep filming until after they left.  And they would publish the entire contents of this “record” with every interview they did.  This is now feasible on the web, whereas it was impractical in the television or print-only world.

Now of course most people wouldn’t care to watch the whole thing; but a few would; and they might post notable things for the inspection of a wider group.  Is this what we want?

Paris Review Style Interview

An alternate model is employed by the literary journal “The Paris Review.”  It’s editors like to select their favorite authors to interview; and they give the authors full license to edit their answers.[ref href=”http://www.nytimes.com/2010/10/23/books/23interview.html”]”Yet over the decades few major writers have turned down The Paris Review. Part of the appeal of these exchanges, for writers, is that they’re allowed to tweak the text later, adjusting and readjusting the masks they want to present to readers.” (New York Times) [/ref]

Naturally, the authors are used to choosing their words carefully; and this approach allows them to extend such care to the interview.  It allows the author to say exactly what they want, potentially resulting in more clarity, or alternatively less accountably.

Speaking about interviewing authors, David Fenza says:

A good literary interview is not faithful to the actual spoken event.  The transcript of the actual spoken interview should only serve as a draft of a dialogue that will, eventually, present the writer as completely and succinctly as possible.  A good literary interview is improvisational, but it’s also revisionary.  Writers are creatures who succeed through revision; they are most themselves when they revise; and this should carry over into the interview.[ref href=”https://books.google.com/books?id=ra2lH0scz9MC&lpg=PA69&ots=UHq3Q9xNk8&dq=paris%20review%20interview%20process%20favorite%20author%20edit%20transcript&pg=PA69#v=onepage&q=paris%20review%20interview%20process%20favorite%20author%20edit%20transcript&f=false”]The Art of the Author Interview: And Interviewing Creative People, by Sarah Anne Johnson. pp 69.[/ref]

When to allow a “Paris-Review” style interview depends on the type of interview desired. In any case, the ground rules should be disclosed.

If a President is given chances to “edit” their answers, there should be some indication of this when the interview is published.  But no matter how the interview is edited or revised, can the full historical record be preserved?

It is common to see something like “This is an edited and condensed version of the interview.” It would be interesting to see some sort of statistical disclosure about how much of the included text was changed; and how much was excluded.

Xanadu: Browser Demo Released!

At a Chapman University Event — Intertwingled:

Ted Nelson announced the release of Xanadu for the web browser:

Instructions

  1. Realize that it loads slowly.  Just wait.  It’s just a demo so the performance hasn’t been optimized.
  2. The user interface isn’t always intuitive (it uses keyboard shortcuts)

View Demo

 

Background:

  • The main content is visible in the middle and the related content is displayed in parallel on the side
  • The connections between the different sources are displayed with colored lines between the documents
  • To see the related source, click on it with your mouse.  To get back to the prior source, click on it.

 

About the document:

  • The document, written by Moe Justes, is a compilation of different sources.
  • I haven’t read all the content; but though Ted Nelson has said he himself is an atheist, Juste’s method of intertextually invoking other sources is actually quite biblical.
  • At this point, it probably isn’t easy to duplicate something like this with your own content, but I wonder how biblical criticism could benefit from this form of parallel writing.