Nice answer. Do you know the question?

Okay, I’m likely going for triple-word-score here, but I’d like to discuss what I call “Qualitative Feature Polarization” in terms of how to read data.  I’ve talked about this before in the specific case of how to rate a sushi restaurant, but I think the principles apply to any situation where people are asked to make a judgement/evaluation.

The problems with most evaluations are as follows:

• ranking things in a straight good-to-bad list throws out too much information
• all data points are contextual, but we rarely understand the context
• the real question– and therefore the real answers– are often not present

That last point is perhaps the most important: many times we are asked to make an evaluation from data that is based on assumptions in the goals or undefined context in the business model.  For example: If Zappos were to match its prices against WalMart, it would “lose”, but in terms of service, it would “win”.  Conversely, wait times on the customer service line to Zappos are “much better” than when I call WalMart.  In an evaluation of customer service level, which one is correct?  It depends on the business model, and on the context.  In Vietnam, the US Army killed bad guys the enemy at a rate far more than our own casualty rate.  Wins used to be defined by simple body count, but we saw how that turned out.

The effficiency expert Pete Abila points out the Toyota quality method of “5 Whys” to try and find the real question (in hopes seting up an alanysis to find the real answer).  By simply asking the question “why?” at least 5 times, we can often get to the heart of the real problem, and strip away any assumptions or differences in the models we have put around the data results.

So, how to run an analysis that can provide sufficient context while also attempting to find the real issue? Whenever possible, I try to run all data in a method I call “Qualitative Feature Polarization”.  This method runs on some basic rules (which echo the problem points above):

• Smash data sets together to see if any patterns arise (but remember your college statistics prof warnings about causality and correlation).  This is the serendipity part– the chance for an “a ha!” moment that might lead to a further inquiry or data set
• Whenever possible, frame your data in left<–>right, blue<–>red, service<–>cost.  Applying such a framework to the Zappos vs. Walmart comparison would then lead you closer to the real issue: the business models are fundamentally different.
• Whenever possible, highlight the specific areas that the results and evaluation have NOT answered yet.  In other words, show that the evaluation has uncovered some results, but that the reader/listener (your boss) should specifically NOT jump to some conclusions  (unless he has a mat) based on the data.  This isn’t because the data is incomplete, it’s because the data was investigated to look at a specific issue.  Thou shalt not extrapolate your summary.
• Structure your analysis that leads you to a contextually-rich, polarized set of numbers.  These numbers should then guide the next choice of left or right, not good or bad.

Receptiveness to this analysis is a mixed bag.  If your audience is a fairly flat organization and the geeks are on equal footing, then qualitative feature polarization works well.  If the organization is fairly vertical, they likely just want “the answer.”  A good way to guage for this is the degree to which the org uses PowerPoint as a communications mainstay, and the level of complexity in the metaphors you spin out.