Newly Emerging Best Practices for Big Data

An interesting angle on low latency data is the desire to begin serious analysis on the data as it streams in, but possibly far before the data transfer process terminates. There is significant interest in streaming analysis systems which allow SQL-like queries to process the data as it flows into the system. In some use cases, when the results of a streaming query surpass a threshold, the analysis can be halted without running the job to the bitter end. An academic effort, known as continuous query language (CQL), has made impressive progress in defining the requirements for streaming data processing including clever semantics for dynamically moving time windows on the streaming data.

.. For example, a single Twitter tweet “Wow! That is awesome!” may not seem to contain anything worth dimensionalizing, but with some analysis we often can get:

  • customer (or citizen or patient),
  • location,
  • product (or service or contract or event),
  • marketplace condition,
  • provider,
  • weather,
  • cohort group (or demographic cluster),
  • session,
  • triggering prior event,
  • final outcome,
  • and the list goes on

One of the charms of big data is putting off declaring data structures at the time of loading into Hadoop or a data grid.

.. This best practice conflicts with traditional RDBMS methodologies, which puts a lot of
emphasis on modeling the data carefully before loading

.. Contrary to popular belief, there are not just 24 time zones around the world, but hundreds! The complexity comes from daylight savings time rules. For example, although the state of Indiana is entirely in the Eastern time zone, part of Indiana observes daylight savings time and part does not. You need a list of Indiana counties to know what time it is in Kokomo!

Alan Alda’s Interviewing Tips for Uncovering Business Requirement

Be curious, but not too smart. Skilled interviewers must be curious. Alda has a natural interest in science, but he warns of the “too-smart syndrome” where interviewers think they’re nearly as well versed in the subject as interviewees:

Perhaps you’ve observed too-smart interviewers. Their questions tend to be long-winded, often eliciting blank stares or responses like “What was the question again?” Interviewers who try to impress others are missing the point. Ask simple, straightforward questions and you’ll have a better chance of understanding complex concepts.

  • Prior to the interview, send out an announcement describing the high-level discussion topics and confirming the interview time and place. Don’t attach a detailed questionnaire to this meeting notice. You can’t achieve a conversational flow if you’re reviewing questionnaire results-presuming anyone bothers to complete the survey.
  • Interview questions prepared in advance are fallback devices, used only if uncomfortable lulls occur in conversations or to ensure key points are covered before ending sessions.
  • Most good conversations tend to wander, so remember your session goals and steer conversations back on track if you stray too far from core issues. Stay at a relatively high level in the interview’s early stages.
  • “There’s one skill that I really make use of in a big way, and that is listening. If you don’t listen deeply, the connection won’t take place…. [You have to be] willing to be changed by the person you’re listening to, where you’re not just waiting for a pause so you can say your thing, but you’re actually letting them have an effect on you if they can.”

Interviews: Dealing with Comatose Users

These business users respond to your classic, open-ended questions with monosyllabic, one-word responses.

.. It’s sometimes effective to ask these people questions from a more negative perspective. For example, rather than trying to get them to envision life outside the box, these users sometimes find it easier to tell you what’s wrong inside the box.