The New Gold Rush? Wall Street Wants your Data

Not that many years ago, some hedge funds would send people to literally stand in front of big-box retail stores and count the number of people coming in and out, and on that basis make predictions about the retail chains themselves and the economy in general.
 
Alternative data now offers an opportunity to do the same thing at an entirely different scale and level of sophistication.
.. The more fundamental funds will use the data as an input into human-driven investment decisions.  For example, they’ll try to predict the sales or churn of a specific company, with the overall gall of of outperforming sell side consensus.  Or they’ll try to predict macro economic trends, for example through the observation of satellite images.
.. At the other end of the spectrum, the quantitative funds will take your data set, combine it with other alternative data sets and feed them into very sophisticated models. The growing trend is to completely or partially automate trading strategies on the basis of those models, fed by alternative data.
.. There are a few key characteristics that make your data more or less interesting to hedge funds:
  • level of detail,
  • history,
  • breadth and
  • rarity.

Challenges and Opportunities Confront the Data-Driven Business

Most companies capture a small fraction of their data’s value

It’s often been said that truly transformative innovations are overhyped in the short term but under-hyped in the long term. Think of electricity and automobiles, the internet more recently and now big data.

When first developed in the late 19th century, electricity was mostly used to replace kerosene lamps and candles with light bulbs. It took several decades for electric appliances, the assembly line and mass production to emerge and help create whole new industries. Similarly, the full impact of automobiles was not felt until the mid-20th century with the rise of suburbs, the Interstate Highway system, and the motels, restaurants and gas stations that sprung up all around them.

 .. In 2000, only one-quarter of the world’s stored information was digital and thus subject to search and analysis. Since then, the amount of digital data has been doubling roughly every three years. By now only a small amount of all stored information isn’t digital, around 1% or so. This could not have possibly happened without the digital revolution
..

  • Micro-segmenting a population based on individuals’ characteristics as revealed by data and analytics;

Mockaroo: realistic data generator

When your test database is filled with realistic looking data, you’ll be more engaged as a tester. When you demonstrate new features to others, they’ll understand them faster. Real data is varied and will contain characters that may not play nice with your code, such as apostrophes, or unicode characters from other languages. Testing with realistic data will make your app more robust because you’ll catch errors that are likely to occur in production before release day.