« i can haz campaign | Main | privacy, pickles, phorm »


Yep. There's no point asking people what they think because they'll tell us any old rubbish. But find out non obvious relationships and it seems they are exist.

I've often wondered if there was anyone in the city who could data mine all the fluctuations in the markets and link it with something that nobody had thought of before.

With any sufficiently large pool of data you will always "see" patterns that that seem to correlate with aparently unrelated phenomena.
As an example,if you charted the position Barnet FC has finished at over the last 20 years you could probably find a stock somewhere in the world who's price exactly correlates with their league postion. (not including stock in Barnet FC!).A fool be he who then buys or sells this stock based on how well Barnet FC's striker is performing!
The temptation is to think this correlation is meaningful in any real way when it's probably not.
The main problem with data minning is the creation of these false correlations. We think we've discovered meaningful insight when in fac tall we are seeing is 2 random patterns that seem to be linked.
Nicholas Taleb makes this point much more elloquently that I ever will be in his excellent book "Fooled by Randomness".

The comments to this entry are closed.