Science for SEO

This is because it takes ages for a lot of methods to be evaluated properly so that they can safely be used in public systems like search engines or social networks for example.  Some systems aren't designed to use some methods, and only when they have gone through many iterations, they suddenly see the need to incorporate a certain method or even a few.

Stemming for example is quite old, it goes back to 1966 when the Lovins stemmer was made.  Google I believe (but not totally sure of the exact date) implemented stemming to queries in 2003.  That's 37 years!  I think they were already using it in the internal system though, it's a pretty standard method in IR after all.  I wrote a stemmer in 2005 and it only started being used in 2007, not a lot of people saw any use for a stemmer that stemmed to exact words, but now it's pretty standard too.  That took 2 years.

PageRank came about in 1995 and was implemented when Google was publicly released in 1998, that's 3 years.  

I work in conversational systems and it has taken a while for the science community and also the industry to see why they could be useful.  Now there's a lot of research in this area, and the first chatbot was invented in 1966 (ELIZA).  It's not until recently that companies have started using chatbots on their websites (Ikea for example) and suddenly the potential for such systems in IR is being realised.  Long wait!  We don't even have all the technology needed yet to make something really good.

I think it's really important for the SEO community to keep track of papers released by IR researchers and also NLP/AI researchers when the work is related to search engines particularly.  It's useful to learn about the methods being developed and then it gives some insight into how they might be implemented (although this could take some time!).  You can use Citeseer to find them, or DBLP, and checking the references too can be useful.  Those are where my massive reading list comes from!

Of course some methods do get implemented quite quickly and I think that this happens when they are specifically built for a system in build progress.  The big search engines have people working solely on this and also companies like IBM for example.  What I mean is that you shouldn't discount methods that have been published a few years ago.  A lot of social media stuff was published quite some years ago too.

Happy reading :)