Science for SEO

It refers to meaning in language (or code or anything else).  It used syntax and pragmatics as well as contextual information to provide the meaning of the text or even audio stream if you want to use that.  It's not just about finding similarities or context between 2 words but rather taking the entire text or query to establish meaning.  

It's a common framework allowing information to be shared and reused.  Information is stored in machine readable formats.  

"The Semantic Web is an extension of the current web in which information is given well-defined meaning, better enabling computers and people to work in cooperation." - Tim Berners-Lee

That's probably the best definition, seeing as he invented it.  It's  relevant to semantic search and uses many of the same techniques and also is based on the same idea.  It's not a new version of the web, but rather an extension.  There are a lot of conferences about this worldwide, such as "The semantic web technology conference" for example.  

Technologies used obviously include ontologies (which are like big storage boxes full of information on how words and concepts link to each other).  These are built in OWL mostly, Natural language processing tools for named entity extraction for example, Data interchage formats (like RDF/XML or turtle for example), schemas like the RDF Schema, XML to provide syntax for content structure and SPARQL which is a web query language for semantic resources.   

Google uses PageRank to identify relevancy, whereas semantic search engines use meaning to return highly relevant results.  Google returns keyword/keyphrase results, and the semantic solution returns information.  

The data has to be really structured in ontologies just like in the semantic web.  A semantic network is created which links all of the concepts and words together.  It used word sense disambiguation (WSD) in order to decipher what a word may related to.  WordNet, which you can download for free, is a machine readable dictionary that a lot of scientists have used for this task, although it's far from foolproof.  Here is a very comprehensive list of which semantic search engines use what kind of procedure, in pretty plain English.

Google does respond to natural language queries, such as "Where was Marilyn Monroe born?".  Hakia doesn't understand the query and tells me what Marilyn's real name was.  Powerset (only searches Wikipedia and Freebase) comes up with the goods, "LA".  Working in a "closed domain", Powerset has an easier job than Hakia who searches the whole web, just like Google.  Google however delivered where Hakia didn't this time round.

Then I tried in all 3 "Is chili bad for you?" - Hakia came up with books reviews for a book called "Bad chili", Google came up with a forum article with that exact question in it, and Powerset delivered an in depth article on the effects of chili on humans.  The following results for Powerset are all off though.  Hakia results continue with the book, but Google gives me loads of results all about the effect of chili on the body.

Have a go yourself and see what happens.  

This little test definitely shows that Google can come up with the goods, whereas the semantic engines struggle.  More work needed there.  I would be very surprised if Google were shunning semantic web technology and natural language queries.  I would leave that open for discussion actually.

The future?  Natural language queries, and natural language generation for a straight answer to a question, and a summary of all of the most relevant resources in one text, and the option to read the individual documents.  That's not an easy feat!

In short, Linkscape has a crawler (perhaps more than one doing different jobs), it uses a popularity measure and assigns a link popularity score to each page.  It uses MozTrust which is apparently like TrustRank to identify trustworthiness.  It provides you with metrics on how your site sits compared to others in the space.  It'll also give you "the quantity and distribution of the anchor text  that a given site or page has  received from its inlinks".

You can try it out here.  It looks good!

SEOmoz always have some really cool tools on offer, some free, some not, but all equally useful.  This announcement is quite exciting, I think it'll grow to be something really fun to play with, or to work with, depending on how geeky you are.

It works by retrieving information by matching concepts and meanings.  They have their own algorithm for ranking called "SemanticRank".  It can categorise results, it uses parallelism (Treat = cure), it makes suggestions and allows for user refinement.  

Their moto I believe should be "quality over popularity".  Google likes popularity, so this is a different take on things.   

QDEX is the system by which they analyse and store web pages.  It's different from the standard inverted index, because it allows for semantically rich data to be processed quickly.   It analyses the content of a web page, extracts all possible queries that could be related to it, then the queries allow the system to fine documents, paragraphs and such things for retrieval.  It's all done off-line.  This works well because it can decompose content.

SemanticRank is based on sentence analysis and and concept match between the query and the best sentence available in each paragraph.  They say they also use syntactic and morphologocal analysis.  There's no keyword matching of course because it;s all semantics and no boolean  matching.

Anyway, today they launched some new features.  They've added the "credible sites" tab, where you can look at results from authorities, such as edu, gov and such sites, and they're asking librarians to suggest sites and "information professionals" (I'm not sure who that covers).  The resources must be current, peer reviewed, non-commercial and authentic (or at least fulfill most of these requirements).

For now you can only use it for the topics of the environment, health and medicine.  The sites are by experts, although anyone can submit a resource.  

 There's also a few other tabs:  "news", "images" and "meet others".  The "meet others" tab is interesting because they're adding social networking to search.  There are different rooms where you can post a comment or upload something for other to see, on the topic of the room of course, and then you can discuss it.  You can rate the messages.  It seems to be used but I'm not sure how much people will get involved, seeing as they usually go to a search engine with a particular information need.  It might do though.    

Hakia also offer a personalisation service.  You get a myHakia account, and then you can get information and news from topics that you specify.  

The results aren't quite as good as Google's just yet as far as my very brief tests show.  It does have some interesting and possibly quite useful features as well though.  The interface is nice and clean and easy to use.  

Take a look and see for yourself, you might like it.

 "All parts should go together without forcing.  You must remember that the parts you are reassembling were disassembled by you.  Therefore, if you can't get them together again, there must be a reason.  By all means, do not use a hammer."

– IBM Maintenance Manual, 1925 

The algorithm decides which pattern and areas (windows) are non-similar rather than establishing a distance measure.  This way unnecessary information is not computed so it's faster.  They say it can detect "low quality patterns, rotated patterns or patterns that are partly occluded."

Goodshop was recently launched and it donates 37% of the sale of any goods purchased.

The results are decent too.  Give it a go, especially if you run a charity.

They say:

“Our Web 2.0 application model uses innovative cloud-computing techniques to create a highly effective distributed search engine that easily scales to meet volume demands without compromising performance. We’ve combined this backend with a social overlay to fine tune and share results and a user interface built with advanced RIA technologies to create a compelling, highly productive user experience. NeXplore Search is poised for growth as users seek more effective and enjoyable ways to find the information they need.”

So they've made it fast using cloud computing and use RIA's (Rich Internet Applications) to enhance interactivity and expressiveness.  It's quite different to the Google model, which is very centered on providing information and not necessarily creating a networking and rich environment like Nexplore.

I tried a couple of searches (to have a clear picture I'd have to do loads more to be fair).  I search for 2 of my favourite things:

Google -> wikipedia, myspace, a random blog, random blog, random blog

Nexplore -> random blog, myspace, wikipedia, and the same random blogs as Google

Google -> wikipedia, wikipedia, yoga school, BBC, Amazon

Nexplore -> ashtanga.com (A very authoritative site), a yoga school, wikipedia, a yoga school, and ashtanga.com again.

So clearly in my searches Google has preferred wikipedia results, which is cool, because I might well want to know the definition of "Ashtanga yoga" or know who "The Breeders" are.  Nexplore also provides that but starts with a random blog.  Really the results are pretty much the same for some searches, just in a different order.  Nexplore does better on "Ashtanga yoga" I think because it gives me a well known authoritative site rather than wikipedia first.  Here the results weren't the same as in Google.

Google gives me 4 video options in the results for "The Breeders".  Nexplore doesn't do that.  You have to search under "video", so no universal search in the results.

I searched for myself and the results contained less instances of the person of the same name who works for NASA, so that's fine.

Nexplore has a super busy interface, do not be duped by the very minimalistic landing search page.  Under every result there's a social media sharing option, a massive preview window pops up when you hover over the results at any time, there's a wiki search box which is constantly on the right of the page...you see what I mean...it's very busy.  You can view result in a line, which makes them a bit hard to navigate through because there's 25 results on a page.  You can also just view the site preview, which looks a bit like a music library.  

Nexplore does personalise results for you as you can bin them, preview them or save them.  Google is looking into this too with the thumbs up or down thing.  It also gives you a "popular" searches library which I like. 

I'm not too sure about ad relevance as I get one for puppies for sale for "The Breeders" search when the engine has already established that it's a rock band.  For "Ashtanga yoga" 2 ads are relevant but one if for a hotel, which has a yoga class.  But that hotel site isn't really about ashtanga yoga.  Maybe it was the best ad to display from the collection available.

So.  I think Nexplore has good results, as good as Google in my short experiment, and way better than Cuil.  I think it's too busy, there's too much going on, and I don't like the preview popup.  The other two option, gallery view and list just aren't clear enough in my view because they don't give enough info, which you get from the description.  I like the personalisation.  I like that it gave me authoritative sites at the top.  

The social media button to share the site are on every result.  This further crowds the interface.  It's a great idea though, just maybe it can be done a bit differently to relieve the interface of clutter.  I hate all this popping up stuff!  Also, I don't like how the pictures display on the image search.

The verdict:  I will use it for w while and see how it grows on me, after all every change takes a while to adjust to.  I don't want to write it off because I don't like the clutter so early on.  I might really like it all once I use it often.  The results are good.  

I'm really not ready to dump my classic, simple, Google interface, and the results are fine for my purposes.  I think that it's very hard for a new search engine to come along with new cool ideas, and this one has really tried hard.  I'll definitely persevere and see how I go.

Social media in this kind of engine is all done for the sites because of the button under every result.  It gives them more chance at visibility than in Google.

The CTO, Dion Hinchcliffe runs a blog called "Musings and ruminations on building great systems".  It's really wordy but quite interesting so read it if you have the time.  You can follow him on Twitter as well.

You can also read about Nexplore at "Beyond Search", where Stephen Arnold gives his take on it.

If we did all migrate to Nexplore, I think we'd still Google things in it anyway.