Science for SEO

There are some issues with these systems which cause degradation in user feedback, here are a few:

There is a lot of research going on to resolve these issues, take a look at these to start with:

Complexity and Social Networks Blog of the Institute for Quantitative Social Science and the Program on Networked Governance, Harvard University 

"A Few Bad Votes Too Many?  Towards Robust Ranking in Social Media" (Jiang Bian, Yandong Liu, Eugene Agichtein, Hongyuan Zha)

"Dynamics of Collaborative Document Rating Systems" (Kristina Lerman)

I came across a very cool paper from SIGKDD 2008 called "Blogosphere: Research Issues, Tools, and Applications" by Nitin Agarwal and Huan Liu from the University of Arizona.  It's an easy but long read, for the geek, but can also be quite happily understood by the layman.  I've pulled out some things that I thought were interesting and given you a short taster here, but I urge you to read the paper, it's brilliant.

We can't map the blogosphere in the same way because the number of links is sparse, and blog posts are dynamic and short-lived quite often.  Also the comment structure which provides for interaction does not exist in the webgraph model.  The webgraph assumes that sites build links over time, this isn't so in the blogosphere.  We cannot use a static graph like the webgraph.

One way to model the blogosphere is to gather data concerning link density,  how often people create blog posts, burstiness and popularity, and how these blog posts are linked.  also it's possible to use the blogrolls to find similar blogs.  This is what Lescovek et al. did, they used a cascade model usually used in epidemiology:

"This way any randomly picked blog can infect its uninfected immediate neighbors probabilistically, which repeats the same process until no node remains uninfected. In the end, this gives a blog network."

Brooks and Montanez used tf-idf to find the top 3 words in every post and then computed blog similarity based on that, which means that they could cluster them.

The problem is that these methods are keyword based clustering and therefore have high-dimensionality and sparsity issues.  You could reduce this by using LSI but the results still aren't so good.

Many companies have already seen the usefulness of blogs for sentiment analysis, trend tracking and reputation management.   Some systems use manually tagged sentences with  negative/positive references, then using a naive-bayed classifier until everything has been classified.  

Another way of finding the edges on the graph is by taking the topic similarity between 2 blogs.  This is a good idea, but using this method is still under research and very difficult.  

iRank is a "blog epidemic analyzer", and  predicts if 2 blogs should be linked (BlogPulse uses this).  They look for "infection" (how the information is propagated), so their aim is to find the blog responsible for the epidemic.  These are the authority blogger, the influential ones in the blogosphere.  It's good news when you find these bloggers because you can use them for word-of-mouth marketing as it were.  They provide valuable information that companies may be interested in, they may employ the blogger for example because s/he gives brilliant information to people about their products.

Another method to infer this has been to predict the odds of a page being copied or read, and also look at topic stickiness.  The most influential node is chosen with each iteration.  It apparently outperforms both PageRank and hits for this task.

Splogs (spam blogs) are the equivalent of link spam in search engines.  On the web algorithms include variables such as keyword frequency, tokenized url, length of words, anchor text and more.  PageRank computed a score which it uses to identify splogs.  This doesn't work on blogs unsurprisingly because they are too dynamic for spam filters to be effective.  This issue hasn't been resolved as yet, although there is research in this area, and things are improving.

Link analysis is also used to find patterns.  The text around the links is used, and based on those links hubs and authorities are found.  You could use comments as links between the blogs.  An influence score could be determined by taking into consideration inbound links, comments, length of posts, and links out.  

This is a fun and really interesting are of research, keep an eye on new things emerging from this research community.

This discussion between Yahoo social media gurus, industry and academics took place at www2008 in Beijing.

We all use social media, well most of use, certainly internet professionals.  The younger generation are communicating very freely via this medium.  It is not a fad, it is trend that will continue to grow strong in the years to come as networks continue to grow and evolve and their users become more expert in their use.  

There is set to be better understanding of user behaviour, data mining, new applications and domains.  There will also be better search capabilities within these networks.  The authors foresee a complete change in the workplace, which we know has already started to take place.

"Current systems utilise similitude as selector of new experience. ‘If you liked that then you’ll like this’. However the more profound and hence lasting experiences are the unexpected ones that are at once accessible and confrontational. It is easy to be either, but being both is a demanding challenge. So far we have little capability in marshalling such experience for users but in 2020 this will be different."

He says we need to root technological developments in the understanding that information interest is a sensory experience, filtered by emotional and cultural memories.  He says that we can gage this by navigation, speed, focus and other factors. He concludes:

"Social online interaction will be mobile and immersive interaction"

He says that the techo-centris view of the web is not in line with the social world.  

"The future of online social interactions requires a conversational redux. Content semantics alone is not sufficient. How we consume media (photos and videos) will become conversation centric. Conversational semantics, found in the conversations that ensue around media, is as important as traditional content-based semantics".

He says that we have to look at how people are sharing content within their communities and understand the supporting online social context.  He believes that conversational semantics will be a central part of the experience and a primary area of research.  

She says that communications will be fully integrated and unified with social software and contextual communication data across media will be shared and analysed, thus driving a new experience.  

"Imagine a search on a keyword that returns a list of items ranked by communicative or contextual relevance as opposed to content and large scale popularity. The ranking could consider if the information or the interaction sought is best from a certain source (a person’s whose opinion is respected by the searcher, or a person with whom there is a history of successful interactions) – through a particular medium (that is more accessible, comprehensible to that searcher), in a particular context (from a particular forum or news site). 

Such a search could return: people available to chat on that subject now, a list of blogs written by people whom you have valued before on that subject, or product ratings from people with like interests and backgrounds, it could set up a forum from a group of people on-line. 

Such a search would not return a list of content, but rather content vehicles, the people, devices, media, modalities that are most valuable to you and at the same time could establish communications directly. These rankings could be accessed directly by users, but more importantly would drive the processes that automate and manage communications."

The full discussion is available in the ACM digital library, well worth the subscription.

A cool paper caught my attention today:"A few bad votes too many?: towards robust ranking in social media" - it's written by researchers from Emroy University and the Georgia institute of Technology (ACM SIGIR '08).

People vote all the time in social networks, be it Digg, Sphinn, Linkedin, and many others that we visit frequently.  These votes are used to rank, filter and retrieve high quality content.  There is a lot of "noise" though due to voting for your friends and gaming the system, this degrades the quality and reliability of that data.  

Their solution to this problem is to build a machine learning based ranking framework for social media.  It integrates user interactions and content relevance.  It is trained to deal with vote spam attacks.  It's not possible to do this "post-factum" because it would be much too slow.  Answers already deals with some obvious vote spam in this way and while it awaits moderation, the user experience is degraded.  As social networks grow, social network spam becomes more sophisticated and it can change significantly due to the varying popularity of the content.  Voting spam includes not only malicious voting but also non-expert voting.  If you don't know anything about VB.Net and you vote on a post about it, your vote is not as important as the vote of an expert.

They're developing a toolkit to support expertise sharing via social networks.  The toolkit "SISN" (Seeking Information via Social Networks) is "is a general purpose toolkit for social network-based information sharing applications that combines techniques in information retrieval, social network, and peer-to-peer system".

Their toolkit requires: