Science for SEO: Blogosphere

December 15, 2008

Identifying the Influential Bloggers in a Community

This paper was presented at WSDM 08 by Nitin Agarwal, Huan Liu, Lei Tang (Arizona State University) & Philip S. Yu (University of Illinois at Chicago). "Identifying the Influential Bloggers in a Community" can be read at the ACM.

They look at the very important area of research concerning how we deal with the huge amount of data generated by bloggers and how we rank these blog posts.

I've presented you with a short summary of the main points:

Whether a blogger is active or not does not necessarily mean that s/he is not influential. Very active bloggers can be influential and just as easily not. The influential ones however are very important because they can help companies in developing new business ideas, identify key concerns and trends, competitive products,...Bloggers can become product advocates, and basically, they are market movers. The blogging on the recent US electoral campaign shows how bloggers can have influence over social and political issues also.

The researchers say that 64% of companies have identified the importance of the blogospere for their business. Instead of trawling through endless posts in the relevant community, the best entry point are the most influential posts.

Technorati reports a 100% increase in the size of the Blogosphere every month. This is huge and means that methods need to be developed in order to deal with this enormous amount of data.

You can't (as we've seen before) use PageRank or HITS or whatever method applied to search engines for the Blogosphere, because the blogs are sparsely linked, and the Random Surfer model just doesn't work for this. Web pages can gain authority over time, but this is not necessarily true of Blogs. As they say, a blog post and a bloggers influence actually decreases over time. This is because even more sparsely linked posts come into existence.

They say that there is research going on regarding ranking on topic similarity but this is still very much on the drawing board right now. They say that you could use traffic information, number of comments and more of these kinds of statistics, however you'd be leaving out all of those inactive bloggers.

They identify 4 groups of bloggers:

"active and influential, active and non-influential, inactive and influential, and inactive and non-influential". They create an influence score based on whether the blogger has any influential posts.

You're influential in the following circumstances (obviously you could probably add quite a few more):

Recognition - An influential blog post is recognized by many.
Activity Generation - A blog post’s capability of generating activity (comments, follow-up discussions...)
Novelty - Novel ideas exert more influence (lots of outlinks means that the post is not novel)
The blog post length is positively correlated with number of comments which means longer blog posts attract people’s attention.

For example:

Active & influential:

"‘Erica Sadun’ submitted 152 posts in the last 30 days, among which 9 of them are influential, attracting a large number of readers evidenced by 75 comments and 80 citations".

Inactive but influential:

"‘Dan Lurie’ published only 16 posts (much fewer than 152 posts comparing with ‘Erica Sadun’, an active influential blogger) in the last 30 days".

This is a very good example of a paper addressing the issues we're encountering in Blog post retrieval, categorisation and so on. It is a very very important area of research and needs imho to receive a lot more attention and budget dare I say :)

November 12, 2008

Blogosphere vs Web - ranking issues

I came across a very cool paper from SIGKDD 2008 called "Blogosphere: Research Issues, Tools, and Applications" by Nitin Agarwal and Huan Liu from the University of Arizona. It's an easy but long read, for the geek, but can also be quite happily understood by the layman. I've pulled out some things that I thought were interesting and given you a short taster here, but I urge you to read the paper, it's brilliant.

There is a model of the web, called the webgraph, where each webpage is a node and each hyperlink an edge. It provides a visual model of the web, which can be used for many things, such as for example search engines that use this graph for ranking documents.

We can't map the blogosphere in the same way because the number of links is sparse, and blog posts are dynamic and short-lived quite often. Also the comment structure which provides for interaction does not exist in the webgraph model. The webgraph assumes that sites build links over time, this isn't so in the blogosphere. We cannot use a static graph like the webgraph.

One way to model the blogosphere is to gather data concerning link density, how often people create blog posts, burstiness and popularity, and how these blog posts are linked. also it's possible to use the blogrolls to find similar blogs. This is what Lescovek et al. did, they used a cascade model usually used in epidemiology:

"This way any randomly picked blog can infect its uninfected immediate neighbors probabilistically, which repeats the same process until no node remains uninfected. In the end, this gives a blog network."

Brooks and Montanez used tf-idf to find the top 3 words in every post and then computed blog similarity based on that, which means that they could cluster them.

The problem is that these methods are keyword based clustering and therefore have high-dimensionality and sparsity issues. You could reduce this by using LSI but the results still aren't so good.

Many companies have already seen the usefulness of blogs for sentiment analysis, trend tracking and reputation management. Some systems use manually tagged sentences with negative/positive references, then using a naive-bayed classifier until everything has been classified.

Another way of finding the edges on the graph is by taking the topic similarity between 2 blogs. This is a good idea, but using this method is still under research and very difficult.

iRank is a "blog epidemic analyzer", and predicts if 2 blogs should be linked (BlogPulse uses this). They look for "infection" (how the information is propagated), so their aim is to find the blog responsible for the epidemic. These are the authority blogger, the influential ones in the blogosphere. It's good news when you find these bloggers because you can use them for word-of-mouth marketing as it were. They provide valuable information that companies may be interested in, they may employ the blogger for example because s/he gives brilliant information to people about their products.

Another method to infer this has been to predict the odds of a page being copied or read, and also look at topic stickiness. The most influential node is chosen with each iteration. It apparently outperforms both PageRank and hits for this task.

Splogs (spam blogs) are the equivalent of link spam in search engines. On the web algorithms include variables such as keyword frequency, tokenized url, length of words, anchor text and more. PageRank computed a score which it uses to identify splogs. This doesn't work on blogs unsurprisingly because they are too dynamic for spam filters to be effective. This issue hasn't been resolved as yet, although there is research in this area, and things are improving.

Link analysis is also used to find patterns. The text around the links is used, and based on those links hubs and authorities are found. You could use comments as links between the blogs. An influence score could be determined by taking into consideration inbound links, comments, length of posts, and links out.

This is a fun and really interesting are of research, keep an eye on new things emerging from this research community.