Science for SEO

ECIR 2008 took place earlier this year and Nicholas Belkin did a keynote speech on the Grand Challenges in IR.  This always feels to me a little bit like a telling off to the IR community on what we haven't done properly or even considered as yet and where we've totally failed.  It's very important for this to happen because it sets the focus on the most important challenges we currently have in IR.

As an SEO person, this is of great importance to you because it tells you clearly where IR is heading.  The message here is loud and clear: Users.

1 - There needs to be more work done in the area of specification of tasks, intentions and information behaviours, in order to go beyond straightforward listings.

2 - There needs to be more research in user-behaviour analysis for IR.  Methods can be developed to infer information related goals, tasks and intentions from previous or concurrent behaviour.  Right now all goals have to be specified. 

3 - We need to develop IR techniques that respond to the above 2 challenges (identification of goals, tasks and characterizations).  This will require an interdisciplinary approach, as there is need for HCI, IR, AI and other people to achieve this goal.  That is also an obstacle right now.

People have a hard time specifying what would help them find the information they're looking for.  They change search behaviour several times during a single session.  We don't know enough about the different information behaviours and why people engage in one behaviour or another.  

We need to identify aspects of context, identify some subset of all possible factors leading to an information seeking situation, so that we can build contextually responsive IR systems.  I'll blog about this next because it's very interesting, but basically an experiment showed that unfortunately, everything is context.  This means that we have no real way of finding such a subset, but maybe we can identify some main aspects of this to improve support for information behaviours.

So far most mainstream IR research has been all about the efficiency and effectiveness of the IR system or the performance of the user.  Affective-computing is still in its infancy, and other fields of computing are also quite behind here, but we need to acknowledge the significance of this when we look at the user's experience of the IR system.  It's all about the role of emotions in the information seeking process.  This could help us understand what subsequent actions the user is likely to take for example, and of course understand where negative feelings arise and allow us to reduce them.

There hasn't been enough work in this area yet, it's all been too restricted.  We've been looking at click through paths, time spent on a page, previous and subsequent queries, relevance feedback,...Belkin says it's not good enough because we're only using one type of evidence in isolation.  There is research being carried out which shows that everything has an effect on everything else, and so if we look at our results in isolation, the results can be misleading.

So: "The challenge with respect to personalization is first to consider the dimensions along which personalization could and should take place; then to identify the factors or values in each dimension that would inform personalization on that dimension; then to investigate how the different factors or types of evidence interact with one another; and finally, to devise techniques which take account of these results in order to lead to a really personalized experience. 

Interacting with information is usually a consequence of someone wanting to achieve another goal or accomplish another task.  We need to ensure that a person never has to interact with a separate IR system, a person never has to leave their task to satisfy an information need.  We need to collaborate with the application communities so that IR gets integrated into real task environments.

Evaluation paradigms for interactive IR

TREC is the usual collection of documents used to evaluate an IR system, but it isn't very good.  There have been efforts at collecting better data sets, but generally, this is always a big problem for the research community.  

   

New less formal models of IR, such as language modelling for example are being introduced in the research community.  It's all good work, but it suffers from the fact that they focus on issues of representations of information objects, static queries, matching and ranking techniques.  They don't focus on  the user and on interaction, so there needs to be more research into formal models of IT that are truly interactive.

...back to work then...