Science for SEO: code

December 03, 2008

Making Twitter bots

As some companies have figured out, there is a lot of information to be gleaned from Twitter that can benefit their business. They can manage reputation and customer service for example. There are however other reasons to collect information which do not involve marketing motivations but for example as I do, for research reasons. I study the language used, behavioural analysis and that sort of thing.

You may also want to collect information for your own purposes and for this you can write your own specialised Twitter bot from scratch or use a library provided by kind souls.

Google provide some help by providing an infrastructure for you. Using this you only need to code the logic rather than the whole thing yourself. It interacts with the Twitter API respecting the terms of service, and manages the enrollment of followers. The twitter account acts as the bot. It's built on the .Net 2.0 framework.

There are other bot code resources available in Google code, here are some nice ones:

Python-Twitter - python wrapper for the Twitter API: allows people to connect via the web, IM, and SMS.

TwitterBroadcastBot - in Ruby - broadcasts a message whenever a friend update contains a certain word. Good for info flagging.

ContactsNearby - ASP.Net & InSTEDD- allows you to find out the geographical location of your friends on Twitter. Plugs into Facebook too.

MadCow - python - has a whole host of features for you to use like tracking bookmarks, ip lookup, and silly things like ASCII art and loads more.

Nathan at Flowing data has a nice tutorial on how to build your own Twitter bot, take a look - he also provides some code to get you started.

You can automate a lot of things using CRON. You can use UNWIN if you're on windows and not Unix.

October 18, 2008

MAMA by Opera Dev

"The Metadata Analysis and Mining Application" (MAMA) engine is a search engine that works in a really different way. It indexes based on page structure: markup style, scripting, coding,...

"Say you want to find a sampling of Web pages that have more than 100 hyperlinks or for pages that use the Font-size CSS property that also use the FONT element with a Size attribute? Many parties would be interested in such a service, even if the market would be smaller than for a "traditional" search engine."

They also say:

Browser manufacturers and others can use MAMA data on the popularity of widely used technologies to prioritize bugs and justify adding support for new technology to in-progress releases.
Standards bodies can use the data to measure the success and adoption rates of various technologies.
Web developers can use the same data to justify support of various technologies in their work.
It can provide real-world, practical samples of the Web developer's "art", for inspiration and instruction.

You can get a load of interesting facts from it like, what the most popular element on the internet is, or how popular Fash is,...there's a key findings report you can have a look at. It has a load of stats on things like which server is the most popular, how many URLs in the index passed validation, the average length of external CSS and so much more.

You'll also find interesting the document "The "average" web page", all about how those look today from a structural perspective.

It's a dream come true for developers, and I think that it'll be quite popular.