introsp.icio.us: Using del.icio.us to identify your interests

The Ruby script below analyzes your del.icio.us links to generate a list of your favourite interests. It lists the tags that are most frequently used in your links. Because you may not have been diligent about descriptively tagging your links, this script uses the tags from *all* users, for *your* links.

The result is a list of tags, sorted by the number of your links that have been associated with each tag. In effect, you get a list of your favourite interests, starting with the ones you enjoy the most. It is a means of introspection, using the descriptive power of shared tagging in del.icio.us. Hence the name of the script: introspicious.

How this script came to be: I stumbled across extispicious, which is a neat mind-map of your del.icio.us tags. But when I tried it, I was disappointed to find that it only used my tags -- you see, I don't use very descriptive tags for my del.icio.us links. I wished there was a way to show the most common tags from all users, for my links, as a pseudo-quantitative way of understanding my favourite interests. Thus, introspicious was born.

Here is the beginning of the results for my links. As you can see, I am very interested in programming, news articles, and blogs:



Here's a graph of the numbers:


Note that there are a few tags that frequently came up in my links (e.g. programming). In contrast, there are a lot of tags that each appear in only one of my links (e.g. zzread200411).

So there you have it -- a script to analyze your del.icio.us links to determine your favourite interests. (Would someone be willing to write a web front-end for this script, so that it is more accessible to others?)



Instructions:
1. Save this script as "introspicious.rb"
2. Edit it to use your del.icio.us username and password
3. Download and install Ruby
4. Download http://www.tartarus.org/~martin/PorterStemmer/ruby.txt and save it as "stemmable.rb" in the same directory as this script
5. Run this script using "ruby introspicious.rb"

#! /local/ruby/bin/ruby
# introspicious.rb. For more information, see
# http://jonaquino.blogspot.com/2005/03/introspicious-using-delicious-to.html

$delicious_username = 'JonathanAquino'
$delicious_password = 'tiger'

require 'net/http'

def get(path, http)
  # Wait 1 second between queries, as per
  # http://del.icio.us/doc/api [Jon Aquino 2005-03-06]
  sleep 1
  puts path
  req = Net::HTTP::Get.new(path)
  req.basic_auth $delicious_username, $delicious_password
  http.request(req).body
end

def checksum(xml)
  xml =~ /hash="([^"]+)"/
  $1
end

tags = []
Net::HTTP.start('del.icio.us') {|http|
  get('/api/posts/all', http).each {|post|
    next if not post =~ /hash/
    tags_for_url = []
    get('/url/'+checksum(post), http).each {|line|
      next if not line =~ /^\s*<.*delNav.*>(.*)</
      tags_for_url << $1.downcase
    }
    puts "Tags: #{tags_for_url.uniq.join(", ")}"
    tags += tags_for_url.uniq
  }
}

# Use the Porter Stemming Algorithm to strip suffixes from tags, thus
# combining similarly-named tags. [Jon Aquino 2005-03-06]
require "stemmable.rb"
class String
  include Stemmable
end
stems = []
stem_to_example_tag_map = Hash.new
tags.each { |tag|
  stems << tag.stem
  stem_to_example_tag_map[tag.stem] = tag
}

class StemCount 
  attr_reader :stem, :count
  def initialize(stem)
    @stem = stem
    @count = 0
  end
  def inc
    @count += 1
  end
end

stem_to_stemcount_map = Hash.new
stems.uniq.each { |stem| stem_to_stemcount_map[stem] = StemCount.new(stem) }
stems.each { |stem| stem_to_stemcount_map[stem].inc }
stem_to_stemcount_map.values.sort { |a,b| b.count <=> a.count }.each { |stemcount|
  puts "#{stemcount.count} page#{stemcount.count==1?"":"s"} tagged as #{stem_to_example_tag_map[stemcount.stem]}"
}

Update: I have written a new version of introspicious that replaces
the Porter Stemmer with screen-scrapes of queries to the Google Directory,
which provides categories for the keywords you enter. Thus, instead
of using the Porter Stemmer to cluster tags etymologically, introspicious2
uses Google Directory to cluster tags semantically. Get introspicious2 here.