Evaluating programming languages for playing with Debtags
Since having workable bindings for the C++ Debtags libraries seems to be still
a bit in the future, I'm planning to build a bit of native infrastructure in
some higher level language. First step is seeing what language I could start
playing with.
The problem
At the most basic level, in Debtags we have a number of packages, each of which
have a set of tags.
The way I usually save tags is a file with the format:
package1, package2: tag1, tag2, tag3
package3: tag1, tag2
That is, every line has a list of packages with the same tags, and the list of
their tags.
Since any script I'm going to write has to at least be able to parse the data
into something like a package -> tags hash, then print it out.
Let's see how perl, python and ruby perform.
Tests
C++
The reference point for the experiment will be the C++ implementation,
tagcoll:
$ time tagcoll copy package-tags > /dev/null
real 0m0.421s
user 0m0.412s
sys 0m0.000s
Perl
First attempt is with Perl, creating the script that parses into a hash of
package => set of tags and prints the result.
There are set modules
for Perl on CPAN, but I have none handy at the moment. However, since they are
implemented using hashes, I can approximate them by using a hash.
Note that I also want to have a different copy of the tag set for every
package, so that I can manipulate them in the future without unwanted side
effects.
Here is the code:
#!/usr/bin/perl -w
use strict;
my %db;
# Read the tag database
while (<>)
{
chop();
my ($pkgs, $tags) = split(': ');
# Create the tagset using keys of a hash
my %tags = map { $_ => undef } split(', ', $tags);
for my $p (split(', ', $pkgs))
{
# Make a copy of the tagset
$db{$p} = {%tags};
}
}
# Write the tag database
while (my ($pkg, $tags) = each %db)
{
print $pkg, join(', ', keys %$tags), "\n";
}
Here is the running time:
$ time ./parse.pl package-tags > /dev/null
real 0m0.448s
user 0m0.436s
sys 0m0.008s
Not so bad, comparable with tagcoll.
Python
Then comes Python. I'm not much of a Python fancier, but I'm rather attracted
by the new set native type introduced with Python 2.4, which seems to have
most of what I need nice and done.
Here is the script:
#!/usr/bin/python
import sys
input = sys.stdin
if len(sys.argv) > 1:
input = open(sys.argv[1],"r")
# Read the tag database
db = {}
for line in input:
# Is there a way to remove the last character of a line that does not
# make a copy of the entire line?
line = line.rstrip("\n")
pkgs, tags = line.split(": ")
# Create the tag set using the native set
tags = set(tags.split(", "))
for p in pkgs.split(", "):
db[p] = tags.copy()
# Write the tag database
for pkg, tags in db.items():
# Using % here seems awkward to me, but if I use calls to
# sys.stdout.write it becomes a bit slower
print "%s:" % (pkg), ", ".join(tags)
Here is the running time:
$ time ./parse.py package-tags > /dev/null
real 0m0.418s
user 0m0.376s
sys 0m0.036s
I'm pleased, very pleased. Using the native set seems to be not only handy,
but efficient.
Ruby
Finally, Ruby. I like to use Ruby. In this case, however, it lacks a native
set implementation, although it has a set module which is implemented using a
hash.
Here is the script:
#!/usr/bin/ruby
require 'set'
infile = ARGV[0] ? File.new(ARGV[0]) : $stdin
# Read the tag database
db = {}
infile.each_line do |line|
line.chop()
pkgs, tags = line.split(": ")
# Create the set using the Set module
tags = Set.new(tags.split(", "))
pkgs.split(", ").each do |p|
# Is this a copy or a reference? I need to find out.
db[p] = tags
end
end
# Write the tag database
db.each do |key, tags|
# Ouch, Set does not do join by itself
print key, ": ", tags.to_a.join(", ")
end
Here is the running time:
$ time ./parse.rb package-tags > /dev/null
real 0m1.637s
user 0m1.572s
sys 0m0.052s
I hope I got something wrong in the script, but I can't see what.
Results
As much as I don't fancy Python, it looks like it's currently the best choice
for playing around with Debtags. I hope the native sets will bring me joy.
If in the future I'll be asked "how come you chose Python for this Debtags
thing?", I can point to this page.
