Linux Format 29 [[ Typographical notes: indented text is program listing text surrounded in _underscores_ is italicised/emphasised ]] Perl Tutorial TITLE: mp3 madness STRAP: Perl isn't the first thing that springs into your mind when you think about playing mp3 music files under Linux, but as Charlie Stross demonstrates, you'd be surprised how handy perl can be. SUBTITLE: A brand new toy I have a CD problem -- there are too many of the things cluttering up my study, and I'm too lazy to put a new disc in the CD player every hour while I'm working. Some time ago I set out to do something about it. Hard disk space is cheap, the MPEG3 compression format lets you store a typical CD's worth of music in 50Mb of space (at 128kb/sec sampling speed), and so a 20Gb drive would solve most of my music problems by giving me a jukebox for my top 400 or so albums. The state of sound on Linux has taken big leaps forward since the early days, with ALSA, the advanced linux sound architecture (www.alsa-project.org) slowly supplanting the old and somewhat crumbly OSS (open sound system) drivers, and a variety of mp3 players, rippers, and related tools becoming available. (I'm not going to focus on Ogg, a new free compressed soundfile format from www.xiph.org, other than to say that this is probably the wave of the future -- but I'm living in the present, and with 400+ disks to re- rip if I want to switch over, I'm still stuck on mp3.) This isn't the place to describe how to play mp3's on Linux, but a brief hitchhiker's guide will make the rest of this tutorial a bit more meaningful. Linux can drive a number of PC sound output devices (and sound chips on non-PC hardware, such as the AWACS sound chip on my Apple iBook). In general, you need to load a kernel module that drives your sound hardware. There are two families of sound driver; the older OSS drivers, and the new ALSA drivers. ALSA is thread-safe, multi-processor safe, and provides a legacy API so that OSS applications can use it; if given a choice, choose ALSA. Your applications usually send data to, and read data from, a sound card via a couple of special device files: /dev/dsp (a digital sampling device -- you read incoming audio data from this), /dev/audio (a Sun compatible audio output device that takes .au files and makes sounds come out of your speakers), /dev/mixer (controls volume settings for each device, left/right balance, and other parameters), /dev/sequencer (if present, lets you play MIDI, FM, and other sequencer files), and /dev/sndstat or /proc/sound (which you can read sound driver settings from). You can copy a sound stream from /dev/dsp to a file, or copy a .au file to /dev/audio and make noises: all else boils down to file format translation. (For a more detailed overview see the Sound-HOWTO: http://tldp.org/HOWTO/Sound-HOWTO/index.html. You may also want to look at the MP-Playing-HOWTO: http://tldp.org/HOWTO/MP3-HOWTO.html.) Now, what exactly is an MP3 file? MPEG3 is a standard for file compression oriented towards frame based video and audio files. The Compact Disk Digital Audio (CDDA) format used on CDs is basically a raw, uncompressed bit-stream with some additional error-correction encoding, recorded in 12-bit chunks at a sampling speed of 44100 samples/second. MPEG3 applies some fairy efficient compression to the bitstream, and reduces the volume of data from the roughly 10Mb/minute of a CD to roughly 1Mb/minute (at high quality), or as little as 200Kb/minute (if you lower the sampling rate to approximate an AM radio broadcast). MP3 files are optimized for decompression, because their usual use is playback; it takes a lot longer to squish the data down than to unpack it again. The MP3 file format isn't just a straightforward stream of sound; it needs to contain two or more channels (for stereo), volume information, timing information, and ID3 tags that provide metainformation about the track's creator and name. The usual process of turning a CD into a set of MP3 tracks is taken care of by a tool such as Grip (from http://www.nostatic.org/grip/); Grip is a front-end for a bunch of other tools. First in the chain is either cdda2wav or the more sophisticated cdparanoia. These tools read a audio CD and 'rip' the tracks on it into WAV files -- straight digital sound streams on disk. Grip then uses another tool (such as Lame, Bladeenc, l3enc, or any other MP3 ripper) that reads a WAV file and writes a corresponding compressed MP3 file. Because CD's don't contain track and artist information in human-readable form, Grip also generates a disk-ID number for the CD and polls the Free-CDDB server to see if anyone's entered the artist and track details for that disk. (Free-CDDB is a centralized database, intended to save the effort of typing in all the details by hand every time; see http://www.freedb.org/modules.php?name=Sections&sop=viewarticle&artid=6 for details of how it all works. In use, all you need to know is that if you use Grip and have a live internet connection, you won't need to type in the track names unless the disk is truly obscure: and if you do, you can do everyone else a favour by telling Grip to upload the details to freedb.org.) To play an mp3 file under Linux, first you need to make sure you've got working sound drivers; then you use an mp3 player to convert the mp3 into a .au stream that /dev/audio can chew on. On Linux, you've got a huge choice -- from the simple-looking command line tools like mpg123 to hairily complex graphical players like Xmms (formerly X11Amp, a WinAmp clone) which can also handle streaming mp3s, ogg files, special effect filters, and act as a graphic equaliser (adjusting gain at various frequencies). I decided to do things a bit differently. Archos (see www.archos.com) make a very nice toy called the Archos Jukebox Recorder 20. This is basically a portable 20Gb hard disk with an mp3 decoder in firmware. You can plug it into your Linux system via a USB cable, and mount it as a USB mass storage device (which makes it look like a slow SCSI disk to the kernel). Once mounted you can copy MP3 files and M3U playlists onto it, then unplug it and leave the job of playing music up to the dedicated jukebox instead of having it bog down your computer. SUBHEADING: Finding files and building playlists One of my first headaches with the jukebox was this: you can tell it to play all the tracks in a directory, but it won't dive into subdirectories. With four hundred odd albums loaded onto it (thanks to a marathon session with Grip) I settled on the obvious file structure of having a directory for each artist or band, and a subdirectory for each album. How do you get a jukebox to play every track by an artist, if it only plays the MP3s it finds in one directory? It turns out that WinAmp introduced a special type of file called a playlist. A playlist is, at its simplest, a list of filenames presented one per line; alternatively URIs may be used for remote streaming files, but if we're doing this on a jukebox all we need is relative pathnames to each file in the list. At its simplest, we can build a playlist without using Perl, because a simple shell script does the job: ///BEGIN CODE/// #!/bin/sh JUKE=/mnt/jukebox $base=$(pwd) for foo in $( find $JUKE -maxdepth 1 -type d -print) do echo processing $foo NAME=$(basename $foo) cd $foo find . -type f -name '*.mp3' -print > $foo/$NAME.m3u cd $base done ///END CODE/// But what happens if we want a hybrid playlist containing, say, all the tracks recorded by Bjork -- and The Sugarcubes (her original band)? Or all the tracks from albums she was involved in between 1989 and 1995? Here's a first cut at the sort of thing we might do: ///BEGIN CODE/// #!/usr/bin/perl use File::Find; my $startdir = '/home/mp3'; my $target = qr(bjork|sugarcubes); find(\&wanted, $startdir); sub wanted { if (/.*\.mp3$/ && lc($File::Find::name) =~ $target) { print "$File::Find::name\n"; } } ///END CODE/// This is a fairly simple application of a very handy perl module -- File::Find. File::Find is part of the standard perl distribution. Perl, in addition to being a sufficiently strict superset of the sed and awk text editing tools to have actual sed-to-perl and awk-to-perl translators, is also a superset of find. find (used in the first example) walks a directory tree and applies a series of tests (specified on the command line) to whatever files it finds in a directory. It discards files as they fail the tests, and then applies any specified actions to the remaining files -- for example the -print action prints the file's name, and the -exec action runs another program (you use two empty curly braces, {}, to denote the filename when it's time to interpolate them into your -exec command). Perl's file test operators are unary operators applied to a scalar variable. For example: ///BEGIN CODE/// -d '/home/mp3' ///END CODE/// Returns true if '/home/mp3' is a directory, and false otherwise. '-r', '-w', and '-x' test for the 'read', 'write', and 'execute' attributes (in the context of the current processes' effective user ID -- that is, they test whether the program can read, write or execute the file). '-z' tests if the file is of zero length, while '-s' returns the size of the file in bytes; '-T' tests whether it's a text file, and '-B' tests whether it's a binary file. You can daisy-chain file tests together; for example: ///BEGIN CODE/// if ( -r $myfile && -s $myfile && -T $myfile ) { # $myfile is readable and of non-zero length and contains text ///END CODE/// Every time you use a file test operator, Perl has to execute the stat() system call. You can cut down on this filesystem access by either calling stat() yourself in Perl (it returns an array of descriptive data relating to the file), or you can use a shortcut -- Perl caches the test information for the last file you tested, and the symbol '_' (a single underscore character) points to this copy: ///BEGIN CODE/// if ( -r $myfile && -s _ && -T _) { # same as above, but with less disk access ///END CODE/// File::Find is a module that lets us write directory-traversal code in Perl. It provides a couple of utility subroutines, the most important of which is find(). Find takes a variety of parameters, but the bare minimum is a code reference pointing to a test subroutine, and a directory to traverse. For example: ///BEGIN CODE/// find(\&wanted, $startdir); ///END CODE/// This tells find() to search all directories within $startdir for files, and for every file, to run the subroutine wanted(), which is defined later in the program. Inside wanted(), we have access to some special variables. $_ is set to the name of the current file, $File::Find::name to its complete pathname, and $File::Find::dir is the current directory's name. wanted() implicitly calls chdir() to move into the current subdirectory before handing control to whatever code you've written. So our example program above does the following: ///BEGIN CODE/// if (/.*\.mp3$/ && lc($File::Find::name) =~ $target) { print "$File::Find::name\n"; } ///END CODE/// If the filename (that's in $_) ends in '.mp3', and the full pathname of the file matches the quoted regular expression $target, print the complete pathname. $target contains a quoted regular expression, specified with qr() -- everything inside the brackets is transformed into a regexp. This makes it easy for us to write programs that take regular expressions as parameters; we can turn strings into regexps and use them later. Here, we're just using it for clarity: our search pattern is 'bjork|sugarcubes', which matches _either_ 'bjork' _or_ 'sugarcubes'. Handy tip: if you can think of a find(1) expression to locate a file, and want some perl that does the same job, use the program find2perl. It'll spit out the perl for you. For example, if you want to write a perl script equivalent to: ///BEGIN CODE/// find /home/mp3 -type f -mtime -3 -name '*.mp3' -print ///END CODE/// (This means: find all items of type 'regular file' with a modification time less than three days ago and a name matching '*.mp3', then print their names) ... you can subsitute 'find2perl' for 'find' in the above line and it will spit out something like this: ///BEGIN CODE/// #! /usr/bin/perl -w eval 'exec /usr/bin/perl -S $0 ${1+"$@"}' if 0; #$running_under_some_shell use strict; use File::Find (); # Set the variable $File::Find::dont_use_nlink if you're using AFS, # since AFS cheats. # for the convenience of &wanted calls, including -eval statements: use vars qw/*name *dir *prune/; *name = *File::Find::name; *dir = *File::Find::dir; *prune = *File::Find::prune; # Traverse desired filesystems File::Find::find({wanted => \&wanted}, '/home/mp3'); exit; sub wanted { my ($dev,$ino,$mode,$nlink,$uid,$gid); (($dev,$ino,$mode,$nlink,$uid,$gid) = lstat($_)) && -f _ && (int(-M _) < 3) && /^.*\.mp3\z/s && print("$name\n"); } ///END CODE/// This shows us how to traverse a directory tree and build a list of files -- if we just modify that final print statement to print to a file handle we've previously opened (on a playlist somewhere) we can turn its output into an m3u playlist containing only those mp3 files that were modified in the past three days. And by using regular expressions we can pick only mp3 files with the right name. But is that the best way to do things? SUBHEADING: Chasing ID3 tags It is a sad fact of life that the name of a file is not inextricably related to its contents. I have no guarantee that my mp3 ripper correctly connected to the FreeCDDB service and correctly retrieved the name of the files it was pulling off the CD; I might have a potload of directories called 'untitled artist' containing subdirectories called 'untitled album' filled with files called '01.mp3, 02.mp3 ...' and so on. Or some merry prankster might have given me a copy of a valuable and rare disk, or what _looks_ like a copy -- with all the right names, but with every track replaced by "I am the Walrus" by The Beatles. And we certainly can't expect a simple-minded find script to pull in all tracks from albums Bjork participated in between 1989 and 1995, because the necessary information (about when the album was recorded) simply isn't in the filename. It turns out that there is a way to name mp3 files in such a way that you can't lose the information simply by renaming the file. mp3 files contain ID tags -- specifically, ID3 tags, a standard record structure embedded in the header of the file that indicates the artist, album, track name, year, copyright status, and other important information. The ID3 tags are also used by MP3 players (like the Archos jukebox) to provide captioning information during playback (so you can see what you're playing). There are numerous ways of setting or examining ID3 tags, but for this article I'm going to look at two -- the command-line utility id3tool (from http://kitsumi.xware.cx/id3tool/), and the Perl modules in MP3::Tag (MP3::Tag::ID3v1 and MP3::Tag::ID3v2). id3tool is a Version 1 ID3 tagfile editor; MP3::Tag, in contrast, can handle v2 tags as well. Install them off CPAN using: ///BEGIN CODE/// perl -MCPAN -e 'install MP3::Tag;' ///END CODE/// Let's look at id3tool first. To set the Version 1 ID3 tags on a file: ///BEGIN CODE/// id3tool -t 'World in my Eyes' -a 'Violator' -r 'Depeche Mode' 01-world\ in\ my\ eyes.mp3 ///END CODE/// To read the tags from a file: ///BEGIN CODE/// #id3tool 01-world\ in\ my\ eyes.mp3 Filename: 01-world in my eyes.mp3 Song Title: World in my Eyes Artist: Depeche Mode Album: Violator # ///END CODE/// (Only ID3 information actually in a file gets returned by id3tool.) We can use id3tool from within perl to get information about MP3 files. It's crude but easy to code: ///BEGIN CODE/// my $file = '01-world in my eyes.mp3'; my $info = `id3tool $file`; chomp $info; my @info = split(/\n/, $info); my %info = (); foreach (@info) { my ($k, $v) = split(/:\s+/, $_); $v =~ s/^\s*//; $info{$k} = $v; } print "Title:", $info{'Song Title'}, "\n"; # and so on ///END CODE/// But this is inefficient for processing lots of files, each time we want to look at a new file we have to spawn a subshell and run an external program. It's far more efficient to use MP3::Tags: ///BEGIN CODE/// use MP3::Tag; $mp3 = MP3::Tag->new($filename); # get some information about the file in the easiest way ($song, $track, $artist, $album) = $mp3->autoinfo(); # or have a closer look on the tags # scan file for existing tags $mp3->get_tags; if (exists $mp3->{ID3v1}) { # do something with them $id3v1 = $mp3->{ID3v1}; # a handy shortcut print " Song: " .$id3v1->song . "\n"; print " Artist: " .$id3v1->artist . "\n"; print " Album: " .$id3v1->album . "\n"; print "Comment: " .$id3v1->comment . "\n"; print " Year: " .$id3v1->year . "\n"; print " Genre: " .$id3v1->genre . "\n"; print " Track: " .$id3v1->track . "\n"; if (! exists $mp3->{ID3v2} ) { # create a new ID3v2 tag $mp3->new_tag("ID3v2"); # create ID3v2 album title $mp3->{ID3v2}->add_frame("TALB", $id3v1->album); # write out the ID3v2 tag $mp3->write_tag; } } $mp3->close(); ///END CODE/// Version 1 ID3 tags are fields with pre-set names -- the ones in the example above should be self-explanatory. Version 2 added the ability to define arbitrary tag names: a _frame_ is a tag and its value, and tags are identified by four-character codes. There are simple frames (name/value pairs) and complex frames that may contain hashes of additional data -- for example the APIC frame, if present, may contain an attached picture (along with its content-type, mime-type, and a description). You can get a list of simple frames supported by MP3::Tag by calling $mp3->id3v2->get_frame(). ID3v2 is useful because it's extensible, and has a whole load of handy frames available. For example, there's a frame for an involved people list (IPLS); if we've got a properly tagged album, that's exactly what we need to identify all records Bjork was involved in. If that isn't enough, there are tags for Lead performer/Soloist (TPE1), Conductor/performer (TPE3), Lyricist (TEXT), Composer (TCOM), and so on. There are also tags that tell us how the MP3 is being used -- Play counter (PCNT), and TPUB (publisher) or WPAY (payment). If we're writing an mp3 server, we can keep an eye on how popular our tracks are by splicing the following code into our download server scripts: ///BEGIN CODE/// use MP3::Tag; my $mp3 = MP3::Tag->new($filename); $mp3->get_tags; if (! exists $mp3->{ID3v2} ) { $id3v2 = $mp3->new_tag("ID3v2"); $id3v2->add_frame("PCNT", 1); } else { my ($info, $name) = $id3v2->get_frame("PCNT"); $info++; $change_frame("PCNT", $info); } $id3v2->write(tag); ///END CODE/// (This opens the mp3 file $filename and gets the tags. If no v2 tags exist, it creates them, adds the new frame PCNT with a value of 1, and saves it. If they exist, it retrieves the value of PCNT, increments it, and re-saves it. So each time we call this code, the play counter is incremented.) Now, here's a fun headache: Grip doesn't automatically write ID3 tags when it rips files. (Neither does Apple's iTunes 2.0, for that matter.) How can we easily build ID3 tags for a directory tree full of files we just ripped? It turns out that MP3::Tag is smart enough that it can try and yank the information out of the file's pathname if no tags can be found. To configure it to do this, we call MP3::Tag->config(), and tell it to use a specific search order among ID3v1 tags, ID3v2 tags, or the filename, when returning information with autoinfo(). We can then use autoinfo to get our basic information, and create some tags. Like this: ///BEGIN CODE/// #!/usr/bin/perl use MP3::Tag; my $file = shift @ARGV; MP3::Tag->config("autoinfo", "filename", "ID3v1", "ID3v2"); my $mp3 = MP3::Tag->new($file); my ($song, $track, $artist, $album) = $mp3->autoinfo(); print "song: $song\ntrack: $track\nartist: $artist\nalbum: $album\n\n"; if (! exists $mp3->{ID3v2} ) { $mp3->new_tag("ID3v2"); my $id3v2 = $mp3->{ID3v2}; $id3v2->add_frame("TALB", $album); $id3v2->add_frame("TOPE", $artist); $id3v2->add_frame("TRCK", $track); $id3v2->add_frame("TIT2", $song); $id3v2->write_tag(); } if (! exists $mp3->{ID3v1} ) { $mp3->new_tag("ID3v1"); my $id3v1 = $mp3->{ID3v1}; $id3v1->song($song); $id3v1->artist($artist); $id3v1->album($album); $id3v1->track($track); $id3v1->write_tag(); } $mp3->close; ///END CODE/// This won't get you the interesting year information, but it helps with the basics by building you a rudimentary set of ID3 tags even if your ripper doesn't do it for you. For that you need to go to CDDB. SUBHEADING: Querying CDDB in Perl The FreeDB and CDDB databbases provide a network interface to allow client programs to talk to them; and if you're using Perl, a variety of modules are available. First, there's CDDB.pm itself. This module is a formal client-side implementation of the CDDB API. Given an actual music CD's table-of-contents data, it composes a query and either retreives the CDDB records for corresponding disks, or provides facilities for submitting a CDDB entry. This is not a lot of use if what you have is a collection of MP3s with partial ID3 tags, and what you want to do is flesh out the information held on them. For example, let's go back to Bjork. Using MP3::Tag, we've created ID3 tags for all our MP3's -- but they don't include release year or some of the supplementary information we want. To solve the problem we need a couple of tools -- one to find the CDDB entry number for an album, and one to retreive the CDDB entry and parse it. You can get an album's CDDB ID via the web-based search interface to CDDB servers. For example, you can search FreeDB via the URL http://www.freedb.org/freedb_search.php, which takes parameters such as "words"; a GET request for: ///BEGIN CODE/// http://www.freedb.org/freedb_search.php?words=bjork ///END CODE/// Retreives an HTML page containing links to a URL like: ///BEGIN CODE/// http://www.freedb.org/freedb_search_fmt.php?cat=blues&id=bf11350e/ ///END CODE// With a text caption of the artist and album title. (And spot the CDDB ID tags are at the end of the URL.) This makes it possible for us to yank a CDDB record for a specific named album, but as with much code for searching online databases it gets messy fast. First, we use LWP::Simple to fire a web request at http://www.freedb.org/freedb_search.php; then we run the response through HTML::LinkExtor to extract the HREF links. We look through these for links that look like hits -- ones with cat=(category)&id=(number) appended to them. We then use LWP::Simple to retreive the appropriate FreeDB entry file from the server, and can use CDDB::File to parse it and turn it into a Perl object that we can work with. For an encore, try adding code to validate these search results -- the database may give more hits than you expect for a given artist and album title (for example, there are three entries for "Life's too Good" by The Sugarcubes -- partly because there were at least two different versions of the album, including one with weird Icelandic versions of all the songs). But once you've done it, typically by counting track numbers and durations, the way is clear to build ID3 tags using them. ///START CODE/// #!/usr/bin/perl use LWP::Simple; use HTML::LinkExtor; use CDDB::File; my $artist = 'depeche mode'; my $record = 'Master and servant'; #"personal jesus"; my $cddb = "http://www.freedb.org/freedb_search.php"; my $words = $artist . " " . $record; $words =~ s/\s/+/g; my $response = ""; $doc = get("$cddb?allfields=YES&allcats=YES&words=$words"); # now we need to parse the result file for links and extract CDDB index numbers my $p = HTML::LinkExtor->new(); $p->parse($doc); my @links = $p->links; @links = grep {$_->[2] =~ /\&id=/ } @links; # get rid of non-relevant URLs @links = map { $_ = $_->[2] } @links; # get rid of spurious fields # @links is now an array of URIs into CDDB. Let's turn 'em into entry ID numbers foreach (@links) { $cat = $id = $_; $cat =~ s/^.*cat=(.*)\&.*$/$1/; $id =~ s/.*id=([\dabcdef]+)$/$1/; # now we have a category and an ID we can retreive the CDDB data file my $request = "http://www.freedb.org/freedb/$cat/$id"; $doc = scalar get($request); open (OUT, ">$cat.$id.tmp"); print OUT $doc; close OUT; my $cddb = CDDB::File->new("$cat.$id.tmp"); unlink "$cat.$id.tmp"; print "Category: ", $cat, "\n"; print "CDDB ID: ", $id, "\n"; print "Artist is: ", $cddb->artist, "\n"; print "Disk is: ", $cddb->title, "\n"; print "Year was: ", $cddb->year, "\n"; } exit; ///END CODE/// (ENDS)