Building a Linux Media Network, one step at a time

Android SwingWorker

2009-07-08T12:18:00.000-07:00

I've been fooling around with the Android development platform. It's quite an adjustment, coming from the iPhone world. More thoughts to follow, although probably on a different blog. I'm getting sick of this blogger nonsense.

Anyways, I thought I'd post a simple equivalent to the SwingWorker class that works with Android's Event Dispatch Thread. Here it is:


package ca.razorwire.util;

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

import android.os.Handler;
import android.os.Looper;
import android.util.Log;

public abstract class UIWorker
{
  private static final ExecutorService __execSvc;
  private static final Handler __edtHandler;
  
  private static final String TAG = "UIWorker";
  
  static
  {
    __execSvc = Executors.newSingleThreadExecutor();
    __edtHandler = new Handler( Looper.getMainLooper() );
  }
  
  public abstract void doInBackground();
  
  public void done()
  {
    // Log.d(TAG, "done() executing in thread " + Thread.currentThread().getName() );
    // This space intentionally left blank
  }
  
  public void execute()
  {
    // Log.d(TAG, "execute() executing in thread " + Thread.currentThread().getName() );
    final Runnable doneRunner = new Runnable()
    {
      public void run()
      {
        UIWorker.this.done();
      }
    };
    
    final Runnable bgRunner = new Runnable()
    {
      public void run()
      {
        // Log.d(TAG, "bgRunner executing in thread " + Thread.currentThread().getName() );
        UIWorker.this.doInBackground();
        __edtHandler.post( doneRunner );
      }
    };
    __execSvc.submit(bgRunner);
  }
}

And here's a sample usage:


      UIWorker worker = new UIWorker()
      {
        public void doInBackground()
        {
          Log.d( TAG, "Sleeping for a bit." );
          try { Thread.sleep( 10000 ); } catch ( Exception ignore ) {};
        }
      };
      Log.d( TAG, "Executing worker..." );
      worker.execute();
      Log.d( TAG, "Returned from execute()" );

If you uncomment the Log calls in UIWorker, you should see some output like this:


07-08 12:14:11.015: DEBUG/rweeks(1078): Executing worker...
07-08 12:14:11.015: DEBUG/UIWorker(1078): execute() executing in threadmain
07-08 12:14:11.025: DEBUG/UIWorker(1078): bgRunner executing in thread pool-1-thread-1
07-08 12:14:11.035: DEBUG/rweeks(1078): Sleeping for a bit.
07-08 12:14:11.035: DEBUG/rweeks(1078): Returned from execute()
07-08 12:14:21.038: DEBUG/UIWorker(1078): done() executing in thread main

I'm 100% sure that Blogger is going to screw up the formatting.

Public domain code, no guarantees, seems to work.

Google/Flickr Image Scraper

2008-01-28T12:08:00.000-08:00

I just received my XO laptop through the Give-1-Get-1 program. I don't have a lot of plans for it yet but one thing I want to do is set up some photo albums for my 3-year-old son. Interests include: helicopters, airplanes, trucks, trains. The usual!

I wrote a small app to grab pictures from Google Images based on a query string. It was a short exercise in concurrent programming, more than anything else. Lesson learned: the Java 1.5 concurrency APIs don't make the producer/consumer design pattern as easy as you might think, particularly when it comes to producer shutdown.

After wrapping up the Google images scraper my friend Gwilli pointed out that Flickr is a much better resource for this sort of thing. D'oh! Fortunately it was a very small change to scrape their database, too.

Instructions:

Make sure you have Java 1.5 or Java 6. Download googlor.jar.
Run java -jar googlor.jar (in OS X, just double-click the jar file)
The fields are pretty straightforward. By default images will go into the images/ subdirectory of the current working directory.
When images are presented, press 'j' to junk them or 'k' to keep them. That's it.

I'm happy to make the source available to anyone who's interested.

(I should point out, the images grabbed from Flickr are low-res, maximum 500x500. This is a good fit for the XO's screen)

WideFinder in Java6: revision3

2007-11-22T13:04:00.000-08:00

Revision 3 of my WideFinder implementation is available here. I see a small performance gain when I stagger the thread starts by 50ms. I've tweaked the processing of the memory mapped "chunk" boundaries. Now it's 99.9999998% correct :(

WideFinder/Java6 Rev2

2007-11-13T17:20:00.000-08:00

... The sum total of the next 6 paragraphs is, "I tested a theory without collecting performance data and my changes seemed to have no impact on performance"... Skip to the DTrace Analysis section if that doesn't sound like a fun read...

In the initial revision of the WideFinder, I couldn't get away from the thought that I was handling the data one too many times. If you look at WideFinder.run (WideFinder.java:57),

  byte[] seq = new byte[ seek - _mbb.position() ];
_mbb.get( seq );
nextLine = new String( seq, LOG_CHARSET );

We're grabbing bytes out of the buffer, sticking them into a fresh byte array, and creating a new String object based on those bytes, using a specific charset. This is because the character set used by CharBuffer treats each byte as half of a 16-byte char, which means that the bytes are interpreted as non-English characters.

Creating new byte[]s and new Strings like that, in a pretty tight loop, is just asking for performance trouble. I wanted to get to the point where the regex could be applied straight to the ByteBuffer. Looking at the implementation of Charset.encode, it seemed like it suffered from the same sort of space-inefficiency as my original approach.

The solution was to wrap the ByteBuffer in a trivial implementation of CharSequence that indexed directly into the current line of the buffer. The only gotcha was to use the mark() and reset() methods of the buffer in the implementation of subSequence.

Unfortunately, my efforts produced only a minimal increase in performance (~2%) on either my single-core Athlon or the T2K. So much for "low-hanging fruit".

DTrace Analysis
I created a new revision of the Java 6 WideFinder with some tweaks to the ByteBuffer implementation. It's available here: wf_02.tar.gz. It turned out to be a few hours' effort for a very minimal performance gain.

With that little misadventure under my belt I decided to run the WideFinder through DTrace to see where the bulk of time was being spent. DTrace is Sun's dynamic tracing tool, it allows you to inspect a bunch of different aspects of a program's performance as it is running. If you've never used it, you should really give it a try. It's pretty amazing, the stuff you get visibility into. I gather it's also available on OS X as the "Instruments" app.

This is the output of running the method-timings.d script against the WideFinder on a 100-line log file (produced by the DataGenerator). Times given are in microseconds and represent the total time spent in methods in classes of the given package:

sun/net/www/protocol/file         103
sun/net/www                       266
sun/net/www/protocol/jar          375
java/nio/channels                 474
java/util/concurrent/atomic       791
sun/security/action              1065
java/util/jar                    1731
java/util/zip                    3146
java/net                         3654
java/security                    3807
java/lang/ref                    5319
java/util/concurrent             7419
java/util/concurrent/locks       8329
java/lang/reflect               10365
sun/misc                        20678
java/nio/channels/spi           25753
sun/nio/ch                      32353
java/nio/charset               160325
java/util                      197794
sun/nio/cs                     204016
java/io                        240720
sun/reflect                    293949
wf                             464158
java/lang                     1061836
java/nio                      1596102
java/util/regex               2738455

The absolute values are not as meaningful as the relative values. The act of observing the program's performance has skewed the measurements (where's that damned cat!), but we hope that it's skewed the measurements equally for each package. From these data, we can see that the "big spenders" are java.util.regex (~39% of total time), java.nio (~25% of total time, with subpackages), and java.lang (~15% of total time).

Here are the 10 most expensive methods in java.nio (not including subpackages), also measured in microseconds:

CharBuffer.arrayOffset     43204
CharBuffer.<init>          44219
ByteBuffer.arrayOffset     51038
HeapCharBuffer.<init>      62420
CharBuffer.wrap           122109
Buffer.position           280288
Buffer.limit              286871
Buffer.checkIndex         375539
DirectByteBuffer.ix       485376
DirectByteBuffer.get     1084831

And here are the 10 most expensive methods in java.util.regex (although there seem to be a lot of very expensive methods in this package):

Pattern.escape                   165501
Pattern$CharProperty.<init>      180954
Pattern.peek                     184470
Pattern$8.isSatisfiedBy          199079
Pattern.isSupplementary          238639
Pattern.atom                     240097
Pattern$Slice.match              261159
Matcher.getSubSequence           282353
Matcher.group                    291688
Pattern$BmpCharProperty.<init>   291799
Pattern$BmpCharProperty.match   1080132

Looks like, if there's low-hanging fruit, it's in the regex processing. A little regex-optimization may go a long way.

Data Processing and the Gravel Biz

2007-11-13T16:42:00.000-08:00

My Dad's in the gravel business. Ever watch The Flintstones? He's like Mr. Slate. His job is to get as much sand and gravel as possible out of a mountain and onto barges. From there, it floats down a river to a depot where (presumably) people are willing to pay for it.

The trick about the gravel business, as with any other commodity industry I guess, is that you pretty much live or die based on 2 factors:

How often you touch the product.
How much it costs you each time you touch it.

In my Dad's case, they can't drive loaded trucks down the steep hill of their quarry to get to the river bank. The road is too narrow to allow the trucks to pass each other. So this is what they do:

A dumptruck pulls up to a big exacavator, which is scraping away at the side of the mountain pretty much non-stop.
The excavator fills the dump truck. Takes somewhere around 5-6 scoops, I think.
The dump truck backs into position at the top of the cliff and waits for the all-clear to dump its load. You don't want to rush that task or you wind up with a lot of expensive metal at the bottom of a cliff.
While that dump truck is getting ready to dump its load, another dump truck (from the pool, see where I'm going with this?) pulls up to the excavator and begins to receive its load.

It's not a perfect setup: there is a finite amount of room on this plateau where the excavation takes place, you can't fit an unlimited number of dumptrucks in there. Sometimes a dumptruck is forced to wait while the excavator fills up another truck. We would not call this problem "embarassingly parallel" but there is definitely a producer-consumer pattern here.

But a similar pattern plays itself out at the bottom of the cliff: loaders scoop up the dumped gravel and deposit it on a conveyor belt, where it is fed into a crusher (from there into another gravel pile, and from there onto a conveyor belt/barge, and from there to the sales facility, where another loader unloads the barge. All told, I think they handle the product 4 times).

The bottleneck here, of course, is the road. At some point as the production capacity up on the plateau expands, the capital and operational expense of widening the road will become less than the cost of handling all that sand and gravel one extra time.

What I find interesting about this problem is that the cost of handing a piece of gravel is infinitesimally small. But when you multiply that cost by several trillion, it adds up to real dollars and cents. And so it is with the Wide Finder. Handling a single byte's worth of data, or a line's worth of data, is so "cheap" we hardly ever think about it. But handling a Gb's worth of data, or 10 million lines worth, now you're talking real money. Because processing time, especially in a batch environment like this, is money.

Meh, some neat parallels there, is all I'm saying.

WideFinder/Java6

2007-11-09T20:11:00.000-08:00

Tim Bray has recently published a series of articles about a project he calls Wide Finder. It pretty much amounts to a multi-threaded text processor for large files. The idea is to implement the requirements in a variety of languages and determine the strengths and weaknesses of each implementation in a highly-parallelized environment.

Here's my stab at it: WideFinder in Java 6. Unfortunately, blogger is a really crappy way to publish this sort of thing, but here goes...

The idea is to treat the log file as a random-access file and use Java's NIO API to memory-map one chunk of it per worker thread. The file is processed on a line-by-line basis, but the chunks are split up based roughly on the file size / number of workers. This means that each worker's "chunk" probably doesn't begin or end on a line break. These edge cases (literally, the edges of the buffer) are resolved after all the worker threads have completed.

The initial implementation is really straight-forward. My first approach used ByteBuffer.asCharBuffer to treat the memory mapped chunk as character data. The problem was that 2 ASCII characters were getting packed into each 16-bit Java char, which meant that all the file data was appearing as CJK characters. It's totally sensible that Java would do this, but I didn't see a quick way around it. I'm going to take a closer look, though, because I think that the current implementation handles the data more than it needs to.

Timing Data

The source file is a 4,000,000 line (178Mb) file where each line matches the regex specified by Tim in the Wide Finder article (see the source for wf.DataGenerator).
Times are given as "number of workers"x"elapsed (wall) time in seconds"
Java VM is 1.6.0
VM arguments are -Xmx1024M -Xms1024M.

AMD Athlon 64 1x2.2Ghz, 2.5Gb RAM, IO is nothing special (SATA something): 1x13.2 2x12.3 4x12.6

Sun T2000 24x1Ghz 8Gb RAM, IO is nothing special (stock 80Gb): 1x142.9 2x53.0 4x28.2 24x9.7

Intel Xeon 4x2.8Ghz, 3Gb RAM, some kind of SCSI I/O: 1x14.6 2x8.67 4x8.49

My 2Ghz Core 2 Duo MacBook had to be excluded from this test because the code is currently dependent on Java 6... very frustrating.

Instructions to Run

Once the code has been compiled, the command is:

java -cp <output directory> wf.WideFinder <log file> <regex> [num-workers]

Where "output directory" is where the classes were compiled to, "log file" is the path to the log file, "regex" is the regex to search for. In this case, regex=='GET /ongoing/When/\d\d\dx/(\d\d\d\d/\d\d/\d\d/[^ .]+)'. If "num-workers" is not specified, the value returned by Runtime.getRuntime().availableProcessors() is used.

Next on the to-do list

Figure out how to use the NIO CharBuffer with US-ASCII encoding... or at least something that lets me deal with the characters as something other than CJK chars. I confess that I'm frightfully ignorant of Unicode and character encodings. About once a year I think, "I gotta learn that stuff..." so I read up on it, and I get to the point where I think I understand it, but I never seem to apply it in my day job, so it's quickly forgotten.
Run this bad boy through DTrace and see where the hot spots are... just a guess, I reckon Pattern.matches is burning up most of the CPU time.
Some low-hanging fruit is probably to do a plaintext match on the initial, constant portion of the regex... ideally take advantage of the 64-bit architecture and compare the first 8 characters of the line all at once!
Oh, yeah, also I should get around to ensuring that the program is actually correct... it's definitely at the point where it's useful to gather performance data but I think there may be a couple corner cases that it'd puke on.
Back-port to Java 5 so I can develop/test on my MacBook (low priority)

... Updated 11/9 21:17, removed inline source, Blogger was having a hissy fit about it.

WideFinder.java

DataGenerator.java

... Updated 11/9 21:41, just realized that to run this against actual log data, you'd probably have to change m.matches to m.find at line 76 of WideFinder.java. That will skew the performance numbers, you may be able to optimize the regex by putting a caret at the beginning.

Comparing Java Performance on Multi-Core CPUs

2007-07-27T22:10:00.000-07:00

The tests in this article measure fixed-point Java performance on a variety of CPU architectures. In summary: Java can take advantage of multiple cores to avoid CPU contention, but in some cases not as well as you'd expect.
I tested on 4 hardware configurations:

Athlon 64 3500+: 1 CPU, 1 Core, 2.2Ghz. Running Linux kernel version 2.6.13, Java 1.6.0_02. This was used as a baseline.

MacBook Core 2 Duo: 1 CPU, 2 Cores, 2.2Ghz. Running OS X kernel version 8.10.1, Java 1.6.0 (b88)

Dual Opteron 248: 2 CPUs, 1 Core each, 2.2Ghz/core. Running Linux kernel version 2.6.13, Java 1.6.0_02

Sun T2000: 1 CPU, 6 cores, h/w support for 4 threads per core, Running SunOS kernel version 5.10

This is the most trivial test class I could come up with. It's actually more complex than I thought it would be. All it does is synchronize 1 or more threads to calculate a lot of large prime numbers at the same time. This task is designed to provide high CPU contention and low IO/memory contention.


package ThreadTester;

import java.math.BigInteger;
import java.util.concurrent.Executors;
import java.util.concurrent.CyclicBarrier;
import java.util.concurrent.ExecutorService;

public class ThreadTester
{
  private static final long START = Long.MAX_VALUE;
  private static final int NUM_PRIMES = 4000;

  private final CyclicBarrier _start;
  private final CyclicBarrier _finish;
  private final int _numThreads;

  public ThreadTester( int numThreads )
  {
    _numThreads = numThreads;
    _start = new CyclicBarrier( _numThreads, new Runnable()
    {
      public void run()
      {
        System.out.print( _numThreads + " " + System.currentTimeMillis() + " " );
      }
    });
    _finish = new CyclicBarrier( _numThreads, new Runnable()
    {
      public void run()
      {
        System.out.println( System.currentTimeMillis() );
      }
    });
  }
  public static void main( String[] args )
  {
    int numThreads = Integer.parseInt( args[ 0 ] );
    new ThreadTester(numThreads).go();
  }

  private void go()
  {
    for ( int i = 0; i < _numThreads; i++ )
    {
      new Thread( new PrimeFinder() ).start();
    }
  }

  private class PrimeFinder implements Runnable
  {
    private BigInteger _bigNum = BigInteger.valueOf( START );

    public void run()
    {
      try
      {
        _start.await();
        for ( int i = 0; i < NUM_PRIMES; i++ )
        {
          _bigNum = _bigNum.nextProbablePrime();
        }
      }
      catch ( Exception ignore ) {}
      finally
      {
        try
        {
          _finish.await();
        }
        catch ( Exception ignore ) {};
      }
    }
  }
}

The raw data follows. The timing data from different architectures should not be compared to each other. In each graph, as the Y value (Time in Seconds) begins to grow linearly with X (Number of Threads), CPU contention among the threads is increasing.
The Sun T2000 server clearly exhibits the best thread utilization. This is unsurprising given the number of independent execution units available in the Niagara processor. Note that had this test involved floating-point math, contention for the Niagara's single FPU among its 24 execution units would be intense.


Athlon-64     
1  6.81  1185600041009 1185600047814
2  13.58 1185600047914 1185600061492
3  20.3  1185600061558 1185600081853
4  26.9  1185600081925 1185600108820
5  33.34 1185600108906 1185600142247
6  40.13 1185600142336 1185600182463
7  46.84 1185600182546 1185600229390
8  54.85 1185600229477 1185600284327
9  60.45 1185600284394 1185600344845
10 65.59 1185600344954 1185600410545
      
MacBook     
1  7.05  1185596894768 1185596901822
2  11.34 1185596902131 1185596913471
3  17.89 1185596914281 1185596932169
4  25.19 1185596932657 1185596957846
5  31.64 1185596958512 1185596990148
6  37.19 1185596990764 1185597027957
7  42.94 1185597028423 1185597071366
8  48.03 1185597071971 1185597120005
9  54.92 1185597120659 1185597175582
10 59.95 1185597175905 1185597235859
      
248     
1  6.16  1185598190089 1185598196249
2  8.32  1185598196398 1185598204713
3  15.57 1185598204849 1185598220423
4  21.08 1185598220564 1185598241640
5  25.96 1185598241766 1185598267726
6  30.86 1185598267868 1185598298726
7  35.15 1185598298870 1185598334015
8  41.11 1185598334145 1185598375253
9  45.52 1185598375373 1185598420891
10 53.08 1185598421038 1185598474117

T2K     
1  31.2  1185597917415 1185597948610
2  31.47 1185597949296 1185597980763
3  32.47 1185597981545 1185598014017
4  34    1185598014867 1185598048869
5  32.84 1185598049745 1185598082581
6  34.11 1185598083555 1185598117668
7  35.38 1185598118647 1185598154031
8  37.43 1185598154995 1185598192425
9  38.97 1185598193408 1185598232373
10 40.06 1185598233337 1185598273396

SuSE 10.0: MP3 Support in K3B for Beginners

2006-06-16T12:53:00.000-07:00

Hey! In the words of the immortal Jim Anchower, I know it's been a while since I rapped at ya. This entry is going to take a bit of a detour from the Media Center setup. Instead I'll talk about building functionality into your Linux OS based on a combination of package management and good-old-fashioned building from open source.

This article is targeted more towards open-source beginners... if you've been following along with the rest of the stuff on this site, you've probably already progressed beyond this point. In any event, I welcome your comments.

SuSE Linux, by default, does not ship with MP3 support for many of its applications. I'm not sure if this is because they would be forced to license such technology from the patent-holders of the MP3 encoding algorithm, or if the RIAA prevents it somehow. Anyways, it's a pain in the ass. For instance, K3B, the popular CD-burning software for the K Desktop Environment (KDE), is pretty much crippled for making audio CDs. Fortunately, it's pretty simple to rebuild K3B and include MP3 support.

I shouldn't say simple. The process itself is actually fairly complex, with dozens of different software products interacting with each other. But the trick is to use the right tools to make it simple.

Every Linux distribution these days, as far as I know, comes with some sort of package management software. This tool's job is to resolve all these interactions and interdependencies between various software packages. In SuSE, the distro I'll be covering here, this tool is called YAST. In Debian or a debian-based distro such as Ubuntu, it's called apt-get. In GenToo I believe it's called emerge. Point is, whatever distribution you've chosen, there will almost certainly be a tool available to help you with this process.

I should say right up front: if your distro hasn't got one, or you can't find it, stop now. In the time it takes you to resolve all the dependencies for building K3B you can install Ubuntu and never have to worry about this again.

SuSE 10.0 Specific Note

You can make these package installs go a lot faster if you (a) have a copy of the original installation media and (b) have some hard drive space to spare. What I did was create a directory, /suse10dvd/CD1, and copied the entire file/directory structure from my SuSE DVD into that directory. Then, in YAST, you create a new installation source from the directory /suse10dvd (it will automatically look in CD1 for all the requisite files).

Setup the development tools

Eventually, all the package management in the world is only going to get us so far. When we reach that point we're going to need to take source code and turn it into a binary executable. And to do that we will need some tools. Fortunately, these tools can be installed by - you guessed it - the package management system. Again, I will be assuming you're using SuSE 10.0 here. As root, fire up YAST, and click on Software Management (might as well keep that screen open for a while).

Install the following packages: gcc g++ make automake autoconf. Accept any dependencies YAST points out.

Setup K3B source prerequisites

All these packages are also installable via YAST. Go ahead and install them now: xorg-x11-devel zlib-devel qt-devel libjpeg-devel kdebase3-devel taglib-devel libmusicbrainz-devel

If you notice that you're installing one of these "devel" packages, but the corresponding package is not selected (ie. libjpeg for libjpeg-devel), go ahead and select that too. That's a lot of stuff, and it'll probably take a while to install. While it's working you can download some of the stuff we'll need that SuSE can't provide.

Getting the source prerequisites

You'll need to get the source code for libmad. This provides K3B with the ability to decode MPEG audio (ie. what we commonly refer to as an mp3 file). Get it here. Use SourceForge or their FTP site, doesn't matter. You don't need the id3tag or madplay stuff, just the libmad download.

You can build a more complete K3B by downloading LAME from lame.sourceforge.net. LAME (Lame Ain't an MP3 Encoder) allows K3B and other tools to encode audio into MP3 format. This isn't necessary for what we want to do, but it will add valuable functionality to your SuSE installation.

These source packages come in a .tar.gz format - commonly known as a tarball. Once you've downloaded them to a convenient spot on your hard disk you can extract them with the "tar xvfz <source package>.tar.gz" command.

Extracting one of these tarballs will create a directory holding the source code, ie. lame-3.97. Once you're in this directory you can build the executable from the source code using these three commands - they're the same for pretty much any open source software package:

./configure

make

(as root) make install

The first command ensures that all the necessary dependencies are available and sets up the eventual install paths. The second command actually takes the source code and turns it into binary files - executable files, code libraries, etc. The third command takes those output files and puts them in the proper locations so that other programs (ie. k3b) can find them.

I should note that the configure script takes a lot of options. You can see some of them by running ./configure --help. Unless you see something that really jumps out at you, you should accept the defaults for this project.

Make sure that you run the last command, make install, as either the root user or someone with root-like permissions. Otherwise you probably won't be able to put the generated files in the default spots.

Building K3B

Is basically a non-event. Download the K3B source tarball from the K3B homepage. Extract it just like you did with the prerequisites (note that if the file ends in a .tar.bz2 suffix, you should use the command tar xvfj <filename>.tar.bz2 rather than tar xvfz. Go into the newly-created directory and run the same ./configure,make,make install commands as above.

Note that when you run the configure script, it will show you which extra features will be built into K3B. You should see something like this:


K3b Configure results:
------------------------------------------
Ogg Vorbis support:                    yes

Mp3 decoding support (libmad):         yes

Audio meta data reading with Taglib:   yes

libsndfile audio decoding support:     yes

FLAC support:                           no
    You are missing the FLAC++ headers and libraries.
    The FLAC decoding plugin won't be compiled.

Musepack support:                       no
    You are missing the Musepack headers and libraries >= 1.1.
    The Musepack audio decoding plugin won't be compiled.

Lame Mp3 encoder plugin:               yes

Audio resampling:
            using version bundled with K3b

FFMpeg decoder plugin (decodes wma and others):
                                        no
    You are missing the ffmpeg headers and libraries
    version 0.4.9 or higher.
    The ffmpeg audio decoding plugin (decodes wma and
    others) won't be compiled.

Resmgr support:                        yes

Audioplayer available (aRts)           yes

Compile K3bSetup 2:                    yes

Tag guessing using MusicBrainz          no
    You are missing the musicbrainz headers and libraries.
    K3b will be compiled without support for tag guessing.

Compile HAL support                     no
    You are missing the HAL >= 0.4 headers and libraries
    or the DBus Qt bindings.


Good - your configure finished. Start make now

Your new version of K3B will by default be installed right overtop of the old version. This is convenient, but you must remember now that the package manager (YAST) doesn't know that you've swapped in this new version, so any updates that SuSE may try to apply will probably blow away your changes. Careful!

If you'd rather put K3B in a different place, use the --prefix argument to configure. Again, configure --help has more information.

That should be it! If you can drag MP3 files into your Audio CD project, you'll know everything's working. Double check via the Help Menu that the version of K3B that you're running corresponds to the version that you just built - you may be running the old executable.

If you have any trouble, Google is your friend. If you're still in trouble, my contact information is on the sidebar to the right of this page.

Good luck!

Day of Disappointment

2006-03-28T10:08:00.000-08:00

What a bummer. I was just about ready to wrap this whole thing up when all of a sudden the LCD display stopped working. It wouldn't even POST, as soon as I hooked up the power it just showed some garbage on the screen. My first assumption was that I need a new display - no big deal, the eBay seller has lots more of them and he was a pleasure to deal with. So I dropped another $15 bucks on a new display. Then I tried to investigate the problem a little bit more and found that the parallel port on my motherboard seems to have crapped out. The values that I write to the SELECT_IN, INIT, AUTO_FD, and STROBE are all reset after about 125Ms, whereas before they would persist until I changed them. This definitely isn't good. They also all reset themselves to low (which in the case of select, auto feed, and strobe is actually +5v), which is kind of weird.

Anyways, Logic Supply has been great to me so far, so I'm hoping that they'll process this as a warranty claim without any hiccups.

Removing the Hard Disk

2006-03-20T15:02:00.000-08:00

One of the last steps in this project is to switch the operating system from the hard disk to a compact flash card.

The CF card I'm using is a 1Gb SanDisk Ultra II. I also have a CFDisk CF-IDE adaptor so I can plug the card into a regular IDE channel.

I thought it would be a simple matter of booting into a separate Debian install (I still have one on it's own partition from my first attempt at all this) and copying the entire file system from the hard disk to the flash card. Unfortunately, I was unable to write to the Ultra II card while it was in the IDE adaptor. Actually, I was able to correctly write about 999,999 bytes out of every 1,000,000, but that just ain't good enough. This turned out to be a very tricky problem to track down. Basically I wound up having to recursively diff each file in the source and the destination to determine that there were some bytes that got mangled during the copy. Also, running /sbin/badblocks -t random -w /dev/hdb indicated that certain writes would just randomly fail. [Note that the above command will destroy all existing data on your hard disk]

The solution was to insert the CF card into a standard USB card reader, the kind you might have to go with your digital camera. Debian automatically recognized the device and mounted it as /dev/sdb. It was all much more convenient than I imagined it would be. In the end, I guess the write speed through USB was slow enough not to cause a problem. Once the filesystem was copied over, and I was ready to switch to a read-only file system, I reconnected the card to the IDE adaptor, because reading from the card works fine.

I followed the instructions that I described here to get the ramdisk up and running, and then borrowed a couple points from here: notably, creating a symbolic link from /proc/mounts to /etc/mtab and editing /etc/fstab to include the "ro" option for the root filesystem.

That's all. Because these disk writes are (infinitesimally) unreliable and it's a real pain in the ass to switch between the hard disk and the cf-card, I'm going to go back to the hard disk for now and then try this whole process one more time when I am sure that I'm 100% finished.

Tying it all together

2006-03-14T09:14:00.000-08:00

Let's recap all the components of the Media Center so far:

Xine video player + Unichrome XXMC driver for hardware playback

IR receiver + LIRC daemon with Xine integration for wireless control

4-line LCD display + LCDProc daemon to show status information

Backed-up DVD media available via NFS mount to backfire

Pretty much all that's left is to wire all these things together in software. The requirements for that piece of software are:

Browse through the available media files

Play selected media file

Update the LCD display with title of current media file, pause/play/stop status, and position in stream

The 'browse' and 'select' actions must be executable via the remote control

That's a pretty rough set of requirements, but it'll do for now. It took me a couple days, but I came up with a bit of software that meets these pretty nicely. It's about 1500 lines of code spead out over a dozen or so Java classes. I'm calling it 'mediabrowser' for now and you can download it here. You'll need JDK 1.5 to build it, or JRE 1.5 to run it. Both can be obtained from java.sun.com.

Building Mediabrowser

You'll need a recent version of Apache Ant to do the build. Just run ant clean dist. The file build/dist/mediabrowser.tar.gz will be created. Extract that to your installation directory and you're ready to go.

Configuring Mediabrowser

Running Mediabrowser is fairly straightforward. Before you get started with it, ensure that the LIRC and LCDd daemons are running. Make sure that you started lircd with the -l option so it can provide the IR data over a TCP/IP port. Then follow these steps to adjust the Mediabrowser to suit your configuration:

Edit conf/xineloader.properties. Verify that xl.cmd.xine matches your Xine command.
Editmb.properties. Adjust mb.media.root to match the root directory for your media collection. If you are using a remote control, change the property names for the ir.btnmap.[...] properties to match the button names in your LIRC configuration file. That will map each button to a Java KeyEvent.
Finally, edit the sample index file index.xml and place it in the root media directory that you specified above. This is what the Mediabrowser will inspect to get information about your collection.

Running Mediabrowser

Running the Mediabrowser is as simple as java -cp conf:bin ca.razorwire.mb.gui.MediaBrowser. A full-screen window should appear, displaying the entries in your index.xml file. You can navigate the screen and select an entry using the buttons on your remote that you defined in the mb.properties file.

How It Works

Briefly: When the GUI launches, it starts a thread to monitor for LIRC output on the default LIRC port, 8765. When it receives LIRC data that it recognizes it generates a corresponding Java AWT event indicating that a key has been typed (this is what the btnmap and keymap properties in mb.properties are for). This event is sent to a hidden UI component in the GUI that always has the keyboard input focus.

As far as this UI component is concerned, a key has just been pressed. You can see this yourself by pressing any of the keys defined in the properties file. The behaviour should be exactly the same as if you'd pressed a button on the remote. It then maps the keypress to a UI action - either scrolling through the title list, scrolling the current title's description summary, or playing the title.

When you start playing a title, the GUI launches a new thread to start the Xine application as a separate process. It also stops processing LIRC output, because we don't to process any button-presses that were meant only for Xine. If the LCD screen is enabled, this thread keeps a reference to the LCD driver and the title of the current track. It monitors our hacked-up Xine output to catch the stream position updates, and prints out the track title and current position to the LCD screen. When the Xine application completes, the thread dies and control is returned to the GUI.

TODO:

Need some screenshots!

Need a sexier GUI! Right now it's just the standard Swing look and feel. Perfectly functional, but not that attractive compared to most Media Center offerings. Maybe whip something up in OpenGL to take advantage of the hardware acceleration?

MythTV integration... haven't got a capture card yet so I'm still quite a ways behind on the Myth scene.

Tweak to DVDBackup

2006-03-10T09:22:00.000-08:00

Just a quick aside here to talk about a software component that will be going on backfire, the media server. There's an incredibly useful utility floating around the net called DVDBackup. As far as I can tell there is no official site for it but some useful information about it can be found here.

Version 0.1 has a small bug in it that caused a segfault while I was backing up the Firefly TV series. A patched version is available here. The only difference is moving the call to ifoClose in DVDGetInfo to line 852.

Extracting command activity from Xine

2006-03-08T14:03:00.000-08:00

In this post I described how to tweak the Xine source code to print the current stream position in a friendly format. Printing out the command activity (ie. 'pause', 'play', 'stop', etc.) is similarly easy. In the file src/xitk/event.c, just add a line in the function gui_execute_action_id that looks like this: fprintf( stdout, "action\t%d\n", action); fflush( stdout );. Then run make debug && make install-debug to deploy the changes.

LCD Status Display

2006-03-08T12:32:00.000-08:00

One neat feature I'd like to incorporate into this media center is the ability to display the unit's status off-screen. By that, I mean the same sort of functionality that your DVD or CD player gives you: a panel to show the current position in the track, the play/pause/stop status, etc.

I started by ordering a 20-character, 4-line display from "rainbow_city_store" on eBay. The service was prompt and wholly satisfactory. This display is compatible with the popular HD44780 LCD controller, and can basically be plugged straight into a parallel port once you've got the wiring all sorted out. The seller provided a couple of schematic diagrams, and also some helpful links to more wiring diagrams and various HD44780 projects. I wired up the display to the parallel port as described here (I didn't use the port cups, but the schematic was great).

(ignore the extra notes for now)

Once the display was wired up I investigated a project called LCDProc. This is a useful little tool that can display things like CPU, Memory, and Disk usage on an external display. But what's more important is the component of LCDProc that can drive the HD44780 over the parallel port (LCDd). This really simplifies the interface to the display by providing simple commands that can be executed via a regular TCP/IP socket.

Here are some notes on establishing communication with the display:

At the hardware level, you can check that your signals are correct by using a multimeter to measure the voltage between certain pins and the ground. The SP800 sets the data pins to 3.3V (high) or 0.5V (low). The other pins will be between 1.5V and 5V high or 0.5V low, I believe.

If you have applied power to Pin 2 on the LCD and grounded pins 1 and 3, you should see lines 2 and 4 on the display illuminate (barely). If you apply power to pin 15 and ground pin 16 (make sure you get that right) you will activate the backlight, and the lines should be much more visible. This is the HD44780 self-test. You may be stuck here a while :)

Pin 1 on the parallel port (pin 6 on the display) is the Strobe. If you are interfacing directly with the hardware (ie. not using LCDd), be sure to set this to high, and then to low to tell the LCD controller that the signal on the data pins are valid.

Access to the data pins of the parallel port on the SP800 is at IO 0x378. You can access the status pins at 0x38A. More info here. Some sample code is here. Compile like so: gcc -o lptout lptout.c. Just tweak the outb call to direct the byte to the data or status pins.

If you are using LCDproc/LCDd and have wired your display like mine, you want the HD44780 driver and the "8bit" wiring configuration.

Build and install LCDd like so:
- # Download and extract the tarball to /usr/local/src cd lcdproc-0.4.5 ./configure --enable-drivers=hd44780 make && make install
- Edit LCDd.conf. Uncomment the line "Driver=HD44780" and comment out any other drivers. In the [HD44780] section, ensure the Port parameter is set to 0x378 and the ConnectionType parameter is set to 8bit. The size should also be set to 20x4. Finally, copy this file to /etc.
- Start LCDd by running LCDd /etc/LCDd.conf. It runs in the foreground by default. This is configurable within the conf file. If LCDd was able to connect and interface with your display, the screen should change from the two active lines (self-test screen) to something like this:LCDProc Server Clients: 0 Screens: 0
  (The display pinout on my LCD screen is actually at the top of the display, so this screen currently appears upside-down. If you know of a way to rotate the display via the controller I would love to know.)
- Test the connection by opening a new terminal window and running lcdproc C. This will try to connect to LCDd on the default address and port and display your CPU usage statistics on the LCD screen. If it works, congratulations!

Extra Credit

That backlight ain't gonna last forever, you know. It would be convenient if we could turn it on and off at will. A simple potentiometer or switch between pin 16 and ground would do the trick. Or... we could wire it up to one of the unused status pins on the parallel port and control it by a transistor... oooh! Good idea!

Turns out, I wasn't the first one to think of it. But I did it slightly differently than described in the (excellent) LCDproc documentation. Here's the schematic:

The transistor is a 2N3904, which we happened to have lying around the office. The collector goes to pin 16 on the LCD, the emitter goes to ground, and the base to pin 17 of the parallel port via a 1K resistor. If you're interested, here's the math, as I understand it:

By hooking up an ammeter in series with the LCD display pin 15 we can see that the backlight draws ~51mA of current.

The two figures we need from the 2N3904 data sheet are the gain (h_fe) and the base/emitter saturation V_BE(sat), which is 0.65 ~= 0.7 for a current draw of 0.051A.

The high voltage on parallel port pin 17 (SEL) will be 1.5V. We subtract the saturation voltage from this to get 0.8V.

We calculate the required resistance for the transistor like so: 0.8V / ( 0.051A / 60 ). This gives us a resistance of 941 Ohms, ~= 1K.

LCDd has some backlight functionality built-in, but I had to tweak it a little bit to get it to work with this setup. There appears to be some conflict with pin 17 and the "multiple display" feature of LCDproc - when I started LCDd, the backlight would blink on and off repeatedly. The easiest way to stop this, if you're just using one display like me, is to force the displayID to 0 in the function lcdwinamp_HD44780_senddata (server/drivers/hd44780-winamp.c). I also had to swap the values for BacklightBrightness and BacklightOffBrightness in LCDd.conf. Then I just set the Backlight parameter in the HD44780 section of the conf file to "yes".

After the code change, run make && make install to deploy the changes. Restart LCDd. You can test the backlight by telnetting to localhost 13666 and entering the following commands:


hello
screen_add 0
backlight on
backlight off

Note that the backlight command has no effect until at least one screen has been created.
Here's a shot of the LCD display in action. If you look closely, you can see that it's displaying the timing data that we just took out of Xine. More on that in the next post.

Stream Position Data in Xine

2006-03-06T14:48:00.000-08:00

Now that Xine is up and running, it would be handy to be able to extract the stream position data from the application and display it elsewhere (on a status display, for instance). This way, we could see how much of the movie we've played, how much time remains, etc.

This is pretty easy, once you figure out where to look. In the xine-ui source, go into src/xitk/stdctl.c. In the function xine_stdctl_loop, change the line:
int secs, last_secs; to int secs, last_secs, len_secs, stream_pos;

Just below that, you should see where last_secs is initialized to 1. Copy that line twice and replace last_secs with len_secs and stream_pos.

Near the end of that function, replace the call to gui_xine_get_pos_length(...) with gui_xine_get_pos_length( gGui->stream, &stream_pos, &secs, &len_secs ). Divide len_secs by 1000 afterwards, as is already done with secs.

Replace the call to fprintf with this: fprintf(stdout, "time\t%d\t%d\n", secs, len_secs).

Run make && make install to deploy your changes.

Finally, Xine suppresses output to standard out (stdout) shortly after it prints its introductory banner/copyright data. The easiest way to prevent this is by prepending the --verbose option to the command line. A slightly cleaner way (only cleaner in terms of the output) is to run make clean && make debug && make install_debug to deploy your changes instead. If everything's working right, you should see a bunch of lines like this displayed on the screen:

time 0 97
time 1 97
time 2 97

The first number is the current position in seconds in the current stream and the second is the length of the current stream, in seconds.

Trimming the disk usage

2006-02-27T12:26:00.000-08:00

Here are some areas where we can probably trim some fat from the hard disk:

/usr/share/doc - 20M
/usr/share/man - 6.7M
/usr/local/bin/via - 9.7M (this was where I downloaded the VIA DRI binaries to, but they've been installed so I can remove them
/usr/X11R6/man - 12M
/usr/X11R6/lib/X11/doc - 14M
/usr/X11R6/lib/modules/dri - 24M (but obviously not the VIA driver)
/lib/modules/2.4xxx/* - 29M

Also removing binutils bison flex g++ g++-3.3 gcc gcc-3.3 gcc-3.3-base kernel-source-2.6.8 linux-kernel-headers make through apt-get will free another 60Mb or so. Best to do this before deleting the above files, in case any dependencies get mucked up.

This will be entirely offset by the installation of the Java 1.5 Runtime Environment, which I'm considering installing... that will weigh in at 91Mb. Ouch. Maybe need to rethink that.

Getting Video

2006-02-25T22:49:00.000-08:00

Just some notes on building and installing the requisite tools for DVD playback on the Via Epia SP800 (CN400 Chipset)... will clean up as time allows... see the sidebar for contact info if you have questions.

Building Xine:
I grabbed Debian's ALSA (Advanced Linux Sound Architecture) user-space packages first since I enabled ALSA support when I rebuilt the kernel... apt-get install libasound2 libasound2-dev. Configuring ALSA is as simple as running alsaconf and selecting the VIA sound driver.

Download source tarballs for xine-lib and xine-ui from http://xinehq.de
For xine-lib:
./configure --prefix=/usr
make && make install

For xine-ui:
./configure --prefix=/usr --x-includes=/usr/X11R6/include --x-libraries=/usr/X11R6/lib
I needed to patch the file src/xitk/oxine/Makefile. It does not include the X11 headers as specified in the ./configure command line. So just search for AM_CPPFLAGS = and add -I/usr/X11R6/include to the end of the line. Then,
make && make install

The last piece of the puzzle is to invoke xine and tell it to use the drivers we have just set up: ALSA for audio, and the Via XVMC driver for video. Let's say that you've backed up the Serenity DVD to /media/movies/SERENITY.iso. The command would then be:

xine -V xxmc -A alsa dvd:/media/movies/SERENITY.iso

Assuming that you have removed the CSS encryption from the DVD. That will be covered in the next lesson.

Note that we specify the "xxmc" video driver. Running xine --help shows that the more intuitively-named driver "xvmc" is available, but specifying that on the command line crashes xine. Using "xxmc" works, and we can tell that it's using the onboard MPEG accelerator because running top while watching the movie shows CPU utilization at around 3%. Contrast that to the un-accelerated driver, where CPU utilization will be around 20-30%.

Building and Configuring LIRC

2006-02-25T14:56:00.000-08:00

Building LIRC is actually pretty simple. Download the source package from lirc.org and extract it to /usr/local/src. As indicated in the INSTALL file in that package, compiling it is as simple as running setup.sh && make install. But you'll need the "dialog" package to run the setup script, so apt-get install dialog first. It's a small package and you can remove it as soon as you're done.

When I ran the setup script it correctly detected my port configuration (IO and IRQ). Note that the onboard serial header is COM2, known to Linux as /dev/ttyS1. The default parameters for this port on the SP800 are 2F8 and IRQ3.

The configure script expects the kernel source to be installed in /usr/src. I had actually moved it to a separate partition to free up some room, so I just mounted the partition and made a symbolic link to the source. The source should be in a directory like kernel-source-2.6.8.

Save the configuration, run ./configure, and then make && make install. If you encounter the error "LIRC modules currently require module unloading...", go back into the kernel source, run "make menuconfig" and select "Loadable Module Support -> Module Unloading" from the menu options. The run make && make install in the kernel source directory, reboot, and try to build LIRC again.

Once LIRC is built you should follow the instructions at LIRC.org to get the driver up and running. The quickest way to test the input is like this:

Tell the kernel to release the serial port by running setserial /dev/ttyS1 uart none

Insert the LIRC serial module into the kernel: modprobe -v lirc_serial

Run mode2. This program should display no output until you start pressing buttons on your remote control. If it prints values while no buttons are being pressed, you probably have an interference problem.

If you get errors complaining that the device is busy, make sure that you've a) enabled COM2 in the VIA BIOS, b) configured LIRC's setup script with the correct values for COM2, and c) referred to the device /dev/ttyS1. I banged my head against the wall for half a day because I didn't twig to the zero-based port numbering.

Then it's just a matter of using irrecord to generate a .conf file, and starting lircd with that .conf file. Those steps are all very well documented at lirc.org

IR Remote Control and LIRC

2006-02-25T13:56:00.000-08:00

LIRC is a Linux utility that allows input from many common types of remote controls. It integrates very nicely with XMMS, the media player I've chosen to install on Fulcrum.

The LIRC website has a bunch of instructions on how to build your own homebrew IR receiver that hooks up to a standard serial port. From what I understand, it's not "true" serial communication as all the data is transferred over one of the handshake lines, but everything seems to work out OK.

The schematic that I used is here. I ordered all the parts from Digikey, except the breadboard, which I got from a local electronics store. And the case, which is an Altoids tin.

Here's the Digikey parts list:

Digikey Part #	Description
PNA4602M-ND	Infrared photoreciver + IC
AN78L05-ND	Voltage Regulator
P974-ND	Capacitor - 4.7 uf 25V
P4.7KBACT-ND	4.7 kilo-ohm resistor
C3APT-1018M-ND	IDC Cable
1N4148DICT-ND	Diode

Some notes:

Digikey charges a substantial service fee for orders under $35 (CAD). So if there's anything else you need, tack in onto the order. Pick up a few of the voltage regulators and a couple of the IR receivers in case anything goes sideways. Digikey also sells several similar receivers (PNA4612M00YB-ND, PNA4612M00XD-ND) with metal holders, which may help reduce interference.

I can't say for sure that the diode I've listed above is correct. I made a mistake and got the surface-mount casing; but I checked the catalog and that looks like the right one.

The pinouts on the IR receiver (substitute for the TSOP1738 in the schematic) and the voltage regulator are different than the ones described on the LIRC homebrew instruction page. As that page suggests, double-check that you've got them wired correctly.

One end of the IDC cable I specified above is a perfect fit for the serial header on the SP800 motherboard. The other end can be munged a little bit to fit into a 0.1" printed circuitboard on an angle. Since I had to visit the local electronics shop anyways, I picked up a proper IDC 16 pin crimp connector and used that instead.

Total costs were $33.60 to Digikey + ~= $10.00 to the local electronics shop. Definitely more practical to pick up a Windows Media Centre remote + USB receiver (which I believe is compatible with LIRC), but not nearly as much fun.

Assembling the circuit is actually really easy, and I say that as someone with very little experience in electronics. I'd suggest wiring it up on a solderless breadboard first, especially if you are as unfamiliar with this sort of thing as I was. Once you're comfortable with how everything hooks together, it's then just a challenge of laying it out on the breadboard with as little soldering as possible.

This is pretty much how I laid it out on the final breadboard, although I stretched it out a little bit to give myself some more room to work:

This is a picture of the final product with the case open:

I added the case (a butchered Altoids Mints container) because I was trying to resolve some interference problems. These problems only arose while I was working on this at the office, not at home. The case didn't actually resolve the problem, but I'm not too worried about it.

Finally, an up-to-date picture of Fulcrum. The Altoids case is closed in the picture. Notice that I have attached a proper momentary contact switch, so I don't have to short the jumpers with the needlenose pliers. That was fun for a while but got tiring. Notice also that the ghetto piece of plywood has been replaced with a ghetto piece of 1/16" sheet steel. However, that piece of steel was taken from a damaged Cray XD1, so I think I score some geek points for that.

That covers the hardware. Next up, building and configuring the software.

Building up from Minimal Debian

2006-02-22T14:47:00.000-08:00

I'm a sucker for punishment, so I've decided to put all my progress on hold for a while and try to rebuild the system from (almost) scratch. The OS was getting a little bloated, and I'd like to keep a tighter rein on what's getting installed. This post is a compilation of several earlier posts, with some extra details added here and there.

Note that you can skip steps 1 and 2 by burning the Debian businesscard ISO to a CD and configuring your BIOS to boot from CD.

Note also that I've kept my previous installation on a different partition. If this doesn't work out, I can switch back with a simple change to the GRUB boot menu.

Finally... building xorg (the X Windows Server) from scratch is really a post unto itself... the documentation for version 6.9.0 is available here. If you still have questions I'll do my best to help.

Format a fresh partition with the ext3 file system.

Follow the instructions here to load the debian installer. Briefly,
mkdir /boot/debian-installer cp businesscard.iso /boot cd debian-installer wget http://ftp.debian.org/debian/dists/stable/main/installer-i386/current/images/hd-media/initrd.gz wget http://ftp.debian.org/debian/dists/stable/main/installer-i386/current/images/hd-media/vmlinuz edit /boot/menu/grub.lst, add new entry. Recall that the partition numbering is 0-based. root(0,0) kernel /boot/debian-installer/vmlinuz ramdisk_size=12000 root=/dev/ram0 initrd /boot/debian-installer/initrd.gz

Activate the installer by rebooting.

accept the defaults up to the screen where you are asked about disk partitioning. Choose to manually edit the partition table. Choose the partition you were just in as the install target. Do not format the partition if you are booting from the hard disk. Specify "/" as the mount point.

Reboot. The installation proceeds. When you are asked about the system's "profile", ie. webserver, desktop, etc. select the packages manually. This launches the aptitude package manager frontend.

Select "kernel-source-2.6.8" "libncurses5-dev" "module-init-tools" "flex" "bison" "g++" "libpam-dev". You can use the "/" key to open a search box and the "n" key to advance through the search hits. Use the "+" key to select packages for installation. Installing these packages will install a number of ancillary developer tools.

Quit the installer

Build a more appropriate kernel as described here
cd /usr/src tar xvfj kernel-source-2.6.8.tar.bz2 && rm kernel-source-2.6.8.tar.bz2 cd kernel-source-2.6.8 make menuconfig

In the kernel config GUI, select "Processor Type and Features" > "Processor Family" > "VIA C3-2 (Nehemiah)"

Optionally disable drivers to get faster compile time: ISDN, Telephony, all graphics drivers except VGA and VESA

Enabled ALSA (and all its sub-items, and the VIA drivers), Parallel Port Support, PC-Style Hardware

Exit, save, run make all install modules_install Wait a long, long time.

Run mkinitrd -o /boot/initrd-2.6.8 2.6.8

Add new entry to /boot/grub/menu.lst:title Debian 2.6.8 root (hd0,2) kernel /boot/vmlinuz-2.6.8 root=/dev/hda3 ro initrd /boot/initrd-2.6.8 savedefault boot

Get and Build the xorg source (for the X server)
cd /usr/local/src wget http://xorg.freedesktop.org/releases/X11R6.9.0/src-single/X11R6.9.0.tar.bz2 tar xvfj X11R6.9.0.tar.bz2 && rm X11R6.9.0.tar.bz2 cd xc/config/cf cp xorgsite.def host.def # Note: the xorg distribution comes with its dependencies, # but it does a shitty job building them. Also, none of the # executable perms have been set. cd ../../extras/expat chmod a+x configure ./configure --prefix=/usr && make && make install cd ../freetype2 chmod a+x configure builds/unix/configure builds/unix/install-sh builds/unix-mkinstalldirs ./configure --prefix=/usr && make && make install cd ../.. make World | tee World.log 2>&1 make install install.man # Not sure if there's a more correct way to do this cd /usr/bin ln -s /usr/X11R6/bin X11

Download and build the Openchrome Drivers as described here. If you haven't got a SVN/CVS client, you can download one of the snapshot tarballs from openchrome.org. Here I have downloaded the source to /usr/local/src.
cd /usr/local/src # Follow directions in README cd drm/linux-core make LINUXDIR=/usr/src/kernel-source-2.6.8 DRM_MODULES=via cp *.ko /lib/modules/2.6.8/kernel/drivers/char/drm/ depmod -ae export PATH=$PATH:/usr/X11R6/bin xmkmf /usr/local/src/xc make && make install cd ../unichrome xmkmf /usr/local/src/xc

The DRM and Mesa stuff that's packaged with X.org is a piece of shit. To install the DRI drivers I had to download the binaries from here (just get the latest VIA drivers). Extract the tarball and copy unichrome_dri.so to /usr/X11R6/lib/modules/dri. You can follow the directions here to verify that hardware acceleration is being used. On GLXGears, the SP800 should be capable of about 30-60fps un-accelerated and approximately 500fps when hardware acceleration is active.

Download, Build and Install a Window Manager

Obtain the fluxbox development release source from http://fluxbox.sourceforge.net. Don't go with the "stable" 0.1.14 release; it doesn't even compile. ("stable"?!?)

Extract to /usr/local/src; Enter fluxbox directory

./configure --prefix=/usr && make && make install

Edit the file /etc/X11/xinit/xinitrc. Remove the lines after "start some nice programs" and replace with:fluxbox & wmpid=$! wait $wmpid

Start the xserver with startx

Disabling the Ramdisk

2006-02-17T09:56:00.000-08:00

Setting up the ramdisk as I just described was a little premature, because I'm not quite done with all my software installations yet. But at least now I know that it's possible. I can disable the ramdisk until I'm ready with these simple commands:


init 1
umount /var
mv /var /var_rd
mv /var_orig /var
chmod a-x /etc/init.d/setup_ramdisk

That still leaves the ramdisk_size kernel parameter in the GRUB configuration, but since we've disabled the ramdisk creation script that memory won't be consumed, so we're OK. Might as well leave it as-is, it's one less thing to re-enable when the time comes.

Setting up a Ramdisk

2006-02-16T14:56:00.000-08:00

Wow, it's been a while since I've updated. I've been busy building the infrared receiver described here: http://www.lirc.org/receivers.html. I'll post more about that once I've got the last few kinks worked out and had a chance to take some pics of it.

For now, though, I'll describe how I've altered the Linux install to make use of a ramdisk for parts of the filesystem. A ramdisk is a section of memory that the operating system treats like a regular disk. One advantage of using a ramdisk instead of a physical disk is that it provides really, really fast read/write times. A big disadvantage is that anything that you write to a ramdisk gets obliterated when your computer shuts down.

I'm using a ramdisk in this situation because when I switch Fulcrum over to compact flash, I want it to treat the CF card as a read-only file system. But because applications expect to be able to write data to /var and /tmp, we need to provide some write capability. We will "graft" the ramdisk onto the root filesystem, and none of the applications will even know they're writing to a ramdisk.

The linux kernel creates a bunch of ramdisks as soon as it starts, and identifies them as separate devices in the /dev filesystem. Do an ls there and you can see them, labelled from /dev/ram0 to /dev/ram15 or so.

The kernel restricts the size of these ramdisks based on the ramdisk_size kernel parameter. The value of this parameter is measured in kilobytes, so to define a ramdisk of 128Mb you would pass this as a kernel parameter in the GRUB configuration file, menu.lst:
ramdisk_size=131072
(131072 = 128 * 1024)

As far as I know, all ramdisks are created with the same size limit. If you know of a way to independently specify the size of each ramdisk, I would love to hear about it.

Let's deal with ram-ifying /var first. Obviously, the /var directory is full of important files and stuff. We want to:

Backup the whole damn thing
Save these files someplace safe
Empty out the /var directory

Then, on every boot:

Tell Linux to move the files from "someplace safe" to the ramdisk
Mount the ramdisk on /var

Here's a quick way to do the backup, presuming you have some unallocated space on your hard disk. In the example below, my OS is installed on /dev/hda2 and I've created a partition /dev/hda3 that is exactly the same size using fdisk -u. (Creating the backup target partition is pretty easy, it just takes a little adding and subtracting to determine the number of sectors and the end point of the partition)


# Do a byte-by-byte backup of /dev/hda2 to /dev/hda3
dd if=/dev/hda2 of=/dev/hda3 bs=16MB

Lastly, you can go into /boot/grub/menu.lst and clone the entries there to boot your new partition - or, you can just edit the existing entries by hand if you ever need to boot into that partition. Remember to change both the root(n,m) line and the root=/dev/hda(n) part of the GRUB entry.

Now that you've got a backup (hey - maybe test that it works, while you're at it), we can proceed with ram-ifying /var. Follow these commands:


#A flimsy way of "quieting" the system
#to prevent writes to /var while we work
init 1
#relocate the files
mv /var /var_orig
#create the new empty directory
mkdir /var
mkdir /mnt/rd
cd /etc/init.d
#create a script to do the work at every boot
cat setup_ramdisk << EOF
#! /bin/sh
echo "**** Initializing Ramdisk ****"
mkfs -V -t ext2 /dev/ram1
mount /dev/ram1 /mnt/rd
cp -a /var_orig/* /mnt/rd
umount /mnt/rd
mount /dev/ram1 /var
echo "Ramdisk Created"
EOF
# make it executable
chmod a+x setup_ramdisk
cd /etc/rcS.d
# make it run very early it the boot process
ln -s /etc/init.d/setup_ramdisk S03setup_ramdisk

The script that's embedded in those commands is actually what's responsible for most of the heavy-lifting (it's saved to /etc/init.d/setup_ramdisk). What it does is format the ramdisk using the ext2 file system, mount it to a temporary directory, copy all the stuff from the original /var directory, and then move the mount point from the temporary directory to /var.

At this point, double check that you've allowed for a big enough ramdisk in the kernel parameter. Do a df --si /var, grow that number by, say, 20%, and convert that to kilobytes and enter it into /boot/grub/menu.lst as described above.

Reboot, cross your fingers, and hopefully you won't even notice the difference!

If something goes wonky and your system won't boot, select your backed-up partition from the GRUB boot menu. Then mount the original parition and check for any mistakes. The messages during boot will tend to fly past before you can read them, so it may be very helpful to include a "sleep 60" command just after the "Ramdisk Created" line in the setup_ramdisk script.

I'm going to assume that everything has gone swimmingly and you've now got linux using a ramdisk for /var. The last step is to get it to use a ramdisk for /tmp as well.

I ran into a problem here. Because every ramdisk is the same size, and I need roughly 100Mb for /var, I didn't have enough free memory to create a whole new ramdisk for /tmp, which only needs a little bit anyways. The easiest workaround here is to just symbolically link /tmp to /var/tmp.


init 1
rm -rf /tmp
ln -s /var/tmp
chmod a+rw /tmp
chmod a+rw /var_orig/tmp

Make sure you get the permissions right, or nobody but root will be able to do very much.

Let me know how that works for you!

LIRC: Help!

2006-02-09T13:51:00.000-08:00

The above graph shows the 'space' values when I activate my remote control. Note that the noise around the 2000-level is constantly displayed during mode2.

The above graph shows the 'pulse' values for the same scenario. Again, the noise around the 2000-level is constantly printed during mode2.

Any ideas, please contact me. My name is Russ, you can reach me at [my name] at razorwire.ca

ps. Sorry that the colors got wanged.

Getting GUI (step 2)

2006-02-01T15:26:00.000-08:00

What the hell, let's throw caution to the wind and just go with GNOME. It's a robust, widely supported GUI, although it's a little heavier than I'd like. Perhaps we can slim it down once we've got everything set up, but even if we can't, I think we'll still be well under 1Gb.

Installing GNOME is as easy as:

apt-get -t sarge-backports install x-window-system-core gnome-session gnome-applets nautilus metacity xscreensaver gdm gedit gnome-terminal

Reboot (or simply startx), and you're done.

Getting GUI (step 1)

2006-02-01T14:41:00.000-08:00

The first step to getting a GUI (a window manager, actually) is to install an X server. With Debian, you can choose from at least 2 X servers: Xfree86 or Xorg. Xfree86 is the more popular choice, but we will need the Xorg server to be able to compile the driver for our on-board graphics card.

The biggest obstacle here is that the Xorg packages are listed as "experimental" and are not readily available with the standard configuration of apt-get. So, follow the instructions here to gain access to these experimental packages. Skip the bit between "To activate a package..." and "...as usual". It's much easier to just use the -t option to apt-get when you want to install one of the experimental packages.

Download the xserver-xorg package by running apt-get -t sarge-backports xserver-xorg. You can accept all the defaults for the X configuration process and just choose the "Vesa" driver for the graphics card.

Download the corresponding source code - we'll need this when we come to building the video driver. I don't think there's a Debian package for this code, so we'll have to download it the old fashioned way. Currently the xserver-xorg package is 6.9.0 and the corresponding source is available here. You can check the version of your xserver-xorg package in aptitude. Expand this file into /usr/local/src.

If you check the disk space with df --si /, you'll see that we're up at around 1.2Gb. At first blush, this is pretty bad. But do a df on /usr/src and /usr/local/src, and you'll see that over 750Mb of that is just source code, which will be removed long before we cut over to compact flash. So we're actually doing OK.

Now that we've got the X server up we just need a Window Manager. That'll be the next post.

Debian Post-Install

2006-02-01T14:08:00.000-08:00

The more I use Debian, the more I like it. apt-get is a really powerful package management tool. The minimal install is a bit larger than I'd like, around 250Mb, but I've basically resigned myself to getting a bigger compact flash card, so it's not that big a deal. As long as I can keep it under 1Gb, I should be OK.

Here's how I customised the standard Debian install:

I got the installer onto the hard disk as described here.

I accepted all the defaults, the only tricky bit was to manually edit the partition table and force the install into one of my 2Gb partitions. That's just a matter of selecting the partition, specifying the file system you wish (I went with ext3), and setting it to be the root (/) mount point.

After the installer reboots, the last step in the installation process is to choose which "profile" you would like to set up the system with: file server, print server, web server, desktop, etc. The last option is to perform the package selection manually using aptitude, a frontend for apt-get. I went with that, and then promptly quit it without selecting any packages. This minimal install weighs in at 246Mb.

It's not quite suitable, though. The installer decided to use the i386 kernel, which is compatible with the VIA C3 chip, but doesn't take advantage of all its features. The modern (2.6.8) kernel has a special configuration for this chip, it's just a matter of specifying it at build-time. So let's build a fresh kernel.

The first thing to do is drag down the necessary packages via apt-get. This is going to incur another couple hundred megs of disk usage, but we can get rid of it all when we're done:


apt-get install kernel-source libncurses5-dev

Then, go into the /usr/src directory and expand the .tar.bz2 file:
tar xfvj kernel-source-2.6.8.tar.bz2
Then in the directory that just got created:


make mrproper menuconfig

This brings up the kernel configuration GUI. Under "processor family" select VIA C3-2 (Nehemiah). Exit. Save when prompted. Next, enter:


make install modules_install

And go get some coffee. Take a long lunch. Wash your car. Seriously. This will take about 3 hours, and 98% of this time will be spent compiling drivers for ISDN cards and SCSI devices and a whole host of other crap you don't need. If you know of a way to speed up this process, I'd love to know.

So far we've built and deployed the kernel and all the necessary kernel modules for the C3 chip. The penultimate step is to build a new initrd image:

mkinitrd -o /boot/initrd.img-2.6.8 2.6.8

And finally, to add the entries in the GRUB boot menu to boot our wonderful new creation:

title  Debian GNU/Linux, kernel 2.6.8
root  (hd0,1)
kernel  /boot/vmlinuz-2.6.8 root=/dev/hda2 ro
initrd  /boot/initrd.img-2.6.8
savedefault
boot

title  Debian GNU/Linux, kernel 2.6.8 (recovery mode)
root  (hd0,1)
kernel  /boot/vmlinuz-2.6.8 root=/dev/hda2 ro single
initrd  /boot/initrd.img-2.6.8
savedefault
boot

Note that your value for the root parameter may vary - details explained here.

Don't expect anything fancy when you reboot - there's still no X server or window manager. I'll get to those next.

Installing Debian

2006-01-30T16:55:00.000-08:00

Installing DSL went fine, but I wanted to take a little side-track and setup Debian on Fulcrum (Via Epia SP800). I have a feeling that the drivers for the CN400 chipset, which can help out in the video decoding, will be better supported under Debian.

I thought that installing Debian without a CD would just take a similar sort of jiggery-pokery as installing DSL or SUSE from one partition to another. Unfortunately, the default Debian installer is hard-coded to look for a CD-ROM drive and will refuse to load its packages from anywhere else. Once I figured out the necessary steps, it was pretty straightforward:

Download the "business card" network install CD
RTFM: see sections 4.5 and 5.1.2 and 6.3.1.5 of Debian Install Guide
In a clean partition, create /boot/newinstall. Copy vmlinuz and initrd.gz from here to /boot/newinstall. Copy businesscard.iso from step one to /boot.

Add new entry to GRUB configuration:


title  New Install
kernel (hd0,3)/boot/newinstall/vmlinuz root=/dev/ram0 ramdisk_size=12000
initrd (hd0,3)/boot/newinstall/initrd.gz

Accept all defaults in installer.

Installing DSL

2006-01-26T10:42:00.000-08:00

At this point, Fulcrum is running a LiveCD version of Damn Small Linux 2.0. Creating an actual hard disk installation is pretty easy. When DSL boots to the desktop, right-click on any blank space and select Apps>Tools>Install to Hard Drive. Specify the target partition, in this case, hda3.

It asks a few other questions that are irrelevant for our purposes. I disabled multi-user logins and enabled the ext3 filesystem. I said no when it offered to configure GRUB, just because I'm not sure how nicely it will play with the boot loader we already configured. So after the install, I went into the menu.lst file I'd already set up and added a new entry:


###dsl-2.0###
title Damn Small Linux 2.0
   root (hd0,2)
   kernel /boot/linux24 root=/dev/hda3 quiet vga=794 noacpi noapm nodma noscsi frugal

The entire menu.lst file is available here. It contains two entries, one for the LiveCD and one for the proper install.

Changing the Hostname

2006-01-25T13:37:00.000-08:00

The DSL Wiki has a thorough article about how to permanently change a machine's hostname. It's the last two steps, the mailname and the knoppix-specific stuff, that I was unaware of. Note that a different page on that Wiki suggests using the boot parameter "host=" or "dslhost=" to set the host name. That don't work, at least not in 2.0.

Linux is Beefy

2006-01-24T16:33:00.000-08:00

I'm not sure if the aphorism "Linux is Beefy" means anything outside of Cray Canada. BEFE is the post code we use to indicate that the kernel is up and running, that the boot sequence is effectively complete. I've just reached this point with Fulcrum. Herewith, a description of how it got there.

I've picked Damn Small Linux as the distro for this machine. It's a compact, full-featured distribution that I've had some good luck with so far. As a LiveCD - a "self-sufficient" CD that can be used to boot a computer - it fits into 50Mb. As an installed operating system, I believe it can squeeze into around 200Mb.

Now, I don't have a spare CD drive to hook up to Fulcrum, and even if I did, the tricky route is a lot more interesting and educational. Fortunately, I have a Linux workstation to help with the task.

The first step is to get the hard disk into a system where I can lay out and format the partitions just the way I want them. So I plugged the drive into my desktop PC, which runs SUSE 10.0 and had a free IDE channel.

I used the parted utility to create three primary partitions. The first was 1GB and the second and third were 2GB each. The first partition will be used to hold the images for the boot loader (1GB is way, way too much here, but it doesn't really matter). The second partition will hold the LiveCD ISO and the third partition will hold the actual installation.

Installing the boot loader was a little harder than it should have been. First, I created three mount points for the partitions and mounted them:


mkdir /mnt/fulcrum
mkdir /mnt/fulcrum/grub
mkdir /mnt/fulcrum/iso
mkdir /mnt/fulcrum/os
mount /dev/hdc1 /mnt/fulcrum/grub
mount /dev/hdc2 /mnt/fulcrum/iso
mount /dev/hdc3 /mnt/fulcrum/os

Then I ran the grub-install command to install the boot loader on the disk:


grub-install --root-directory=/mnt/fulcrum/grub /dev/hdc

Now, as far as I know, what this should do is install GRUB boot loader into the master boot record of the disk, and install all the extra GRUB data into the /dev/hdc1 partition. But I got an error message that looked a lot like a disk error when I tried that... something like "unable to read stage1 file".

Looking at the newly-created directory tree, it was pretty apparent that some files were missing from /mnt/fulcrum/grub/boot/grub. So I cleaned out the partition and just copied over the entire /boot directory into /mnt/fulcrum/grub. What this means is, I took SUSE's /boot directory and all the GRUB data it uses and cloned it for Fulcrum.

I tweaked the device.map file a little bit - it's fairly self-explanatory and pretty well documented online - and also added a fresh entry to boot the DSL LiveCD, which we're about to install. The new entry looks like this:


###dsl-2.0-live###
title Damn Small Linux 2.0 LiveCD
    root (hd0,1)
    kernel /boot/isolinux/linux24 root=/dev/hdb6 init=/etc/init lang=us vga=normal BOOT_IMAGE=knoppix
    initrd /boot/isolinux/minirt24.gz

This tells GRUB to look in the 1'th partition for the Damn Small Linux kernel and initrd image. I believe GRUB numbers disks and partitions starting from 0, so this corresponds to the partition we called /dev/hdc2 above.

The last step is to actually get the Damn Small Linux LiveCD into that partition, which is empty right now. This is fairly straightforward. First, we mount the LiveCD iso (which I had downloaded earlier via BitTorrent from damnsmalllinux.org):


mkdir /tmp/dsl2
mount -o loop /usr/local/downloads/dsl-2.0.iso /tmp/dsl2

Then we copy the contents into the empty partition:


cp -R /tmp/dsl2/* /mnt/fulcrum/iso

And that's it! At this point I shut down my desktop PC, removed the hard drive, and connected it back to Fulcrum. DSL booted successfully and asked me a couple questions related to video and mouse configuration, and that was it.

So now what we have is a fake LiveCD distribution. It's running Linux, sure, but it's basically treating the hard drive as a read-only file system and making all its changes in a ramdisk. That's sub-optimal, because none of our configuration changes will be saved next time we reboot.

The next step is to install the operating system to that third partition we created and really get to work setting up this media center. After that, we can start to think about cutting over to the compact flash and getting rid of this hard disk entirely.

It Boots!

2006-01-24T16:20:00.000-08:00

I just got Fulcrum (the client node) to boot. It would've been easier if I had a proper case rigged up for it. Since it's just sitting on a cheesy piece of plywood right now, I had to jumper pins 6 and 8 on the F_PANEL pin array. This is described on page 26 (actually page 34) of the manual.

The only trick is that these pins should be hooked up to a momentary contact switch - which makes sense. Think about the power button on your PC: you push it in, it pops right back out, and your PC starts to boot. So to simulate that behaviour without a switch, you just need to short the two pins briefly. I used a pair of needlenose pliers to push the jumper down and then immediately pull it off. Thanks to Brian for pointing this out.

The Hardware

2006-01-23T15:28:00.000-08:00

As I mentioned in the intro, our little media network will initially have two nodes.

First, the server, which we shall christen "Backfire", because it is large, loud, and ungainly:

Wow. Okay, so the labelling didn't turn out so good. Anyways, these are some of the components:

Antec 4U Rackmount (no, I don't have a rack in my house. But I have a sturdy shelf, and a large storage space under the stairs. It'll do, for now...)
Dual Opteron 248's. This server is going to be doing a lot of stuff other than acting as a server for this project. These 64-bit processors ought to give it plenty of horsepower.
The cheapest PCI-express video card I could find. This thing will be running headless, so I just needed the video card to get it set up. I believe it's an NVidia 6200.
The motherboard, a Tyan Thunder K8WE (minus the SCSI controller). Stocked with 2Gb of PC3200 RAM, or 1Gb per processor.

This stuff isn't cheap - at least, not on my budget. But I got a sweet deal on the processors and the RAM, and the rest of the gear came to about $900CAD total. You could definitely save a few hundred bucks and go with a cheaper motherboard, for instance the ASUS K8N-DL. I went with the Tyan 'cause it's got a little more room for expandability. And the rackmount ... well, that's a bit of an extravagance. But I've always wanted a rackmount.

The first client in this network will be a Mini-ITX based PC, which we shall name "Fulcrum" because... well, because I like that name.

(Sharpie provided for scale)

The centrepiece of this computer is the Via EPIA SP800 motherboard, an 800Mhz motherboard with onboard LAN, Video, TV-out, and sound in a 17cmx17cm form factor. I strongly recommend the folks at Logic Supply as a source for this part - they were great to me even when some trouble arose with shipping that was not at all their fault. The SP800 is currently the fastest fan-less motherboard Via makes. As I mentioned, one of the goals of this system is to have no moving parts.

Another advantage to the SP800 is that it has some hardware built-in to help out with video and audio encoding/decoding. This means that the CPU doesn't have to busy itself with these tasks, and will be available to do other stuff while we're watching a movie or playing an MP3.
Power is supplied to the board by a 60W DC-DC converter and a 12VDC power supply. This is the Morex 60W kit available from Logic Supply.
512 Mb of PC3200 RAM. Because I hope to construct a read-only file system (I'll get to that in a minute), it would be good to have as much RAM as possible. The SP800 supports up to 1Gb but I only had a 512Mb stick lying around.
A Compact Flash/IDE adaptor. This is also available from Logic Supply (those guys are great, did I mention that?) The idea here is to load the operating system up onto a compact flash card and have the computer treat it just like a regular hard drive. But because compact flash only supports a limited number of write cycles (granted, a large number), we need to make it read-only.
A hard drive cage, with a 6Gb hard drive inside. We'll use this disk to get everything set up, and only cut over to the compact flash at the last minute.

Well, that's what we've got to work with! Let's see how far we can get...

Introduction

2006-01-23T15:12:00.000-08:00

This blog is an ongoing journal of my attempt to build a digital media network. The rough idea goes like this:

I will have a media server in my basement, located in a spot where it can make as much noise as it wants. This computer will host all my digital photos (roughly 3600 ~ 8Gb worth), all my movies (which, depending on compression, could easily break 1Tb), all my MP3's (about 20Gb), and any digital video I choose to capture (probably from a combination of cable TV and my in-laws' camcorder).

In addition to this, I will have at least one "client" computer, which will display media from the server. The primary requirement for this computer is that it be quiet. Secondary requirements are:

No moving parts (so I can leave it on all the time without wearing out fan bearings, drive motors, etc) - this obviously relates to the noise level.
Cheap! I want to put one of these in the living room, in the rec room, in the bedrooms, etc.

I expect there are a lot of people that would like to have a setup like this, so hopefully this site can become a useful resource. My contact info can be found in the "About" section that shows up on all pages; if you have any questions that aren't answered on the site please don't hesitate to ask.

Russ

Just getting started...

2006-01-23T14:35:00.000-08:00

First post, just figuring stuff out. Finding my blog-legs, as it were.