Running FreeBSD 4.8 on the Soekris "net4801"

After having only positive experience with the Soekris "net4521" that I used to create a IEEE 802.11 wireless access point, I couldn't wait to get my hands on the brand new model "net4801". The main reason was that I wanted to get rid of the fat, noisy and power hungry (126W) AMD Athlon based PC that I had switched on 24x7, just to receive email, keep a CVS server up-to-date and serve out some (static) web content.
Aside from faster CPU (266 Mhz NSC Geode® SC1100) and more memory (128 MB), the major feature of the Soekris net4801 is a UltraDMA-33 interface with room to fit a 2.5 inch (notebook) drive. And, it's a great saver on the power plant's bill as it draws just about 8W on average (compare that to the AMD I mentioned and the net4801 is almost for free :-))

Making friends with FreeBSD and the Soekris "net4801"

The National Semiconductor Geode® SC1100 chipset used in the "net4801" has a few quirks that require some hacking in the FreeBSD kernel. On the Soekris web site, the issues are explained and fixes suggested. I've taken a slightly different path: I MFC-ed a few fixed and made some configurable through the kernel config file. Speaking about MFC-ing: FreeBSD 5.x-current runs well on the "net4801", but if you want 5.1-RELEASE, you have some work to do too.

My interest, for now, is FreeBSD 4.8, so the rest of this document explains how to get FreeBSD 4.8 running on the Soekris "net4801" (and it's likely to apply to 4-STABLE as well). Before you proceed, make sure you have a fairly recent FreeBSD system with full sources installed. This document is written for RELENG_4_8, the security branch of RELENG_4_8_0_RELEASE, commonly known as "FreeBSD 4.8". Also check that the kernel on your development system contains the vnode driver (the GENERIC kernel doesn't!). When applicable, rebuild your kernel with this option:

  pseudo-device   vn              # Vnode driver (turns a file into a device)

Since the net4801 has neither CD-ROM nor floppy drive, it's likely that you have to do the initial install over the network, unless you have a notebook that you can swap disks with and use that one for staging the installation. The "net4801" (and other models) contain a smart boot ROM that contains Intel's Preboot eXecution Environment (PXE). FreeBSD supports this with pxeboot, a modified version of the system third-stage bootstrap. After DHCP configuration, pxeboot retrieves the kernel, modules, and other files either via NFS or TFTP.

1. Prepare PXEBOOT, LOADER and DHCP/TFTP

The pxeboot program can fetch loader and kernel via TFTP or NFS. If you want to avoid having to set up NFS on your server, you can re-compile pxeboot to use TFTP only. Additionally, pxeboot and the loader by default try to emulate a cons25 terminal, which causes the usual spinning-characters to spread all over the screen and effectively yields the boot details almost unreadable.
Apply the modifications listed below. Note that pxeboot, the loader and others will be installed in /tftpboot in a later stage, when you will be running a make (see chapter 2).

  # vi /etc/make.conf
    LOADER_TFTP_SUPPORT= YES

  # cd /usr/src/sys/boot/i386/libi386
  # vi Makefile
    #dont#CFLAGS+= -DTERM_EMU

  # cd /usr/src/sys/boot
  # make clean depend all install

1.1 Prepare the DHCP and TFTP server

If you don't have a running DHCP server already, build isc-dhcp3 from the ports collection:

  # cd /usr/ports/net/isc-dhcp3
  # make all install clean

PXE needs two additional directives from the DHCP server: the next-server it will query and the root-path of the tree where to find the files it is supposed to retrieve. Here's a sample entry from the DHCP configuration file /usr/local/etc/dhcpd.conf:

  # vi /usr/local/etc/dhcpd.conf
    host soekris {
       hardware ethernet 0:0:24:aa:bb:cc;
       fixed-address soekris.example.net;
       filename "pxeboot";
       next-server 192.168.1.2;
       option root-path "/tftpboot";
    }

To enable the TFTP server, uncomment the tftp line from /etc/inetd.conf to read:

  tftp	dgram	udp	wait	root	/usr/libexec/tftpd	tftpd -s /tftpboot

Don't forget to restart inetd:

  # kill -HUP `cat /var/run/inetd.pid`

2. PBI: PXE Boot Image

In order to be able to do anything useful on the Soekris, aside from staring at the little box, admiring its color, we must find a way to tell it to boot FreeBSD and give us a shell prompt. Basically, there are just a few things we really want to do in the early stages, like:

• Boot FreeBSD using a (minimal) kernel so that we can verify that at least the hardware is okay.
• Provide the tools to initialize the hard disk or CompactFlash card (fdisk, disklabel, newfs, mount, ...) and to configure the network (ifconfig, route, ...)
• Network download of a diskimage to copy to CompactFlash (tftp, dd) and/or provide /stand/sysinstall to do a normal FreeBSD installation from e.g. a FTP server.

Our goal is to build both kernel and a (small) root filesystem. The latter will load in memory (MFS filesystem) and mounted on /. I call the package that builds both, the PBI (PXE Boot Image).

2.1 PBI: the kernel

The most visible problem with the SC1100 chipset is that the kernel will hang in the PCI probe. This implies also that you cannot boot a pre-built GENERIC kernel as distributed on the CD-ROM. From here, there's basically two ways you can go: for the fast route, skip to the end of section 2.2 and download the kernel and MFS root image that I created, or, for the more adventurous trip, patch the kernel source tree and build a kernel yourself.

2.1.1 patching the kernel source tree

  # cd /usr/src
  # patch < 4.8-SOEKRIS_NET4801.diff

2.1.2 kernel configuration file

  options	CLK_TSC_IS_BROKEN	#Local hack (16/Aug/2003 EHK)
  options	ATA_NSC_GEODE_SC1100	#Local hack (16/Aug/2003 EHK)

2.1.3 building the kernel

2.2 PBI: the MFS root filesystem

Creating a MFS root filesystem is a bit more work, but the good news is that it's just a matter of simply running make and you will be ready. The techniques used resemble those used in e.g. PicoBSD, but a bit simplified.

Download the tarball pbi-4.8.tar.gz and unpack it in a directory of your choice. If you prefer, you can browse the PBI tree before downloading the gzipped tarball (which is actually only 10 KB or so).

  # cd /where/i/want/it
  # tar zxpvf pbi-4.8.tar.gz

Take a few minutes to walk yourself through the files in the "pbi" directory. You'll see that the Makefile does most of the work. In the data/sbin is a setup script that may be useful if you're working with a CompactFlash card/image.

If the home directory of your TFTP server is not /tftpboot, you might want to edit the pbi/Makefile and change the line that reads:

   PXEROOT?=       /tftpboot       # TFTP root directory

The file crunch.conf.in is used by crunchgen to create a build environment for the tools you want to have available in the root filesystem once booted on the Soekris. Included are:

atacontrol, boot0cfg, cat,  chmod, clri, cp, crunch,  dd, df, disklabel,
dmesg,  echo,  expr, fdisk,  fetch,  fsck,  gzip, halt,  ifconfig,  ipf,
ipfstat, ipnat, kill, ln, ls, mkdir, mknod, mount, mount_mfs, mount_std,
mtree, newfs, oinit, pax, ping, ps,  pwd, reboot, rm, route, sh, sysctl,
tar, test, traceroute, tunefs, umount, vi, /stand/sysinstall

If you're confident enough to start the build, simply run make (as 'root' !) in the "pbi" directory:

  # cd pbi && make

This will do the complete build (kernel + root mfs) and install the results (with the 'pxeboot' modified in chapter 1) in the directory /tftpboot. Here's a transcript of the output from the make command on my system for comparison. Now, inspect the /tftpboot directory; it should look similar to this:

  /tftpboot:
    drwxr-xr-x  3 root  wheel      512 Jan  6 21:52 boot
    -rwxr-xr-x  1 root  wheel   995264 Oct 12 16:09 kernel.gz    [kernel.gz.asc]
    -rw-r--r--  1 root  wheel  6144000 Oct 12 16:09 mfsimage.bin [mfsimage.bin.asc]

  /tftpboot/boot:
    drwxr-xr-x  2 root  wheel     512 Jan  6 21:52 defaults
    -r-xr-xr-x  1 root  wheel  155648 Aug 16 16:53 loader*
    -r--r--r--  1 root  wheel    9237 Apr  3 10:55 loader.4th
    -rw-r--r--  1 root  wheel     154 Sep 21 23:23 loader.conf
    -rw-r--r--  1 root  wheel      31 Sep  7 16:42 loader.rc
    -r--r--r--  1 root  wheel   25121 Apr  3 10:55 support.4th

  /tftpboot/boot/defaults:
    -r--r--r--  1 root  wheel   11067 Oct 11 21:42 loader.conf

Even if you prefer another procedure to install, you most likely want to download the kernel from the listing above, to get past the kernel hang on PCI probe challenge.

Alternatively, you may choose to download the mfsroot.bin.gz image which is in fact the content of the FreeBSD-4.8-RELEASE second boot floppy. If you do, copy this one to /tftpboot/, gunzip it and edit /tftpboot/boot/loader.conf to refer to this image instead.

2.3 Boot the Soekris

To connect the serial line of another PC to the serial port of the Soekris, you need a DB-9 female-female null-modem cable. Here's the pinout of the one I made:

   1  /--o          o--\  1      although you might get   1  o          o  1
   2  |  o -------- o  |  3       away with just this :   2  o -------- o  3
   3  |  o -------- o  |  2                               3  o -------- o  2
   4  +--o          o--+  4                               4  o          o  4
   5  |  o -------- o  |  5                               5  o -------- o  5
   6  \--o          o--/  6                               6  o          o  6
   7     o -------- o     8                               7  o          o  8
   8     o -------- o     7                               8  o          o  7
   9     o          o     9                               9  o          o  9

Assuming that you have the serial ports connected, you can power-on the Soekris and watch it start up. Stay awake to Press Ctrl-P for entering Monitor in time and boot from the network (the port labelled "Eth0"):

  > boot f0

If DHCP is working on your server and the TFTP server enabled, it should now retrieve the kernel and MFS and start executing it. Here's a transcript of the boot messages from my system.

At this point, we've succesfully booted FreeBSD. You may now proceed with a normal FreeBSD install, e.g. by starting sysinstall (re-setting the TERM environment variable is optional):

  # set TERM=vt100
  # /stand/sysinstall

2.3.1 Before Rebooting

Warning: at the end of a normal sysinstall, a GENERIC kernel will be installed in the root directory. Recall that this one won't boot on the Soekris. So, once the installation is finished, but before rebooting the system, fetch and install the same kernel you used to get the PBI bootstrap to boot!

3. Normal Operation

There's a few things you might want to fine tune during normal operation:

• Start a getty on the console port.
  In case you need to get console access via the serial port, it helps if there's a getty running displaying a login prompt. To enable this, edit the ttyd0 entry in /etc/ttys to read:

    ttyd0   "/usr/libexec/getty std.9600"   dialup  on secure
  

  Additionally, disable the ttyv[0-3] entries that are likely on.

4. References

• FreeBSD Jumpstart Guide by Alfred Perlstein.
• Booting FreeBSD with PXE by Matt Simerson.
• miniBSD - reducing FreeBSD by Manuel Kasper.
• m0n0wall and m0n0BSD by Manuel Kasper.