Basic FreeBSD Setup

This file documents how I've configured the FreeBSD install on my home server. It is an amateur's collection of notes, not a guide. Use it at your own risk.

The root file system lives on a small, unencrypted SSD while everything important (user home directories, shared folders, and jails) live in a ZFS pool on three large, external HDDs mounted on /storage. This machine contains personal backups, archives of cough legally acquired cough media, and several jails running network services, most notably this website.

Preliminary Setup

Starting Services

FreeBSD uses a simple, old fashioned init system with /etc/rc.conf being the initialisation file for services that are run after booting up. We use it to start and instruct daemons and configure the machine's network interfaces.

/etc/rc.conf

# Start daemons
weekly_certbot_enable="YES"
clear_tmp_enable="YES"
sendmail_enable="NONE"
sshd_enable="YES"
ntpd_enable="YES"
powerd_enable="YES"
ftpd_enable="YES"

# Start storage stuff
zfs_enable="YES"
smartd_enable="YES"

# Jail stuff
pf_enable="YES"
haproxy_enable="YES"
pflog_enable="YES"
jail_enable="YES"
cloned_interfaces="lo1"
jail_list="www" # <- Append jails

# Set up a static IP
hostname="monolith"
ifconfig_re0="inet 192.168.5.19 netmask 255.255.255.0"
defaultrouter="192.168.5.1"

# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"

Some nice-to-haves in the Environment

pkg install neovim rsync git ranger w3m haproxy

FreeBSD comes with three shells: sh, csh, and tcsh. The root user defaults to tcsh, and I'll stick with that for the sake of simplicity. With some minor quality-of-life improvements in /etc/csh.cshrc, tcsh makes for a really nice, interactive shell.

/etc/csh.cshrc

umask 22

alias ls        ls -GF
alias rm        rm -i
alias mv        mv -i
alias cp        cp -i

# The neovim and nginx-full packages have incompatible Lua dependencies,
# so $EDITOR should gracefully downgrade to vim then vi.
setenv EDITOR vi
command -v vim >/dev/null && setenv EDITOR vim
command -v nvim >/dev/null && setenv EDITOR nvim

# Settings for the interactive shell
if ($?prompt) then
	set filec
	set autoexpand
	set autorehash

	set promptchars = "%#"
	set history     = 1000
	set autolist    = ambiguous
	set savehist    = (1000 merge)
	set prompt      = "%{\033[32m%}%B%n@%m%b%{\033[0m%}:%~ %# "

	# Set a distinctive prompt if we're running in a jail
	ps 1 >/dev/null || set prompt = "[JAIL]$prompt"

	# Plain old csh doesn't support these
	if ($?tcsh) then
	bindkey "^R" i-search-back
		bindkey "^W" backward-delete-word
		bindkey -k up history-search-backward
		bindkey -k down history-search-forward
	endif
endif

In order to have a nice environment in each of the jails we'll copy this into the template dataset later.

Encrypting the Disks

Initialise partition tables on the disks

# gpart create -s GPT ada1
# gpart create -s GPT ada2
# gpart create -s GPT ada3

Generate master encryption keys

# dd if=/dev/random of=/root/ada1.key bs=64 count=1
# dd if=/dev/random of=/root/ada2.key bs=64 count=1
# dd if=/dev/random of=/root/ada3.key bs=64 count=1

Encrypt the disks

# geli init -K /root/ada1.key -s 4096 /dev/ada1
# geli init -K /root/ada2.key -s 4096 /dev/ada2
# geli init -K /root/ada3.key -s 4096 /dev/ada3

Note that the disk metadata is backed up at /var/backups/ada?.eli and may be restored like so: geli restore /var/backups/ada?.eli /dev/ada?

Decrypt the volumes:

# geli attach -k /root/ada1.key /dev/ada1
# geli attach -k /root/ada2.key /dev/ada2
# geli attach -k /root/ada3.key /dev/ada3

There are now /dev/ada?.eli devices

Creating a ZFS Pool

Add the decrypted devices to a ZFS pool and mount it

# zpool create storage raidz ada1.eli ada2.eli ada3.eli
# mkdir /storage
# zfs mount storage

In the future the process of decrypting and mounting can be automated with this bash script:

/usr/local/sbin/mount-storage.bash

#!/usr/bin/env bash
# decrypt and mount the storage pool

# `-s` is not supported in posix shell and visible passwords are bad  
read -s -p 'Storage Decryption Password: ' pass

echo -n "$pass" | geli attach -j - -k "/root/geli/ada1.key" "/dev/ada1" || exit 1
echo -n "$pass" | geli attach -j - -k "/root/geli/ada2.key" "/dev/ada2" || exit 2
echo -n "$pass" | geli attach -j - -k "/root/geli/ada3.key" "/dev/ada3" || exit 3

# zfs mount -a mounts everything
zfs mount storage

# Now that the jail environments are present, we can start the jails
service jail start

Moving Home Directories to the Storage Pool

Create a new dataset so that the home directories can be managed independently of the other files on encrypted drives.

# zfs create storage/home

Copy the users' home directories over and set up symlinks

# cp -rp /home/* /storage/home
# umount /usr/home
# rm -rf /home /usr/home
# ln -s /storage/home /home
# ln -s /storage/home /usr/home

Now reboot the machine, log in, run the mount script, log out, and log back in. Everything should be working.

Isolating Services with Jails

For my use-case making a new jail involves creating a ZFS dataset, copying the FreeBSD root filesystem onto it, tweaking some files in the jail, and adding entries to /etc/jail.conf and /etc/rc.conf. Most resources recommend creating a template dataset and cloning it to create new jails, however I'm fairly sure that that will make upgrading it a hassle. To expedite this process I wrote a shell script:

/usr/local/sbin/add-jail.sh

#!/bin/sh
# Automate the process of creating a new jail

[ -z "$1" ] && {
	printf "Create a new jail. Usage: add-jail.sh [jailname]\n"
	exit 1
}

jail="$1"
file=ftp.freebsd.org/pub/FreeBSD/releases/"$(uname -p)"/"$(uname -r)"/base.txz 

zfs create storage/jails/"$1"
fetch "$file" -o - | tar -xf - -C /storage/jails/"$jail"
freebsd-update -b /storage/jails/"$1" IDS

# The base tarball doesn't include home directories
mkdir -v /storage/jails/"$jail"/usr/home/
ln -vs /storage/jails/"$jail"/home/ /storage/jails/"$jail"/usr/home/

cp -vf /etc/localtime /storage/jails/"$jail"/etc/localtime
cp -vf /etc/csh.cshrc /storage/jails/"$jail"/etc/localtime

cat <<- EOF
======> Completed
  If freebsd-update detected errors, remove the storage/jails/$jail
  dataset and try again. Otherwise you should now:
    [ ] Add the $jail entry to /etc/jail.conf
    [ ] Append $jail to jail_list in /etc/rc.conf
    [ ] Restart the jail service
    [ ] Set a root password
EOF

Whenever I create a new jail I'll need to append a corresponding entry to /etc/jail.conf. I prefer to keep jails as small and disposable as possible, so any important data should be stored in directories mounted to the jail using the mount.fstab directive. For reasons I'll be getting into below, this is also required for SSL certificates.

/etc/jail.conf

# set filetype=conf

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

host.hostname = $name;
path = "/storage/jails/$name";
exec.consolelog = "/var/log/jail_${name}_console.log";
exec.prestart = "cp /etc/resolv.conf $path/etc";
exec.poststop = "rm $path/etc/resolv.conf";

www {
	ip4.addr = "lo1|127.1.1.1/32";
	ip6.addr = "lo1|fd00:1:1:1::1/64";
	allow.chflags;
	allow.raw_sockets;
}

In order to facilitate updating jails, I wrote the following script and corresponding monthly entry in the root user's crontab.

/usr/local/sbin/update-jails.sh

#!/bin/sh
# Automatically update all the running jails
# crontab -e:
# 0	0	1	*	*	/usr/local/sbin/update-jails.sh

find /storage/jails -type d -depth 1 | while read -r j; do
	n="$(basename "$j")"
	jls | grep "$n" >/dev/null && {
		freebsd-update -b "$j" fetch --not-running-from-cron
		freebsd-update -b "$j" install
		pkg -j "$n" upgrade
	}
done

service jail restart

Sending Packets to Jails

For the sake of convenience, we'll use /etc/hosts to map domain names and aliases to the internal IP addresses of the appropriate jails.

/etc/hosts

::1		localhost	localhost.ulthar.xyz
127.0.0.1	localhost	localhost.ulthar.xyz

127.1.1.1	ulthar.xyz	www
fd00:1:1:1::1	ulthar.xyz	www

In order to run network services in jails, we need to set up a NAT to send traffic from external ports to the appropriate jails, and vice verse.

/etc/pf.conf

# MACROS/TABLES
XIF = "re0"
JAILNET_V4 = "127.1.1.0/24"
JAILNET_V6 = "fd00:1:1:1::0/64"
EXT_V6ADDR = "2001:db8::1"

# TRANSLATION
## NAT
nat on $XIF inet from $JAILNET_V4 to any -> ($XIF)
nat on $XIF inet6 from $JAILNET_V6 to any -> $EXT_V6ADDR

## REDIRECT (Port Forwarding)
### Jails running HTTP services are managed by HAProxy
### Redirect SMTP and IMAP services to the ProtonMail Bridge jail
### Pass ports 522 and 5269 to Prosody's jail

# FILTERING (Pass/Block)
## Open the Prosody jail's ports to the external network

Pf doesn't have any understanding of higher-level protocols like HTTP, so to run multiple web services in different jails we'll need to use some kind of reverse proxy. While you could accomplish this with Nginx, I'm using HAProxy. This will bind to ports 80 and 443, resolving secure connections, then proxying the requests to port 80 on the relevant jails.

/usr/local/etc/haproxy.conf

# set filetype=conf

global
	daemon
	maxconn 256

defaults
	log global
	mode http
	timeout connect 5000ms
	timeout client 50000ms
	timeout server 50000ms

##### Redirect HTTP to HTTPS
frontend http_listener
	bind *:80
	redirect scheme https

#### Resolve HTTPS and redirect requests to jails
frontend https_listener
	bind *:443 ssl crt /usr/local/etc/ssl/haproxy/
	# ulthar.xyz hosts
	acl is_www hdr(host) -i www.ulthar.xyz
	acl is_www hdr(host) -i ulthar.xyz
	use_backend www if is_www

#### Redirection Targets
backend www
	option forwardfor
	server http_www 127.1.1.1:80 check

Having HAProxy handle secure connections means that I don't have to worry about keeping track of certificates in each jail. The downside of the current configuration is that it prohibits plain HTTP, but this isn't a huge deal since most of us aren't browsing the web on 486s and whatnot.

SSL Certificates with LetsEncrypt and Certbot

One of the advantages of sticking HAProxy in front of my different web services is that it allows me to manage all my SSL certificates in one place.

# certbot certonly --standalone

This will store certificates and keys like so:

/usr/local/etc/letsencrypt/live/
|- README
|- domain.tld
   |- README
   |- cert.pem
   |- chain.pem
   |- fullchain.pem
   |- privkey.pem
 ...

The FreeBSD package for certbot ships an auto-renewable bot which is enabled above in /etc/rc.conf. Certbot needs to bind to port 443 and we've configured HAProxy to expect fullchain certs and private keys concatenated together in /usr/local/etc/ssl/haproxy, so we'll need to write some renewal hooks.

/usr/local/etc/letsencrypt/renewal-hooks/pre/haproxy-pre.sh

#!/bin/sh
service haproxy stop

/usr/local/etc/letsencrypt/renewal-hooks/post/haproxy-post.sh

#!/bin/sh
find /usr/local/etc/letsencrypt/live -type d -depth 1 | while read -r d; do
	cat "$d"/fullchain.pem "$d"/privkey.pem \
	    >/usr/local/etc/ssl/haproxy/"$(basename "$d")".pem
done

service haproxy start

Some jailed services like Prosody require access to certificates. For these you can see the fstab-style mount directives above in the appropriate sections of /etc/jail.conf.

Viewable With Any Browser Join the Fediverse!!! 100% mothra-compatible Powered by FreeBSD This site made with GNU Emacs

Last Modified: 2021-08-21 Sat 21:29