Basic FreeBSD Setup

This file documents how I've configured the FreeBSD install on my home server. It is an amateur's collection of notes, not a guide. Use it at your own risk.

The root file system lives on a small, unencrypted SSD while everything important (user home directories, shared folders, and jails) live in a ZFS pool on three large, external HDDs mounted on /storage. This machine contains personal backups, archives of media, and several jails running network services.

Most configuration files presented here just contain stubs and skeletons, all the meat is present in the "configuring the host system" section of other service-specific writeups.

Preliminary Setup

Starting Services

FreeBSD uses a simple, old fashioned init system with /etc/rc.conf being the initialization file for services that are run after booting up. We use it to start and instruct daemons and configure the machine's network interfaces.

/etc/rc.conf

# Change some default behaviors
clear_tmp_enable="YES"
sendmail_enable="NONE"
dumpdev="NO"

# Start some general services
zfs_enable="YES"
sshd_enable="YES"
ntpd_enable="YES"
powerd_enable="YES"
smartd_enable="YES"

# Start services for hosting stuff
pf_enable="YES"
jail_enable="YES"
nginx_enable="YES"
pflog_enable="YES"

# Jail configuration
cloned_interfaces="lo1"
jail_list="" # <- Append jails

# Set a hostname and static IP
hostname="monolith"
ifconfig_re0="inet 192.168.5.19 netmask 255.255.255.0"
defaultrouter="192.168.5.1"

Some nice-to-haves in the Environment

# pkg install neovim rsync git ranger w3m nginx py38-certbot

FreeBSD comes with three shells: sh, csh, and tcsh. The root user defaults to tcsh, and I'll stick with that for the sake of simplicity. With some minor quality-of-life improvements in /etc/csh.cshrc, tcsh makes for a really nice, interactive shell.

/etc/csh.cshrc

umask 22

alias ls        ls -GF
alias rm        rm -i
alias mv        mv -i
alias cp        cp -i

# The neovim and nginx-full packages have incompatible Lua dependencies,
# so $EDITOR should gracefully downgrade to vim then vi.
setenv EDITOR vi
command -v vim >/dev/null && setenv EDITOR vim
command -v nvim >/dev/null && setenv EDITOR nvim

# Settings for the interactive shell
if ($?prompt) then
	set filec
	set autoexpand
	set autorehash

	set promptchars = "%#"
	set history     = 1000
	set autolist    = ambiguous
	set savehist    = (1000 merge)
	set prompt      = "%{\033[32m%}%B%n@%m%b%{\033[0m%}:%~ %# "

	# Set a distinctive prompt if we're running in a jail
	ps 1 >/dev/null || set prompt = "[JAIL]$prompt"

	# Plain old csh doesn't support these
	if ($?tcsh) then
	bindkey "^R" i-search-back
		bindkey "^W" backward-delete-word
		bindkey -k up history-search-backward
		bindkey -k down history-search-forward
	endif
endif

Encrypting the Disks

Initialise partition tables on the hard drives, generate master encryption keys, then encrypt them

# gpart create -s GPT ada1
# gpart create -s GPT ada2
# gpart create -s GPT ada3
# dd if=/dev/random of=/root/ada1.key bs=64 count=1
# dd if=/dev/random of=/root/ada2.key bs=64 count=1
# dd if=/dev/random of=/root/ada3.key bs=64 count=1
# geli init -K /root/ada1.key -s 4096 /dev/ada1
# geli init -K /root/ada2.key -s 4096 /dev/ada2
# geli init -K /root/ada3.key -s 4096 /dev/ada3

Note that the disk metadata is backed up at /var/backups/ada?.eli and may be restored like so: geli restore /var/backups/ada?.eli /dev/ada?

Now decrypt the volumes, producing /dev/ada?.eli devices

# geli attach -k /root/ada1.key /dev/ada1
# geli attach -k /root/ada2.key /dev/ada2
# geli attach -k /root/ada3.key /dev/ada3

Creating a ZFS Pool

Add the decrypted devices to a ZFS pool and mount it

# zpool create storage raidz ada1.eli
# zpool create storage raidz ada2.eli
# zpool create storage raidz ada3.eli
# mkdir /storage
# zfs mount storage

In the future the process of decrypting and mounting can be automated with this script. I'm using zsh because the POSIX read builtin doesn't have a flag to avoid echoing the password back to the terminal, and I have zsh installed anyway as my preferred interactive shell.

/usr/local/sbin/mount-storage.zsh

#!/usr/bin/env zsh
# decrypt and mount the storage pool

set -e

read -s pass\?'Storage Decryption Password: '; echo

echo -n "$pass" | geli attach -j - -k "/root/geli/ada1.key" "/dev/ada1"
echo -n "$pass" | geli attach -j - -k "/root/geli/ada2.key" "/dev/ada2"
echo -n "$pass" | geli attach -j - -k "/root/geli/ada3.key" "/dev/ada3"

zfs mount -a
service jail start

Moving Home Directories to the Storage Pool

Create a new dataset so that the home directories can be managed independently of the other files on encrypted drives.

# zfs create storage/home

Copy the users' home directories over and set up symlinks

# cp -rp /home/* /storage/home
# umount /usr/home
# rm -rf /home /usr/home
# ln -s /storage/home /home
# ln -s /storage/home /usr/home

Now reboot the machine, log in, run the mount script, log out, and log back in. Everything should be working.

Isolating Services with Jails

I use FreeBSD's jail subsystem to isolate various self-hosted services from each other and the host system. This means that if one of them is compromised the attacker won't have access to everything else, and it also makes it easy for to keep a clean install which is important when fiddling with software written clunky, hard-to-manage languages like Python, Ruby, and JavaScript.

To create new jails I wrote this little script. After running that we append the corresponding entry to /etc/jail.conf, add the jail name to jail_list in /etc/rc.conf, and restart the service. Since all jails run network services we also need to route packets or forward HTTP requests, which will be covered in the next section.

/etc/jail.conf

# set filetype=conf

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

host.hostname = $name;
path = "/storage/jails/$name";
exec.consolelog = "/var/log/jail_${name}_console.log";
exec.prestart = "cp /etc/resolv.conf $path/etc";
exec.poststop = "rm $path/etc/resolv.conf";

I prefer to keep jails as small and disposable as possible, so any important data should be stored in directories mounted to the jail using the mount.fstab directive. Similarly, I manage SSL certificates in the host system.

In order to facilitate updating jails, I wrote the following script and corresponding monthly entry in the root user's crontab.

/usr/local/sbin/update-jails.sh

#!/bin/sh
# Automatically update all the running jails
# crontab -e:
# @monthly	/usr/local/sbin/update-jails.sh

find /storage/jails -type d -depth 1 | while read -r j; do
	n="$(basename "$j")"
	jls | grep "$n" >/dev/null && {
		freebsd-update -b "$j" fetch --not-running-from-cron
		freebsd-update -b "$j" install
		pkg -j "$n" upgrade
	}
done

service jail restart

Sending Packets to Jails

For the sake of convenience, we'll use /etc/hosts to map domain names and aliases to the internal IP addresses of the appropriate jails.

/etc/hosts

::1		ulthar.xyz	localhost
127.0.0.1	ulthar.xyz	localhost

In order to run network services in jails, we need to set up a NAT to send traffic from external ports to the appropriate jails, and vice versa.

/etc/pf.conf

# MACROS/TABLES
XIF = "re0"
JAILNET_V4 = "127.1.1.0/24"
JAILNET_V6 = "fd00:1:1:1::0/64"
EXT_V6ADDR = "2001:db8::1"

# TRANSLATION
## NAT
nat on $XIF inet from $JAILNET_V4 to any -> ($XIF)
nat on $XIF inet6 from $JAILNET_V6 to any -> $EXT_V6ADDR

## REDIRECT (Port Forwarding)

# FILTERING (Pass/Block)

Pf doesn't know anything about higher-level protocols like HTTP, so in order to run multiple web-services we need a reverse-proxy. Using Nginx for this rather than a dedicated program like Haproxy also means that I can simply serve my personal website from the host system.

/usr/local/etc/nginx/nginx.conf

# set filetype=conf

http {
	sendfile      on;
	include       mime.types;
	default_type  text/plain;
	keepalive_timeout 65;

	server {
		server_name ulthar.xyz;
		listen 80;
		return 301 https://ulthar.xyz$request_uri;
	}

	server {
		server_name ulthar.xyz;
		listen 443 ssl;
		root /storage/home/thalia/public_html;
		ssl_certificate /usr/local/etc/letsencrypt/live/ulthar.xyz/fullchain.pem;
		ssl_certificate_key /usr/local/etc/letsencrypt/live/ulthar.xyz/privkey.pem;
	}
}

SSL Certificates with LetsEncrypt and Certbot

In order to simplify the certificate renewal process I register them in the host systema and do all the HTTPS redirection for other web services in Nginx.

# certbot certonly --standalone

This will store certificates and keys like so:

/usr/local/etc/letsencrypt/live/
|- README
|- domain.tld
   |- README
   |- cert.pem
   |- chain.pem
   |- fullchain.pem
   |- privkey.pem
 ...

In order to automate renewal I wrote a short script and added it as a weekly cron job, it should renew the certificates if they're set to expire within 30 days.

/usr/local/sbin/renew-certs.sh

#!/bin/sh
# Automatically renew all certificates
# crontab -e:
# @weekly	/usr/local/sbin/update-jails.sh

service nginx stop
certbot renew -q
service nginx start
service jail restart

Ejabberd, running in the XMPP jail, requires access to the certificates but doesn't have permissions to read them even when they're mounted. Rather than give them unsafe permissions, I added a startup script to the jail responsible for cp'ing and chown'ing the certificates; hence the directive to restart the jail daemon following renewal.

Viewable With Any Browser Join the Fediverse!!! 100% mothra-compatible Powered by FreeBSD This site made with GNU Emacs

Last Modified: 2022-02-08 Tue 22:30

I wish I could meet you IRL