Basic FreeBSD Setup

• https://docs.freebsd.org/en/books/handbook/jails/

For my use-case making a new jail involves creating a ZFS dataset, copying the FreeBSD root filesystem onto it, tweaking some files in the jail, and adding entries to /etc/jail.conf and /etc/rc.conf. Most resources recommend creating a template dataset and cloning it to create new jails, however I'm fairly sure that that will make upgrading it a hassle. To expedite this process I wrote a shell script:

/usr/local/sbin/add-jail.sh

#!/bin/sh
# Automate the process of creating a new jail

[ -z "$1" ] && {
	printf "Create a new jail. Usage: add-jail.sh [jailname]\n"
	exit 1
}

jail="$1"
file=ftp.freebsd.org/pub/FreeBSD/releases/"$(uname -p)"/"$(uname -r)"/base.txz 

zfs create storage/jails/"$1"
fetch "$file" -o - | tar -xf - -C /storage/jails/"$jail"
freebsd-update -b /storage/jails/"$1" IDS

# Copy some stuff over
cp -vf /etc/localtime /storage/jails/"$jail"/etc/localtime
cp -vf /etc/csh.cshrc /storage/jails/"$jail"/etc/csh.cshrc
# cp -vf /etc/login.conf /storage/jails/"$jail"/etc/login.conf

cat <<- EOF
======> Completed
  If freebsd-update detected errors, remove the storage/jails/$jail
  dataset and try again. Otherwise you should now:
    [ ] Add the $jail entry to /etc/jail.conf
    [ ] Add a table to /etc/hosts
    [ ] Add rules to /etc/pf.conf or /usr/local/etc/haproxy.conf
    [ ] Append $jail to jail_list in /etc/rc.conf
    [ ] Restart the jail service
    [ ] Set a root password
EOF

Whenever I create a new jail I'll need to append a corresponding entry to /etc/jail.conf. I prefer to keep jails as small and disposable as possible, so any important data should be stored in directories mounted to the jail using the mount.fstab directive. For reasons I'll be getting into below, this is also required for SSL certificates.

/etc/jail.conf

# set filetype=conf

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

host.hostname = $name;
path = "/storage/jails/$name";
exec.consolelog = "/var/log/jail_${name}_console.log";
exec.prestart = "cp /etc/resolv.conf $path/etc";
exec.poststop = "rm $path/etc/resolv.conf";

www {
	ip4.addr = "lo1|127.1.1.1/32";
	ip6.addr = "lo1|fd00:1:1:1::1/64";
	allow.chflags;
	allow.raw_sockets;
}

In order to facilitate updating jails, I wrote the following script and corresponding monthly entry in the root user's crontab.

/usr/local/sbin/update-jails.sh

#!/bin/sh
# Automatically update all the running jails
# crontab -e:
# 0	0	1	*	*	/usr/local/sbin/update-jails.sh

find /storage/jails -type d -depth 1 | while read -r j; do
	n="$(basename "$j")"
	jls | grep "$n" >/dev/null && {
		freebsd-update -b "$j" fetch --not-running-from-cron
		freebsd-update -b "$j" install
		pkg -j "$n" upgrade
	}
done

service jail restart

• https://cbonte.github.io/haproxy-dconv/2.5/configuration.html

For the sake of convenience, we'll use /etc/hosts to map domain names and aliases to the internal IP addresses of the appropriate jails.

/etc/hosts

::1		localhost	localhost.ulthar.xyz
127.0.0.1	localhost	localhost.ulthar.xyz

127.1.1.1	ulthar.xyz	www
fd00:1:1:1::1	ulthar.xyz	www

In order to run network services in jails, we need to set up a NAT to send traffic from external ports to the appropriate jails, and vice verse.

/etc/pf.conf

# MACROS/TABLES
XIF = "re0"
JAILNET_V4 = "127.1.1.0/24"
JAILNET_V6 = "fd00:1:1:1::0/64"
EXT_V6ADDR = "2001:db8::1"

# TRANSLATION
## NAT
nat on $XIF inet from $JAILNET_V4 to any -> ($XIF)
nat on $XIF inet6 from $JAILNET_V6 to any -> $EXT_V6ADDR

## REDIRECT (Port Forwarding)
### Jails running HTTP services are managed by HAProxy
### Redirect SMTP and IMAP services to the ProtonMail Bridge jail
### Pass ports 522 and 5269 to Prosody's jail

# FILTERING (Pass/Block)
## Open the Prosody jail's ports to the external network

Pf doesn't have any understanding of higher-level protocols like HTTP, so to run multiple web services in different jails we'll need to use some kind of reverse proxy. While you could accomplish this with Nginx, I'm using HAProxy. This will bind to ports 80 and 443, resolving secure connections, then proxying the requests to port 80 on the relevant jails.

/usr/local/etc/haproxy.conf

# set filetype=conf

global
	daemon
	maxconn 256

defaults
	log global
	mode http
	timeout connect 5000ms
	timeout client 50000ms
	timeout server 50000ms

##### Redirect HTTP to HTTPS
frontend http_listener
	bind *:80
	redirect scheme https

#### Resolve HTTPS and redirect requests to jails
frontend https_listener
	bind *:443 ssl crt /usr/local/etc/ssl/haproxy/
	# ulthar.xyz hosts
	acl is_www hdr(host) -i www.ulthar.xyz
	acl is_www hdr(host) -i ulthar.xyz
	use_backend www if is_www

#### Redirection Targets
backend www
	option forwardfor
	server http_www 127.1.1.1:80 check

Having HAProxy handle secure connections means that I don't have to worry about keeping track of certificates in each jail. The downside of the current configuration is that it prohibits plain HTTP, but this isn't a huge deal since most of us aren't browsing the web on 486s and whatnot.

• https://certbot.eff.org/lets-encrypt/freebsd-haproxy.html

One of the advantages of sticking HAProxy in front of my different web services is that it allows me to manage all my SSL certificates in one place.

# certbot certonly --standalone

This will store certificates and keys like so:

/usr/local/etc/letsencrypt/live/
|- README
|- domain.tld
   |- README
   |- cert.pem
   |- chain.pem
   |- fullchain.pem
   |- privkey.pem
 ...

The FreeBSD package for certbot ships an auto-renewable bot which is enabled above in /etc/rc.conf. Certbot needs to bind to port 443 and we've configured HAProxy to expect fullchain certs and private keys concatenated together in /usr/local/etc/ssl/haproxy, so we'll need to write some renewal hooks.

/usr/local/etc/letsencrypt/renewal-hooks/pre/haproxy-pre.sh

#!/bin/sh
service haproxy stop

/usr/local/etc/letsencrypt/renewal-hooks/post/haproxy-post.sh

#!/bin/sh
find /usr/local/etc/letsencrypt/live -type d -depth 1 | while read -r d; do
	cat "$d"/fullchain.pem "$d"/privkey.pem \
	    >/usr/local/etc/ssl/haproxy/"$(basename "$d")".pem
done

service haproxy start

Some jailed services like Prosody require access to certificates. For these you can see the fstab-style mount directives above in the appropriate sections of /etc/jail.conf.

Certificates can then be renewed by running certbot renew as root in the host system. I still need to set up a good way to do this automatically.