Basic FreeBSD Setup

# Change some default behaviors
clear_tmp_enable="YES"
sendmail_enable="NONE"
dumpdev="NO"

# Start some general services
zfs_enable="YES"
sshd_enable="YES"
ntpd_enable="YES"
powerd_enable="YES"
smartd_enable="YES"

# Start services for hosting stuff
pf_enable="YES"
jail_enable="YES"
nginx_enable="YES"
pflog_enable="YES"

# Jail configuration
cloned_interfaces="lo1"
jail_list="" # <- Append jails

# Set a hostname and static IP
hostname="monolith"
ifconfig_re0="inet 192.168.5.19 netmask 255.255.255.0"
defaultrouter="192.168.5.1"

# pkg install neovim rsync git ranger w3m nginx py38-certbot

umask 22

alias ls        ls -GF
alias rm        rm -i
alias mv        mv -i
alias cp        cp -i

# The neovim and nginx-full packages have incompatible Lua dependencies,
# so $EDITOR should gracefully downgrade to vim then vi.
setenv EDITOR vi
command -v vim >/dev/null && setenv EDITOR vim
command -v nvim >/dev/null && setenv EDITOR nvim

# Settings for the interactive shell
if ($?prompt) then
	set filec
	set autoexpand
	set autorehash

	set promptchars = "%#"
	set history     = 1000
	set autolist    = ambiguous
	set savehist    = (1000 merge)
	set prompt      = "%{\033[32m%}%B%n@%m%b%{\033[0m%}:%~ %# "

	# Set a distinctive prompt if we're running in a jail
	ps 1 >/dev/null || set prompt = "[JAIL]$prompt"

	# Plain old csh doesn't support these
	if ($?tcsh) then
	bindkey "^R" i-search-back
		bindkey "^W" backward-delete-word
		bindkey -k up history-search-backward
		bindkey -k down history-search-forward
	endif
endif

# gpart create -s GPT ada1
# gpart create -s GPT ada2
# gpart create -s GPT ada3
# dd if=/dev/random of=/root/ada1.key bs=64 count=1
# dd if=/dev/random of=/root/ada2.key bs=64 count=1
# dd if=/dev/random of=/root/ada3.key bs=64 count=1
# geli init -K /root/ada1.key -s 4096 /dev/ada1
# geli init -K /root/ada2.key -s 4096 /dev/ada2
# geli init -K /root/ada3.key -s 4096 /dev/ada3

# geli attach -k /root/ada1.key /dev/ada1
# geli attach -k /root/ada2.key /dev/ada2
# geli attach -k /root/ada3.key /dev/ada3

# zpool create storage raidz ada1.eli
# zpool create storage raidz ada2.eli
# zpool create storage raidz ada3.eli
# mkdir /storage
# zfs mount storage

#!/usr/bin/env zsh
# decrypt and mount the storage pool

set -e

read -s pass\?'Storage Decryption Password: '; echo

echo -n "$pass" | geli attach -j - -k "/root/geli/ada1.key" "/dev/ada1"
echo -n "$pass" | geli attach -j - -k "/root/geli/ada2.key" "/dev/ada2"
echo -n "$pass" | geli attach -j - -k "/root/geli/ada3.key" "/dev/ada3"

zfs mount -a
service jail start

# zfs create storage/home

# cp -rp /home/* /storage/home
# umount /usr/home
# rm -rf /home /usr/home
# ln -s /storage/home /home
# ln -s /storage/home /usr/home

• https://docs.freebsd.org/en/books/handbook/jails/

I use FreeBSD's jail subsystem to isolate various self-hosted services from each other and the host system. This means that if one of them is compromised the attacker won't have access to everything else, and it also makes it easy for to keep a clean install which is important when fiddling with software written clunky, hard-to-manage languages like Python, Ruby, and JavaScript.

To create new jails I wrote this little script. After running that we append the corresponding entry to /etc/jail.conf, add the jail name to jail_list in /etc/rc.conf, and restart the service. Since all jails run network services we also need to route packets or forward HTTP requests, which will be covered in the next section.

/etc/jail.conf

# set filetype=conf

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

host.hostname = $name;
path = "/storage/jails/$name";
exec.consolelog = "/var/log/jail_${name}_console.log";
exec.prestart = "cp /etc/resolv.conf $path/etc";
exec.poststop = "rm $path/etc/resolv.conf";

I prefer to keep jails as small and disposable as possible, so any important data should be stored in directories mounted to the jail using the mount.fstab directive. Similarly, I manage SSL certificates in the host system.

In order to facilitate updating jails, I wrote the following script and corresponding monthly entry in the root user's crontab.

/usr/local/sbin/update-jails.sh

#!/bin/sh
# Automatically update all the running jails
# crontab -e:
# @monthly	/usr/local/sbin/update-jails.sh

find /storage/jails -type d -depth 1 | while read -r j; do
	n="$(basename "$j")"
	jls | grep "$n" >/dev/null && {
		freebsd-update -b "$j" fetch --not-running-from-cron
		freebsd-update -b "$j" install
		pkg -j "$n" upgrade
	}
done

service jail restart

• https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/

For the sake of convenience, we'll use /etc/hosts to map domain names and aliases to the internal IP addresses of the appropriate jails.

/etc/hosts

::1		ulthar.xyz	localhost
127.0.0.1	ulthar.xyz	localhost

In order to run network services in jails, we need to set up a NAT to send traffic from external ports to the appropriate jails, and vice versa.

/etc/pf.conf

# MACROS/TABLES
XIF = "re0"
JAILNET_V4 = "127.1.1.0/24"
JAILNET_V6 = "fd00:1:1:1::0/64"
EXT_V6ADDR = "2001:db8::1"

# TRANSLATION
## NAT
nat on $XIF inet from $JAILNET_V4 to any -> ($XIF)
nat on $XIF inet6 from $JAILNET_V6 to any -> $EXT_V6ADDR

## REDIRECT (Port Forwarding)

# FILTERING (Pass/Block)

Pf doesn't know anything about higher-level protocols like HTTP, so in order to run multiple web-services we need a reverse-proxy. Using Nginx for this rather than a dedicated program like Haproxy also means that I can simply serve my personal website from the host system.

/usr/local/etc/nginx/nginx.conf

# set filetype=conf

http {
	sendfile      on;
	include       mime.types;
	default_type  text/plain;
	keepalive_timeout 65;

	server {
		server_name ulthar.xyz;
		listen 80;
		return 301 https://ulthar.xyz$request_uri;
	}

	server {
		server_name ulthar.xyz;
		listen 443 ssl;
		root /storage/home/thalia/public_html;
		ssl_certificate /usr/local/etc/letsencrypt/live/ulthar.xyz/fullchain.pem;
		ssl_certificate_key /usr/local/etc/letsencrypt/live/ulthar.xyz/privkey.pem;
	}
}

• https://certbot.eff.org/lets-encrypt/freebsd-nginx.html

In order to simplify the certificate renewal process I register them in the host systema and do all the HTTPS redirection for other web services in Nginx.

# certbot certonly --standalone

This will store certificates and keys like so:

/usr/local/etc/letsencrypt/live/
|- README
|- domain.tld
   |- README
   |- cert.pem
   |- chain.pem
   |- fullchain.pem
   |- privkey.pem
 ...

In order to automate renewal I wrote a short script and added it as a weekly cron job, it should renew the certificates if they're set to expire within 30 days.

/usr/local/sbin/renew-certs.sh

#!/bin/sh
# Automatically renew all certificates
# crontab -e:
# @weekly	/usr/local/sbin/update-jails.sh

service nginx stop
certbot renew -q
service nginx start
service jail restart

Ejabberd, running in the XMPP jail, requires access to the certificates but doesn't have permissions to read them even when they're mounted. Rather than give them unsafe permissions, I added a startup script to the jail responsible for cp'ing and chown'ing the certificates; hence the directive to restart the jail daemon following renewal.