Basic FreeBSD Setup
This file documents how I've configured the FreeBSD install on my home server. It is an amateur's collection of notes, not a guide. Use it at your own risk.
The root file system lives on a small, unencrypted SSD while everything important (user home directories, shared folders, and jails) live in a ZFS pool on three large, external HDDs mounted on /storage. This machine contains personal backups, archives of media, and several jails running network services.
Most configuration files presented here just contain stubs and skeletons, all the meat is present in the "configuring the host system" section of other service-specific writeups.
Preliminary Setup
Starting Services
FreeBSD uses a simple, old fashioned init system with /etc/rc.conf being the initialization file for services that are run after booting up. We use it to start and instruct daemons and configure the machine's network interfaces.
/etc/rc.conf
# Change some default behaviors clear_tmp_enable="YES" sendmail_enable="NONE" dumpdev="NO" # Start some general services zfs_enable="YES" sshd_enable="YES" ntpd_enable="YES" powerd_enable="YES" smartd_enable="YES" # Start services for hosting stuff pf_enable="YES" jail_enable="YES" nginx_enable="YES" pflog_enable="YES" # Jail configuration cloned_interfaces="lo1" jail_list="" # <- Append jails # Set a hostname and static IP hostname="monolith" ifconfig_re0="inet 192.168.5.19 netmask 255.255.255.0" defaultrouter="192.168.5.1"
Some nice-to-haves in the Environment
# pkg install neovim rsync git ranger w3m nginx py38-certbot
FreeBSD comes with three shells: sh, csh, and tcsh. The root user defaults to tcsh, and I'll stick with that for the sake of simplicity. With some minor quality-of-life improvements in /etc/csh.cshrc, tcsh makes for a really nice, interactive shell.
/etc/csh.cshrc
umask 22 alias ls ls -GF alias rm rm -i alias mv mv -i alias cp cp -i # The neovim and nginx-full packages have incompatible Lua dependencies, # so $EDITOR should gracefully downgrade to vim then vi. setenv EDITOR vi command -v vim >/dev/null && setenv EDITOR vim command -v nvim >/dev/null && setenv EDITOR nvim # Settings for the interactive shell if ($?prompt) then set filec set autoexpand set autorehash set promptchars = "%#" set history = 1000 set autolist = ambiguous set savehist = (1000 merge) set prompt = "%{\033[32m%}%B%n@%m%b%{\033[0m%}:%~ %# " # Set a distinctive prompt if we're running in a jail ps 1 >/dev/null || set prompt = "[JAIL]$prompt" # Plain old csh doesn't support these if ($?tcsh) then bindkey "^R" i-search-back bindkey "^W" backward-delete-word bindkey -k up history-search-backward bindkey -k down history-search-forward endif endif
Encrypting the Disks
Initialise partition tables on the hard drives, generate master encryption keys, then encrypt them
# gpart create -s GPT ada1 # gpart create -s GPT ada2 # gpart create -s GPT ada3 # dd if=/dev/random of=/root/ada1.key bs=64 count=1 # dd if=/dev/random of=/root/ada2.key bs=64 count=1 # dd if=/dev/random of=/root/ada3.key bs=64 count=1 # geli init -K /root/ada1.key -s 4096 /dev/ada1 # geli init -K /root/ada2.key -s 4096 /dev/ada2 # geli init -K /root/ada3.key -s 4096 /dev/ada3
Note that the disk metadata is backed up at /var/backups/ada?.eli
and may be restored like so: geli restore /var/backups/ada?.eli
/dev/ada?
Now decrypt the volumes, producing /dev/ada?.eli devices
# geli attach -k /root/ada1.key /dev/ada1 # geli attach -k /root/ada2.key /dev/ada2 # geli attach -k /root/ada3.key /dev/ada3
Creating a ZFS Pool
Add the decrypted devices to a ZFS pool and mount it
# zpool create storage raidz ada1.eli # zpool create storage raidz ada2.eli # zpool create storage raidz ada3.eli # mkdir /storage # zfs mount storage
In the future the process of decrypting and mounting can be automated with this script. I'm using zsh because the POSIX read builtin doesn't have a flag to avoid echoing the password back to the terminal, and I have zsh installed anyway as my preferred interactive shell.
/usr/local/sbin/mount-storage.zsh
#!/usr/bin/env zsh # decrypt and mount the storage pool set -e read -s pass\?'Storage Decryption Password: '; echo echo -n "$pass" | geli attach -j - -k "/root/geli/ada1.key" "/dev/ada1" echo -n "$pass" | geli attach -j - -k "/root/geli/ada2.key" "/dev/ada2" echo -n "$pass" | geli attach -j - -k "/root/geli/ada3.key" "/dev/ada3" zfs mount -a service jail start
Moving Home Directories to the Storage Pool
Create a new dataset so that the home directories can be managed independently of the other files on encrypted drives.
# zfs create storage/home
Copy the users' home directories over and set up symlinks
# cp -rp /home/* /storage/home # umount /usr/home # rm -rf /home /usr/home # ln -s /storage/home /home # ln -s /storage/home /usr/home
Now reboot the machine, log in, run the mount script, log out, and log back in. Everything should be working.
Isolating Services with Jails
I use FreeBSD's jail subsystem to isolate various self-hosted services from each other and the host system. This means that if one of them is compromised the attacker won't have access to everything else, and it also makes it easy for to keep a clean install which is important when fiddling with software written clunky, hard-to-manage languages like Python, Ruby, and JavaScript.
To create new jails I wrote this little script. After running that we
append the corresponding entry to /etc/jail.conf, add the jail name to
jail_list
in /etc/rc.conf, and restart the service. Since all
jails run network services we also need to route packets or forward
HTTP requests, which will be covered in the next section.
/etc/jail.conf
# set filetype=conf exec.start = "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; exec.clean; mount.devfs; host.hostname = $name; path = "/storage/jails/$name"; exec.consolelog = "/var/log/jail_${name}_console.log"; exec.prestart = "cp /etc/resolv.conf $path/etc"; exec.poststop = "rm $path/etc/resolv.conf";
I prefer to keep jails as small and disposable as possible, so any important data should be stored in directories mounted to the jail using the mount.fstab directive. Similarly, I manage SSL certificates in the host system.
In order to facilitate updating jails, I wrote the following script and corresponding monthly entry in the root user's crontab.
/usr/local/sbin/update-jails.sh
#!/bin/sh # Automatically update all the running jails # crontab -e: # @monthly /usr/local/sbin/update-jails.sh find /storage/jails -type d -depth 1 | while read -r j; do n="$(basename "$j")" jls | grep "$n" >/dev/null && { freebsd-update -b "$j" fetch --not-running-from-cron freebsd-update -b "$j" install pkg -j "$n" upgrade } done service jail restart
Sending Packets to Jails
For the sake of convenience, we'll use /etc/hosts to map domain names and aliases to the internal IP addresses of the appropriate jails.
/etc/hosts
::1 ulthar.xyz localhost 127.0.0.1 ulthar.xyz localhost
In order to run network services in jails, we need to set up a NAT to send traffic from external ports to the appropriate jails, and vice versa.
/etc/pf.conf
# MACROS/TABLES XIF = "re0" JAILNET_V4 = "127.1.1.0/24" JAILNET_V6 = "fd00:1:1:1::0/64" EXT_V6ADDR = "2001:db8::1" # TRANSLATION ## NAT nat on $XIF inet from $JAILNET_V4 to any -> ($XIF) nat on $XIF inet6 from $JAILNET_V6 to any -> $EXT_V6ADDR ## REDIRECT (Port Forwarding) # FILTERING (Pass/Block)
Pf doesn't know anything about higher-level protocols like HTTP, so in order to run multiple web-services we need a reverse-proxy. Using Nginx for this rather than a dedicated program like Haproxy also means that I can simply serve my personal website from the host system.
/usr/local/etc/nginx/nginx.conf
# set filetype=conf http { sendfile on; include mime.types; default_type text/plain; keepalive_timeout 65; server { server_name ulthar.xyz; listen 80; return 301 https://ulthar.xyz$request_uri; } server { server_name ulthar.xyz; listen 443 ssl; root /storage/home/thalia/public_html; ssl_certificate /usr/local/etc/letsencrypt/live/ulthar.xyz/fullchain.pem; ssl_certificate_key /usr/local/etc/letsencrypt/live/ulthar.xyz/privkey.pem; } }
SSL Certificates with LetsEncrypt and Certbot
In order to simplify the certificate renewal process I register them in the host systema and do all the HTTPS redirection for other web services in Nginx.
# certbot certonly --standalone
This will store certificates and keys like so:
/usr/local/etc/letsencrypt/live/ |- README |- domain.tld |- README |- cert.pem |- chain.pem |- fullchain.pem |- privkey.pem ...
In order to automate renewal I wrote a short script and added it as a weekly cron job, it should renew the certificates if they're set to expire within 30 days.
/usr/local/sbin/renew-certs.sh
#!/bin/sh # Automatically renew all certificates # crontab -e: # @weekly /usr/local/sbin/update-jails.sh service nginx stop certbot renew -q service nginx start service jail restart
Ejabberd, running in the XMPP jail, requires access to the certificates but doesn't have permissions to read them even when they're mounted. Rather than give them unsafe permissions, I added a startup script to the jail responsible for cp'ing and chown'ing the certificates; hence the directive to restart the jail daemon following renewal.