My migration experience with SERVFAIL (and CertBot)

As I usually do, I was browsing Social Media on my way to vocational school, while sitting in the subway.
While looking through YouTube to find something interesting to spend the ride on, I stumbled over this C38C3 talk by sdomi, famfo, merlin. "we made a globally distributed DNS network for shits and giggles" was something that sounded more then interesting and something to feed my hunger for silly sysadmin/developer projects.

FYI I am not going to go in depth about the project, watch the talk for that :3
After I was done listening to it, that hunger wasn't clenched and I wanted to try it out for myself. I reached out to Merlin on Mastodon, asked for a access key for their beta and voila, I got access to SERVFAIL.

Migrating

I know this introduction was long, but I really wanted to empathise how I had no plans and found this project by pure luck.
Anways... armed with access, I did what every sane human would do and migrated my personal domain to this hobby project. Again, for no reason more then for shits and giggles.
This was suprisingly easy, because Cloudflare (my old "provider") allows for exporting and after reformatting it, I could plug it into the raw editor, BOOM all records migrated.

Not everything was as smooth sailing as that, because of some quirks that I didn't know or that hadn't been worked out.
Just as Cloudflare does, SERVFAIL allows for substitution of your domain via the @ sign.
I used this heavily when migrating my CNAME records, but I was unaware that this is only supported in the name of a record, not the value. This was fixed, after I noticed that my mailclient couldn't collect any mails from the server.
After reaching out to the servfail irc, I was informed that this feature hasn't been implemented, so I just search and replaced the @s with my root domain.

Setting up Certbot

Now to the jucier part, my SSL certificates.
I will assume that you know what Let's Encrypt, Certbot and a wildcard certificates is.
To get a wildcard certificate, you need to use DNS Verification and if you don't want to do this manually, you will need a plugin plus credentials.
Because SERVFAIL uses PowerDNS in the backend, we can use this plugin made by kaechele. It's important to note that this plugin needs Python 11 or 12!
We need to setup a config file in the ini format. This should be in a safe location, with minimal permissions, so that your api key stays safe!
The endpoint is https://beta.servfail.network, the api key can be generated on the SERVFAIL webui settings and last but not least server id is the primary nameserver you selected, when creating your zone.
The ini file should look like this:

1dns_pdns_endpoint = https://beta.servfail.network
2dns_pdns_api_key = vsaSXCroiuZ#ZzYDab&XCjxrvEo3k6ThQan9$&MSQSs6vDBUiapWfqGDaZbhge7!
3dns_pdns_server_id = ns1.famfo.xyz
4dns_pdns_disable_notify = false

Now you can get yourself a wildcard certificate like this:

1certbot certonly \
2    --authenticator dns-pdns \
3    --dns-pdns-credentials ~/.secrets/pdns-credentials.ini \
4    -d *.luiggi33.de

It could happen, that something changes and you want to verify that this process still works, just run:

1certbot renew --dry-run

Fun on the side: Automatic Backup

A little fun project I do wanna share is my little backup script that automatically saves my raw DNS records to a GitHub Repo via Actions.
It consists of a bash script and a GitHub Actions workflow that runs on a schedule.

 1#!/bin/bash
 2
 3# First we get ourselfs a cookie
 4cookies=$(curl -s --cookie-jar - 'https://beta.servfail.network/login/' -X POST --data-raw "user=$USERNAME&pass=$PASSWORD" | awk '{print $6 "=" $7}')
 5
 6# Extract em out of the curl cookie jar
 7while IFS='=' read -r name value; do
 8    [[ -z "$name" || "$name" =~ ^# ]] && continue
 9
10    declare "$name=$value"
11done <<< "$cookies"
12
13# Remove the old backup
14rm dns.bak 2> /dev/null
15
16# This gets the edit raw page, translates the double quotes html ent and extract the textarea field content
17wget -q -O - 'https://beta.servfail.network/zones/ns1.famfo.xyz./luiggi33.de./raw' --header="Cookie: sh_session=$sh_session; username=$username" | sed 's/&#34;/"/g' | perl -0777 -ne 'print $1 if /<textarea name="raw" autofocus>(.*?)<\/textarea>/s' >> dns.bak
18
19# Now we log out and invalidate the token :3
20curl 'https://beta.servfail.network/logout' -s -H "Cookie: sh_session=$sh_session; username=$username"

This is the mentioned workflow to run the script and push, when there are any changes to the backup

 1name: Run Backup Schedule
 2on:
 3  schedule:
 4    - cron: "0 0 * * *"
 5  workflow_dispatch:
 6permissions:
 7  contents: write
 8jobs:
 9  do-backup:
10    runs-on: ubuntu-latest
11    steps:
12      - uses: actions/checkout@v3
13      - name: Create backup file
14        env:
15          PASSWORD: ${{ secrets.PASSWORD }}
16          USERNAME: ${{ secrets.USERNAME }}
17        run: bash create-backup.sh
18      - name: Commit backup
19        run: |
20          if [[ $(git status --porcelain | wc -l) -gt 0 ]]; then
21            git config --global user.name 'github-actions[bot]'
22            git config --global user.email 'github-actions[bot]@users.noreply.github.com'
23            git commit -am "Push backup"
24            git push
25          fi          

This is all for now. I am working on some more fun stuff, but that needs to wait until its done :)

Thanks for reading my ramblings, I think you can notice that it's my first time actually writing a blog post :P