Ah, the trials and tribulations of upgrading Debian 12 Bookworm to Debian 13 Trixie! I learned a valuable lesson this weekend with the update - always read the apt listchanges email. Always.
The upgrade itself went smoothly with no errors during the packages upgrading/installing. I run three main services on my server - nginx, exim, and dovecot. On reboot, all three had issues with the upgraded packages.
I received the following error in my nginx error.log:
Noticed these lines in journalctl when nginx didn’t start after a reboot:
Dec 10 17:43:30 mail1 nginx[3485]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "ocsp.int-x3.letsencrypt.org" in the certificate "/etc/letsencrypt/live/www.example.com/fullchain.pem"
Dec 10 17:43:30 mail1 nginx[3485]: nginx: [emerg] bind() to [<IPv6 address>]:80 failed (99: Cannot assign requested address)
Dec 10 17:43:30 mail1 nginx[3485]: nginx: configuration file /etc/nginx/nginx.conf test failed
Dec 10 17:43:30 mail1 systemd[1]: nginx.service: Control process exited, code=exited status=1
Dec 10 17:43:30 mail1 systemd[1]: nginx.service: Unit entered failed state.
Dec 10 17:43:30 mail1 systemd[1]: nginx.service: Failed with result 'exit-code'.
Dec 10 17:52:35 mail1 systemd[1]: nginx.service: Failed to read PID from file /run/nginx.pid: Invalid argument
Hmmm…
I ran into two issues when setting up Let’s Encrypt SSL certificates on two of my servers - permission issues for Exim and the certbot cron job supplied by the package doesn’t handle the renew very well for nginx, exim or dovecot.
1. Create a new group. I named it sslcerts. Add the exim user to that group. If you’re not using Debian, adjust the user in the command below.
I ran into a problem with NGINX failing to start on boot/reboot on my Debian 8 (Jessie) server. After reviewing what seemed like a hundred sites to try to find a fix, I stumbled across one solution that worked, but was incredibly inelegant. This was to add:
RestartSec=30s
Restart=on-failure
to nginx.service in the [Service] section using the override.conf. It worked but didn’t fix the underlying problem.
A quick look using journalctl -u nginx showed that the service was failing because the IPv6 address hadn’t been assigned to the network adaptor yet. This caused nginx to fail because it couldn’t bind to the IPv6 port. Here are the log lines:
merge-ngx-conf.pl is a perl script used to assemble a set of nginx configuration files for one site. It has a number of options. See the bitbucket page or the help documentation in the script itself.
In its simplest form, it’s called by issuing this command:
merge-ngx-conf.pl /path/sites-available/filename
The output is an assembled nginx configuration file with all the includes inserted. Using nginx.conf and domain.conf (or just domain.conf depending on the options selected), the script iterates through the include directives in the files and inserts the text from the referenced file. The script handles wildcard masks and follows include directives down multiple levels (i.e. nested levels). It will also follow referenced files in directories external to the nginx configuration directory.