I'm guilty of not reading the f..ing documentation
I'm kind of that way. I will browse documentation, get a good idea as to what has to happen, then I raw dog it. Then, after many failed attempts, I go read the documentation. I agree with twinnie@feddit.uk tho, a ton of documentation either assumes you are a certified, dyed in the wool, sysadmin veteran with a wall of certs, or it's just too sparse for me to put together.
Oh it happens to the best of us. I was working on a simple cron the other day with the cron string that would insert the cron into my cron config something like 'echo' and the normal string you'd recognize, and ended with a '-'. I wasn't paying attention and issued the command which did insert itself into the cron config, but in a manner in which I didn't want. It replaced the whole cron file with that one string. #$@^$$ Luckily I have a cron to back up the crontabs.
I’m currently self-hosting several services and looking to harden my setup
If you are looking to harden your server, might I suggest installing Lynis. Lynis will extensively scan your server, and at the end of the scan it will print the output of the test including a score of 1-100 and recommendations on how to fix, secure and harden the server. Not all of the recommendations will be applicable to you.
A proper WAF (Web Application Firewall)
I use Crowdsec. While Crowdsec is not a trad WAf, it is more than capable when set up correctly. I use Crowdsec in conjunction with UFW, Fail2ban, Tailscale, rkhunter, and chkrootkit. I have fail2ban in aggressive mode, since I am the only user, or legit user that is. ;) I have been accused of going overboard on security.
Traefik, BunkerWeb, and Pangolin.
You mentioned these, and I have never used them. I'm assuming you are going for a reverse proxy as part of your hardening methods. I use Caddy, and it's real simple to set up. It also takes care of your cert renewals automatically.
Bonus: nice dashboard or at least logs that make sense
Most of the logging apps like Syslog, Graylog, or similar I've found to be quite heavy as they need a lot of additional modules to run like databases, elastic, et al. I recently discovered lnav, with the help of the kind folks here. It is not a pretty, graphical, dialed out dashboard. You view the logs in the terminal. Very light on resources, and does exactly what it says on the tin. Check it out.
Tailscale
Do it! It's easy to set up and works very well. You can pipe all manner of things through Tailscale like ssh, sftp, etc.
Also, don't forget about using ssh keys. I know there is a lot of discussion about changing the ssh port number and how effective it really is. For about 5 minutes of your time, you can have it all set up, and it will at the very least, cut down a lot of noisy bots. If you want to go even further, you can set up host allow/host deny: sudo host.allow and sudo hosts.deny. Make sure you edit host allow first. LOL
You may want to look into Debsums, AIDE, iptraf-ng, Apparmor Deborphan, unattended updates, Maldet, etc, which will probably recommended by Lynis when you initiate a scan.
I recommend Caddy. It's very easy to deploy, and configuring it is a snap. This tutorial helped me out a bunch. There is a Docker version of Caddy, tho I have never used it. I figured, Caddy would do better installed on bare metal. I use Caddy in conjunction with Duckdns.org. Caddy also takes care of renewing your certs when it's time.
I think it’s overkill for homelab
How about a remote VPS?
Do you have a particular risk that you are worried about?
A couple of the Docker compose files I've used have non-hashed secrets in the compose itself. I am assuming, should someone penetrate the firewall, and gain access to Portainer somehow, they could see these compose entries just like I can. While I feel like I have adequately hardened the server (Lynis reports a score of 87) and I have rather robust ids/ips, firewall, and assorted accoutrements to support a secure server, there's always that 'what if' scenario running in my brain and it causes doubt. Perhaps a secrets manager is over kill for a single user, docker container server.
“Don’t feel crazy/bad/dumb, I’ve had the same thing happen to me!”
There you go. As far as 'my area' I didn't grow up in the US or any particular area. I grew up around the world and multiculturaly, so there is no telling where I picked that up at. LOL
The way you go about it on your wiki, is almost the same process/format, tho not as fancy . I'll even throw in a couple links to tuts I found useful for that particular segment in the notes.
was some variation of crazy/bad/dumb?
No, no, no. I wouldn't call you crazy or dumb. It was meant as 'don't feel singled out' or 'don't feel like you're the only one'.
NOTE: The instructions aren’t exactly difficult! So, I don’t see how I’d have gotten it wrong!
Dude, don't feel pregnant. It took me an embarrassingly long time to wrap my noodle around Caddy. Seriously, I just couldn't grasp what was going on in the Caddyfile. Then, after extensive trial and error, I happened upon one tutorial that changed everything. Now it's so simple for me, but at the time, I felt like a complete dumbfuck.
Is it a single server?
Well I run a hybrid set up of 3 VPS and one rack in the closet.
Maybe something like sops is all you need
Unpack that a little for me if you would.
That's a K8s application I've never explored, and the 'commit the encrypted secrets to git and deploy with ArgoCD' is well above my pay grade at the moment. Not saying it's not unattainable, however, I'm still trying to thoroughly understand Docker, which should take me quite a while. LOL
Ansible is one of those 'on the list' things to check out. It seems to have a broad range of applications.
as they are often stored unencrypted by the service/app that needs it
That's what I'm worried about
An encrypted disk might be better in that case.
See, that is one of the ideas I've been toying with. I've never encrypted a remote VPS. I have encrypted all my local drives. There is some response loss on local drives it seems to me. Negligible, but still. I'm just wondering if there would be a performance hit.
Thinking about getting some or most of this over to a service like hetzner, perhaps even splurging on a baremetal dedicated system.
If I may, I find LUXVPS to be quite capable and responsive hosts.
Black Luxury Deal #1
4 vCores (Xeon Gold 6150)
26 GB DDR4 RAM
150 GB Raid 1 NVMe
1 Gbit internet speed | 40 TB Traffic
1x IPv4
1x /64 IPv6
3.2Tbit Premium DDoS Protection
24/7 Ticket Support
4 Backups
For ONLY 10€/Mo (recurring)
I've never used Hetzner, and I don't know what you are hosting, but I'm sold on LuxVPS. I also use Contabo, and Ethernet Services. The latter would indeed be bare-bare-metal as there are no frills. However, for a test server and for $35 a year, it works.
proxmox
You will enjoy Proxmox. When you get it all jammy, check out the Proxmox Helper Scripts: https://community-scripts.github.io/ProxmoxVE/
You can restrict Caddy access to use your tailscale. For instance in your Caddyfile:
For tailscale ip range:
myverycoolserver.duckdns.org {
@allowed {
remote_ip 100.64.0.0/10 # Allow Tailscale IP range
}
respond @allowed 200 # Allow access
respond 403 # Deny access for others
reverse_proxy localhost:YOUR_SERVICE_PORT # Your service configuration
}
For specific tailscale IP:
myverycoolserver.duckdns.org {
@allowed {
remote_ip YOUR_TAILSCALE_IP # Replace with the specific Tailscale IP
}
respond @allowed 200 # Allow access
respond 403 # Deny access for others
reverse_proxy localhost:YOUR_SERVICE_PORT # Your service configuration
}
I am the note taking king probably. I worked in the construction industry for 20 years. The rule was, 'if you didn't write it down, it didn't happen.' That has just carried over to every other aspect of my life including selfhosting. Whenever I sit down to my terminal to do anything, I open Notepad++ and a regular windows notepad session. The windows notepad session is a little script I came up with that opens windows notepad with 1000 empty lines. It's one of the many quirks I have, but I hate having to hit the enter key to start a new line. I like to be able to click on a new line for a new line of thought and start typing.
@echo off
(for /l %%i in (1,1,1000) do echo.) > empty_lines.txt
start notepad empty_lines.txt
(Save as a bat link on desktop)
Anyways, the Notepad ++ session is for after things get worked out, I make an official entry into the Notepad++. The windows notepad session is just a scratch pad or 'thinking paper' from which I transfer to the Notepad ++ doc. Convoluted, no? LOL You asked, and I just pulled back the curtain for you a bit. Careful what you ask for, could stain your brain.
I try to document everything. I feel like, if I'm going to take the time to learn something, I might as well write it down. I take my Grok sessions and distill them down if I found the info relevant. I also do all of this because after my TBI which gave me a seizure condition as well as other mental/neuro issues, my memory is shit, even for someone of my age bracket. But I can stand up a server and secure it, just from my notes in a step by step manner conducive to my limited mental acuity. I've often wondered if anyone would be interested in my notes, like maybe some newcomer to selfhosting wouldn't have to reinvent the wheel since I have a penchant for fucking things up.
Interesting. I read a lot. Probably tb's of data per day. I don't watch tv not even news or weather. It's not a religious thing and it doesn't make me holier than thou. I just find that reading is best for me. However, if you hand me a traditional book, I will never crack the binding. Put that same book in a digital format that I can read from my devices, and I'll read it cover to cover and probably storage the document to read later.
We're all kind of quirky and we all have our own optimum way to learn. Mine is usually just screwing shit up until I get it.