I have an Ansible playbook I run each weekend to do all the apt updates. As well as keeping everything up to date, it’s a good check-in that everything’s alive and working as expected. I have Uptime Kuma checking the services are alive, and that no one is running out of disk or memory so there shouldn’t be any drama right?
This weekend, three instances (two remote, one local) timed out with “unreachable”.
I have a sort of muscle memory for starting little web projects now. I seem to have landed on node/express SSR apps with HTMX sprinkles. So it goes a bit like this:
npm init in the directory to create the package.jsonpublic sub directory, and drop htmx.min.js in there, and create a styles.css there. I’m always conflicted about what to do about this htmx dependency. I’d rather host it rather than use their CDN because reasons . But I also feel bad about committing it on Github. I could .gitignore it, but then when I clone the project on the production server I’d need to add another step to download it. HTMX is only 44K, and Microsoft can afford the bandwidth, so for the moment I commit them, but I need a better solution for the future..DS_Store to .gitignore (which also creates it), then edit it to also ignore node_modulesnpm install expressnpm install ejsreadme.md
I’ve taken to running lots of my services in LXC containers under Proxmox. I like the feeling of installing in a VM, but it’s lightweight. I like the backups, I like things being isolated from each other, I like moving them around between machines easily. I’m just a big LXC lover at the moment.
I’m also a Tailscale lover, and the generous number of nodes in the free tier means I now just routinely install them in my VMs and containers without a thought.
I’ve got a domain that’s not currently used, so I’m going to set it up as a virtual host under NGINX. This server is already serving two domains set up with Certbot for SSL. Is it going to be possible to add another site and have Certbot manage the certificates for it after I’ve run Certbot once?
When I googled around to find out, I didn’t find anything - which is usually a sign I’m either asking a wrong question, or it’s so little drama that no one ever mentions it. I decided just to move the site, check it was all working for the http version, then run Certbot and see what it said.
I’ve been managing SSL certificates for my domains purchased from PorkBun by going there every 90 days downloading the certificates, joining them together to make the fullchain.pem then scp-ing them to my servers. That’s been sort of manageable, but less than ideal.
It also doesn’t work for my Australian domains. Since there’s strict rules about who can own a domain in the .au space (you have to have some sort of right to the name - a random person can’t obtain the coke.com.au domain unless that’s a trading name, a trademark, or something similar), they have to be managed by one of about eight organisations, and the offerings are much simpler.
Years ago, I was very keen on the SETI@home project that used a distributed computing model whereby packets of digitized received radio data were farmed out to individuals’ computers to be processed to look for any unusual signals that could potentially be from an intelligent extra-terrestrial source.
That’s long since defunct, but the idea lives on with BOINC - a system run out of Berkley that allows different science organisations to offer projects to run on individuals’ computers.
I’ve picked up an new TP-Link WAP with Omada, so I wanted to spin up an Ubuntu 20.04 LXC to run the controller software in, and ended up spending a couple of hours figuring out why things where not working.
The initial problem was I was having connectivity issues pulling down the updates for all the packages required. I went down a bit of a tangent because I installed an apt cache the other day, so I was looking for problems there. Eventually I narrowed it down to DNS not working and started A/B testing like this:
It’s bothered me for a while that all these VM’s are pulling down a lot of the same updates. As well as needlessly using some bandwidth, I’m hammering the update servers (that I don’t pay for) with the same requests over and over. I did briefly consider running my own mirror, but that’s not simple, plus I’d then be mirroring a heap of files in a complete repository that I’d never use. What I really needed was some sort of cache so once I’ll pulled down an update, it would hang around for a few days being available to other machines on the local network. Luckily, that exact thing exists - APT Cacher NG .
Having written my little monitoring endpoint in Go, it needs pushed out to all my servers and VM’s. Clearly this is a job for Ansible which I’ve already dabbled my toes in . Before we get onto doing that though, we need to have a think about how to make it a service.
A service in Linux is just a program, but one that’s usually required to be running all the time to provide some piece of functionality. The “program” can be any executable, but to allow systemd to manage it, we need to tell it a bit about what we want in a .service file. This file is used by systemd to know how to manage the service. They can get quite complex, but here’s the simple one for vitals-glimpse - my little monitoring API endpoint.
I’d like a small, quick, low load endpoint on all my nodes and VM’s that exposes a text keyword indicating if that machine is okay for RAM and disk space. I’m currently using Uptime Kuma to monitor if these machines are pingable, but I’d love a tiny bit more information from them so I’d get a Ntfy buzz on my phone if a machine is in trouble.
I mentioned a couple of weeks ago that the benefit of doing it in C rather than Node.js was probably not worth the trouble, but then being a fickle developer, decided to write it in Go.
If you create an unprivileged LXC container on Proxmox, then try to back it up to an NFS share, for example on a NAS, you’ll get an error when it tries to build the temporary file.
The clue is in the Permission denied line. It is trying to create a temporary file on my NAS, and failing because of a permissions problem. If I try the same backup to the local storage, it works fine.
If you’ve got a script, or some code to work on, and it’s on a VM somewhere, you can always ssh in and use nano or vim to make your edits. Like a caveman. With an archaic editor, no intellisense, and no spell checking.
Or….
This magic - of editing a files on a remote server over SSH is achieved by using a Microsoft plugin for VS Code - “Remote - SSH ”
This always makes me laugh:
It’s like half the traffic on the internet is bots trying random passwords on root accounts over ssh. This is on an Ubuntu VPS on BinaryLane that had only been spun up five minutes or so. Looks like about one attempt every 10 seconds.
This is why the number three thing on my new install list is to disable root access via ssh. Here’s my system - possibly just for Ubuntu and related systems:
I’ve been using the excellent Uptime Kuma for my monitoring, but a couple of recent incidents - an external USB mount disappeared on a remote machine, an NVME drive filled up on a different node and stopped backups working because of a configuration error - have made me start to think about more robust monitoring.
The are many great tools for this - Nagios , Prometheus etc. but they are pretty substantial time investments for the excellent power. They can save time series data and display them beautifully. However, all I really want is to add some extra ability to Uptime Kuma.
I’m up to the point in a web app where it needs to come off my lan and into the hands of a couple of users for alpha feedback. Before that happens, I have to add some sort of login/authentication system since it I want to use real, sensitive data. There’s lots of detailed blog posts and videos of how to implement this in an Express app with passport, but what I was missing was the big picture of what actually needs to happen.
I couldn’t help but measure some VPS disk speeds while I was busting out the fio.
Binary Lane only claims “pure SSD drives” but seems pretty great. The difference between Digital Ocean SSD and NVME is disappointing. Obviously you’re sharing a drive with other users, so perhaps this depends on what else is going on.
If you clone an old node project and npm install it, you’ll most likely get a bunch of errors and warning messages. If you just decide to yolo it and run the project, you’ll get a bunch more.
I’ve been doing this exact thing. I want to add some auth to my app, and I’ve been following WebDevSimplified ’s video about using passport . I was building into my app without really understanding what I was doing, ran into problems and decided just to clone his repo and integrate the code into my app. The repo is four years old.
Now I’ve added NVME drives to my nodes, plus added an external NMVE RAID, I’ve got quite the collection of storage options. For one of my nodes, it looks like this:
For my dev VM’s I often set them up to have their storage on the NAS - it’s just super easy to move them around then. The production VM’s currently have their storage on the SSD (that machine hasn’t had the NVME upgrade yet), but obviously with all these options, it’d be interesting to think about what goes where.
When I popped in an NVME drive and freshly installed Proxmox to it, I assumed I’d just be able to wipe the SDD that had previously been the boot drive to set it up as a ZFS pool. However, when I tried to do the wipe, I was greeted with the error:
disk/partition '/dev/sda3' has a holder (500)
I assume this means there’s a flag set on one of the Proxmox partitions to prevent accidental deletion or Proxmox thought that’s where it was running from. It’s likely that it’s related to this message I had during installation that I haven’t seen before:
As part of my strategy to not worry about the slightly dodgy SMART reporting on the SDD’s in my HP Elitedesk G2 800 Mini Proxmox nodes, I’d decided to make use of the full sized M.2 slot to install 256GB NVME drives. That way I can boot from those, and have the SSD’s running ZFS which allows scrubbing to check the integrity of all the data. My VM disks can live on this drive.