Homelab on dev.endevour

Getting Ghostty to Work on Synology

Mon, 28 Jul 2025 00:00:00 +0000

Ghostty is a terminal application that I don’t really need (it’s listed features either already exist in the MacOS terminal, or seem so esoteric or marginal that I can’t imagine any real benefit from them in my normal use), but I wanted to be one of the cool kids, so I thought I’d give it a try.

After fiddling around with the themes for a bit I renamed it to ’term-ghosty.app’ so I’d remember to use it (ie when I pop up spotlight and type ’term’ it will come up) and got on with my day. Ten minutes later I’d run into a problem.

Manually adding SSL certs in Nginx Proxy Manager

Mon, 31 Mar 2025 00:00:00 +0000

A large part of the reason for my use of Nginx Proxy manager over vanilla NGINX, is that it has built-in Let’s Encrypt certificate requesting and renewing. This works perfectly for all my public facing services, and until recently, my homelab services. Before I dive into how I’ve fixed the problem I ran into, I better explain how my homelab domain is set up, and before I do that, an over-simplified description of how the SSL system works is required

Command chaining with NTFY for long running commands

Mon, 03 Feb 2025 00:00:00 +0000

NTFY is a great open-source push notification service that’s self-hostable or free to use (although I suggest you pay for it as I do). I’ve written before how I use it with UptimeKuma for my uptime monitoring, but another common use is just when I’m initiating long-running commands and backgrounding them.

This magic is possible since we can just curl to send a NTFY notification. For example:

curl -d "😀 demo push message via NTFY" ntfy.sh/blog_demo

Since I’m subscribed to the “blog_demo” topic in NTFY, this message will be pushed to my phone and watch:

Share files securely with Enclosed

Mon, 27 Jan 2025 00:00:00 +0000

My accountant works for one of those giant firms, and it bugs me that I’m emailing him password protected zip files of my accounts rather than to a secure upload facility at his firm. I can fix this with the power of self hosting, by running my own secure file dropping app on a VPS.

There’s a number of applications that do this sort of thing - allow you to upload a file, get a link in return which you can then share to people to download the file. For this to be more secure than emailing, the file needs to be encrypted on the server, and we want to be able to set a password, impose limits on downloads, and limit how long the link lives for. I’ve previously looked at Sharry which adds the ability for unauthenticated users to upload files to you securely, but for this slightly simpler job, I chose Enclosed by Corentin Thomasset .

NGINX proxy manager - setting headers to use basic auth in your apps

Mon, 09 Dec 2024 00:00:00 +0000

When I’m spinning up side projects, I frequently ignore auth, and just rely on NGINX basic auth - one of the side benefits of reverse-proxying everything.

Regular NGINX

This article in the docs explains how to set up basic auth to protect different paths. To make it work in my node apps, I need the successful user name passed in so I check it against the user table to work out access rights etc.

rsync between Synology NAS

Mon, 30 Sep 2024 00:00:00 +0000

A while ago, I devised a complicated system where I could drop files in a web interface running on an LXD container and the files would then magically appear in a directory on a remote NAS in the morning. It turned out to not be very robust, and I gave up on it after a while.

Also, really there should be no need for it - underneath, it was just using rsync to move the files, so why not just do that direct from one NAS to another? Well, mainly because my NASs are all Synology - which I love, and they’ve been great, but in an effort to make them usable by muggles, Synology tend to somewhat complicate things for Linux command line wizards.

Containerised NGINX Proxy Manager & the 502 error

Mon, 16 Sep 2024 00:00:00 +0000

If you’re used to running NGINX Proxy Manager in front of your web apps, and switch to running it in a container, you’re going to need to learn a little about Docker networks to get everything connected. If you just do your regular setup, and direct the proxy for an address to 127.0.0.1:<some port>, it won’t exist, and you’ll visit your page to find the “502 Bad Gateway openresty” message.

Moving from Docker volumes to bind mounts

Mon, 05 Aug 2024 00:00:00 +0000

When I started with Docker, the docs seemed to suggest that using Docker volumes was a good thing. With a Docker volume, you just create the volume and Docker manages the rest. You don’t have to worry about where it is, or really ever think about it.

Here’s a docker-compose for Uptime Kuma using a volume.

services:
 uptime-kuma:
 image: louislam/uptime-kuma:1
 container_name: uptime-kuma
 volumes:
 - kuma_data:/app/data
 ports:
 - 80:3001
 restart: unless-stopped

volumes:
 kuma_data:

This is telling Docker we want to create a volume called “kuma_data” and then map it into the container file system at /app/data

Upgrading to Forgejo 7.0.1

Mon, 06 May 2024 00:00:00 +0000

It’s not that long ago that I wrote about doing routine upgrades on containerised web apps using Forgejo as an example as I upgraded Forgejo (my git repository manager) between patch versions of 1.21, then a few days later, they dropped 7.0.0

They say the major version jump is due to it being an LTS (long term support) release, and changing to semantic versioning 2.0.0 , but that doesn’t quite explain it to me, and I assume this is partly signifying the fork’s drift away from the gitea codebase. In any case, the upgrade to 7.0.0 it does involve some breaking changes, and signifies to me that a lot has been on, which makes me keen to wait for a patch release (I’m always keen for other people to debug these things) which has now landed.

Virtual Hosts on "Static Web Server"

Mon, 22 Apr 2024 00:00:00 +0000

I’ve been running NGINX Proxy Manager (NPM) in my homelab for a bit, and I’ve been meaning to clean up the VPS that runs most of my websites and public facing servers, so I’m considering running NGINX Proxy Manager on that VPS. While NGINX Proxy Manager wraps up the configs in a beautiful GUI, in the process you lose some of NGINXs capabilities. In particular there’s no GUI way to serve static virtual hosts from NGINX Proxy Manager.

NGINX Proxy Manager

Mon, 15 Apr 2024 00:00:00 +0000

I’ve mentioned using NGINX as an interface between the internet and a service a while ago. This works by all incoming traffic coming to NGINX, and NGINX determining which service that traffic should go (from the NGINX config files) then acting as a middleman. This functionality is generally referred to as a ‘reverse proxy’.

This is nice for a few reasons:

Due Diligence on a Docker Image

Mon, 08 Apr 2024 00:00:00 +0000

Photo by Brett Jordan on Unsplash

I need a survey tool, and a quick search turned up LimeSurvey , there’s a ‘community edition’ so naturally I plan to self-host it. I scrolled down to the ‘installation’ section of the manual which has a big list of PHP dependencies.

Ain’t nobody got the time for that in 2024, I scroll further looking for the docker-compose but there isn’t one. Huh. No official Docker image.

My Web App Update Process

Mon, 01 Apr 2024 00:00:00 +0000

I’ve settled on a very standard, reproducible setup for services in my homelab. This post looks at that, then runs through the update I did today to Forgejo which only took a few minutes and felt relatively risk free.

Standard Setups

My system is based around Proxmox. I have three physical machines - one for production apps, a production spare, and a development/testbed machine. A Synology NAS serves for backups. Moving a VM or LXC between the machines is trivial; but it’s done manually - the machines are not clustered for high availability.

Deploying a Node app in Docker

Sun, 31 Mar 2024 00:00:00 +0000

When I wrote the install instructions for mdserver (little Markdown server Node app) on it’s github page it was something like:

Have node.js installed and working
Clone the repo
Start with npm start

Which is great if you know how to do those things (they are bread and butter to a web dev) but not if you’re a self-hoster who just wants a web server that converts markdown to HTML on the fly. For any situation where you just want to use the app, what you probably want is a Docker image of the app.

Hosting Your Own Docker Registry

Mon, 25 Mar 2024 00:00:00 +0000

The Docker Personal (ie free tier) plan currently allows one private repository, but even if you want to pay for the next level where you can have unlimited repositories, you may still want to host your own private registry - it’s going to be quicker inside your network, and you won’t run up against Docker’s pull/push limits if you are hammering it with your CI/CD system.

Fly.io, Uptime Kuma & scraping a status page

Fri, 02 Feb 2024 00:00:00 +0000

I’ve been aware since I set up Uptime Kuma for my monitoring, that having an instance on my local network monitoring my VPS websites wasn’t ideal. The main reason being that the flakiest part of my infrastructure is my 4G home internet, so if that goes down I have no website monitoring, and even if I did, the notifications couldn’t get out.

Of course, it would also be a simple matter to run an instance on the VPS that I host the sites on, but that has a similar problem in that if the VPS goes down, so does my monitoring of the VPS. What I really need is a third, independent space to run an instance.

What's unfinished in your Udemy?

Fri, 19 Jan 2024 00:00:00 +0000

If you work or study in tech, I always feel a good getting-to-know-you question is “what courses or tutorials did you start, but not finish?”

My Udemy doesn’t look too bad:

The ZTM course was good, but I got stuck on an AI API exercise. I think it’s a common sticking point for students since Andrei includes a little rant about how it definitely does work - but I downloaded his repo with the solution and it was having the same errors I was and I gave up in frustration. I probably should have just skipped that one.

Using LXC templates in Proxmox

Sun, 24 Dec 2023 00:00:00 +0000

I wrote a couple of weeks ago about a standard workflow I use to spin up a web service in an LXC container to add to my self-hosted collection of services. It went a bit like: do this, and then this, then this other thing. Whenever you find yourself repeating a set of steps like this, it’s usually a sign that you should be automating it. Not just to save time (although this is a key benefit) but also to improve repeatability and to avoid introducing errors.

Practice your restore strategy

Thu, 21 Dec 2023 00:00:00 +0000

My homelab set up is a production node, (pve-prod1) a backup production node (pve-prod2) and a development machine (pve-dev1). They are all G2 800 minis, but pve-prod1 has a i7 6700T and 32GB RAM, where as the other two are i5 6500T with 16GB. My thinking is that the older two can easily share the workload of the main production machine for disaster recovery. Everything is virtualised on top of Proxmox, so sharing up the VM’s and containers is trivial.

Gogs, Gitea, Forgejo

Mon, 18 Dec 2023 00:00:00 +0000

I’ve been really pleased with Gogs - it’s lightweight, was simple to spin up, and has worked perfectly. But then this morning on Mastodon, there’s a post from @Codeberg.org describing a security vulnerability in their Git hosting project Forgejo. This issue also apparently affects Gitea and Gogs - what’s up with that?

I actually already did spend a bit of time comparing Gogs and Gitea before deciding on Gogs, since I’d heard of people running Gitea over the past year or so, but only seen that Gogs seemed to be popular with self-hosters in a Lemmy post I’d read. My first impression was that Gitea was more focused on CI/CD and seemed to have a more complicated install process.

Gogs - your own tiny GitHub

Wed, 06 Dec 2023 00:00:00 +0000

(edit: - I’ve had a rethink about my source hosting)

Once you’re familiar with coding tools, like the excellent VS Code , and git , it’s immediately apparent that these tools can be applicable for other purposes. A great example is that I now do my financial accounting in plain text (using beancount ). I have a python script that converts by bank account data in to the beancount format text files, I edit them in VS Code with a plugin that does the syntax highlighting and checks everything balances.

New Self-Hosted Service Workflow

Sun, 03 Dec 2023 00:00:00 +0000

I’ve developed a bit of a workflow for setting up a new service of some type on the homelab. Installing it is the obvious thing, but I also have a few quality of life things I do to make it a full production-quality part of my installation. I thought it might be helpful to run through those things using a recent example of adding audiobookshelf .

audiobookshelf

audiobookshelf is a web based system for viewing, playing, downloading and/or generally managing your audio books. I’ve been an Audible user/subscriber, but recently got grumpy at them about something - I think I had paused my subscription, and my downloaded books were still available on my phone. I was halfway through one, upgraded the app, and then wasn’t able to play the book without re-subscribing. That might not be exactly right, but it was some type of frustrating carry on like that.

ViewTube

Mon, 27 Nov 2023 00:00:00 +0000

Whenever I encounter one of those “What are you self-hosting?” threads, I know I’m about to waste an hour looking at, and often trying out, software I probably don’t really need, and that was the case with this post on the lemmy.world Selfhosted community.

The basic idea of ViewTube is that it’s a self-hosted front end for YouTube, which just happens to strip out all the advertising and tracking. You can create your own local accounts which allows you to subscribe to channels and which keeps your progress so you don’t start over if you go back to a video - although I couldn’t see a history list. Forgetting your history might be a feature in an app designed to prevent tracking.

Docker volume backup is more complicated than it should be

Fri, 17 Nov 2023 00:00:00 +0000

When I set up my first Docker container (I think for Uptime Kuma ), I had read around and understood there were two choices for persistent; bind mounts (where the data inside the container is effectively a symlink to a location on the local file system) or name volumes where Docker abstracted that away a bit, so you didn’t have to worry where it was - I sort of understood Docker ‘managed’ it.

Ansible playbook to start Proxmox hosts

Sun, 05 Nov 2023 00:00:00 +0000

In my last post , I talked about tagging guests in a Proxmox node so I could easily see which VMs and LXCs I needed to manually start before I ran an Ansible script to run all my apt updates. It would have been reasonable to wonder why I didn’t just add things to my playbook to magically do that.

The answer would be, I haven’t gotten around to it yet, so here goes:

Proxmox tags to solve a problem

Thu, 02 Nov 2023 00:00:00 +0000

Each weekend I run an Ansible script that updates all my apt based VMs and containers. For the production machines, that’s everything, but my dev Proxmox is full of half-finished projects. Some of these have IP addresses reserved and are in the Ansible hosts file (because whatever service they are running is almost ready to move to the production server) others do not.

Long story short, the dev server has some containers and VM’s that need turned on before I run the updates, and some that don’t. I could just start them all up, for the ten minutes the updates usually take, but that seems wasteful somehow. If there was only some way to mark the ones I need to turn on in the Proxmox webgui! Well, there is. We can add tags to machines in Proxmox.

apt update - BADSIG 871920D1991BC93C

Mon, 30 Oct 2023 00:00:00 +0000

I have an ansible script that runs each weekend which basically does an apt update && apt upgrade -Y on every Debian based instance. This weekend it failed on one Ubuntu host. When I went it to try it manually, this was the output:

Hit:1 http://au.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 https://download.docker.com/linux/ubuntu jammy InRelease 
Hit:3 http://au.archive.ubuntu.com/ubuntu jammy-backports InRelease 
Hit:4 http://au.archive.ubuntu.com/ubuntu jammy-security InRelease 
Get:5 http://au.archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 
Err:5 http://au.archive.ubuntu.com/ubuntu jammy-updates InRelease 
 The following signatures were invalid: BADSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>
Get:6 https://pkgs.tailscale.com/stable/ubuntu jammy InRelease
Fetched 125 kB in 1s (125 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
11 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://au.archive.ubuntu.com/ubuntu jammy-updates InRelease: The following signatures were invalid: BADSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>
W: Failed to fetch http://au.archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease The following signatures were invalid: BADSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>
W: Some index files failed to download. They have been ignored, or old ones used instead.

Solved

The first google result mentions apt-cache - which I also run , so a first level debug step is to delete the /etc/apt/apt.conf.d/00aptproxy file that redirects apt requests to the cache I run in an LXC container. After that, if I re-run the apt update it works perfectly. Seems like a problem with the cache then. I’m not sure why it would only affect this host though - I have other Ubuntu VM’s in the fleet that are not getting the original error.

Tailscale keys expire

Tue, 24 Oct 2023 00:00:00 +0000

I have an Ansible playbook I run each weekend to do all the apt updates. As well as keeping everything up to date, it’s a good check-in that everything’s alive and working as expected. I have Uptime Kuma checking the services are alive, and that no one is running out of disk or memory so there shouldn’t be any drama right?

This weekend, three instances (two remote, one local) timed out with “unreachable”.

Getting Tailscale working in LXC containers

Wed, 18 Oct 2023 00:00:00 +0000

I’ve taken to running lots of my services in LXC containers under Proxmox. I like the feeling of installing in a VM, but it’s lightweight. I like the backups, I like things being isolated from each other, I like moving them around between machines easily. I’m just a big LXC lover at the moment.

I’m also a Tailscale lover, and the generous number of nodes in the free tier means I now just routinely install them in my VMs and containers without a thought.

Certbot - adding more virtual hosts

Sun, 15 Oct 2023 00:00:00 +0000

I’ve got a domain that’s not currently used, so I’m going to set it up as a virtual host under NGINX. This server is already serving two domains set up with Certbot for SSL. Is it going to be possible to add another site and have Certbot manage the certificates for it after I’ve run Certbot once?

When I googled around to find out, I didn’t find anything - which is usually a sign I’m either asking a wrong question, or it’s so little drama that no one ever mentions it. I decided just to move the site, check it was all working for the http version, then run Certbot and see what it said.

BOINC in an LXC container

Mon, 09 Oct 2023 00:00:00 +0000

Years ago, I was very keen on the SETI@home project that used a distributed computing model whereby packets of digitized received radio data were farmed out to individuals’ computers to be processed to look for any unusual signals that could potentially be from an intelligent extra-terrestrial source.

That’s long since defunct, but the idea lives on with BOINC - a system run out of Berkley that allows different science organisations to offer projects to run on individuals’ computers.

Solved DNS Issues - Proxmox, LXC, Ubuntu, Tailscale

Fri, 06 Oct 2023 00:00:00 +0000

I’ve picked up an new TP-Link WAP with Omada, so I wanted to spin up an Ubuntu 20.04 LXC to run the controller software in, and ended up spending a couple of hours figuring out why things where not working.

The initial problem was I was having connectivity issues pulling down the updates for all the packages required. I went down a bit of a tangent because I installed an apt cache the other day, so I was looking for problems there. Eventually I narrowed it down to DNS not working and started A/B testing like this:

Caching APT updates

Tue, 03 Oct 2023 00:00:00 +0000

It’s bothered me for a while that all these VM’s are pulling down a lot of the same updates. As well as needlessly using some bandwidth, I’m hammering the update servers (that I don’t pay for) with the same requests over and over. I did briefly consider running my own mirror, but that’s not simple, plus I’d then be mirroring a heap of files in a complete repository that I’d never use. What I really needed was some sort of cache so once I’ll pulled down an update, it would hang around for a few days being available to other machines on the local network. Luckily, that exact thing exists - APT Cacher NG .

Installing service with Ansible

Sat, 30 Sep 2023 00:00:00 +0000

Having written my little monitoring endpoint in Go, it needs pushed out to all my servers and VM’s. Clearly this is a job for Ansible which I’ve already dabbled my toes in . Before we get onto doing that though, we need to have a think about how to make it a service.

Linux Services

A service in Linux is just a program, but one that’s usually required to be running all the time to provide some piece of functionality. The “program” can be any executable, but to allow systemd to manage it, we need to tell it a bit about what we want in a .service file. This file is used by systemd to know how to manage the service. They can get quite complex, but here’s the simple one for vitals-glimpse - my little monitoring API endpoint.

Simple API endpoint in Go

Wed, 27 Sep 2023 00:00:00 +0000

I’d like a small, quick, low load endpoint on all my nodes and VM’s that exposes a text keyword indicating if that machine is okay for RAM and disk space. I’m currently using Uptime Kuma to monitor if these machines are pingable, but I’d love a tiny bit more information from them so I’d get a Ntfy buzz on my phone if a machine is in trouble.

I mentioned a couple of weeks ago that the benefit of doing it in C rather than Node.js was probably not worth the trouble, but then being a fickle developer, decided to write it in Go.

Problems backing up LXC to NFS in Proxmox

Sun, 24 Sep 2023 00:00:00 +0000

If you create an unprivileged LXC container on Proxmox, then try to back it up to an NFS share, for example on a NAS, you’ll get an error when it tries to build the temporary file.

The clue is in the Permission denied line. It is trying to create a temporary file on my NAS, and failing because of a permissions problem. If I try the same backup to the local storage, it works fine.

Use VS Code to work on remote files

Thu, 21 Sep 2023 00:00:00 +0000

If you’ve got a script, or some code to work on, and it’s on a VM somewhere, you can always ssh in and use nano or vim to make your edits. Like a caveman. With an archaic editor, no intellisense, and no spell checking.

Or….

This magic - of editing a files on a remote server over SSH is achieved by using a Microsoft plugin for VS Code - “Remote - SSH ”

Disable SSH root logins

Mon, 18 Sep 2023 00:00:00 +0000

This always makes me laugh:

It’s like half the traffic on the internet is bots trying random passwords on root accounts over ssh. This is on an Ubuntu VPS on BinaryLane that had only been spun up five minutes or so. Looks like about one attempt every 10 seconds.

This is why the number three thing on my new install list is to disable root access via ssh. Here’s my system - possibly just for Ubuntu and related systems:

Lightweight Web Servers

Fri, 15 Sep 2023 00:00:00 +0000

I’ve been using the excellent Uptime Kuma for my monitoring, but a couple of recent incidents - an external USB mount disappeared on a remote machine, an NVME drive filled up on a different node and stopped backups working because of a configuration error - have made me start to think about more robust monitoring.

The are many great tools for this - Nagios , Prometheus etc. but they are pretty substantial time investments for the excellent power. They can save time series data and display them beautifully. However, all I really want is to add some extra ability to Uptime Kuma.

Testing Storage Speed

Sun, 03 Sep 2023 00:00:00 +0000

Now I’ve added NVME drives to my nodes, plus added an external NMVE RAID, I’ve got quite the collection of storage options. For one of my nodes, it looks like this:

The 256GB NVME the OS is installed to
The 512GB SSD, currently running ZFS
The Synology NAS - 4 x 6TB drives in RAID 5 on a 1GB switch
A pair of 256GB NVME sticks in an external USB3 enclosure set up as a mirrored ZFS pool.

For my dev VM’s I often set them up to have their storage on the NAS - it’s just super easy to move them around then. The production VM’s currently have their storage on the SSD (that machine hasn’t had the NVME upgrade yet), but obviously with all these options, it’d be interesting to think about what goes where.

Error wiping old drive in Proxmox

Thu, 31 Aug 2023 00:00:00 +0000

When I popped in an NVME drive and freshly installed Proxmox to it, I assumed I’d just be able to wipe the SDD that had previously been the boot drive to set it up as a ZFS pool. However, when I tried to do the wipe, I was greeted with the error:

disk/partition '/dev/sda3' has a holder (500)

I assume this means there’s a flag set on one of the Proxmox partitions to prevent accidental deletion or Proxmox thought that’s where it was running from. It’s likely that it’s related to this message I had during installation that I haven’t seen before:

How to install M.2 SSD in HP G2 800 Mini

Mon, 28 Aug 2023 00:00:00 +0000

As part of my strategy to not worry about the slightly dodgy SMART reporting on the SDD’s in my HP Elitedesk G2 800 Mini Proxmox nodes, I’d decided to make use of the full sized M.2 slot to install 256GB NVME drives. That way I can boot from those, and have the SSD’s running ZFS which allows scrubbing to check the integrity of all the data. My VM disks can live on this drive.

Installing a Node app on a server

Tue, 22 Aug 2023 00:00:00 +0000

Before I write a fancy Ansible playbook to automatically set up the Nginx/Node combo on my web servers, it might be worth going through how to deploy a Node app so it can run on a server without you being logged in.

Until now, I’ve been running my tests on my laptop, or in a server logged in as myself - sometimes detaching from tmux. But we need a bit more professional set up than that. The process will look something like this:

Ansible with Secrets

Sun, 13 Aug 2023 00:00:00 +0000

We wrote a nice little Ansible playbook the other day to install nginx on our web servers and ensure it was running. We were able to store the usernames in the hosts inventory file using the ansible_ssh_user variable. Then, we ran the playbook with the command:

ansible-playbook web_installs.yaml --ask-become-pass

This asked us the password to use with the usernames in the hosts file. Luckily that day, it was the same username/password combo to use for sudo on every server. What happens if that’s not the case? Here’s our new hosts file for today. There’s a cool new sysadmin in town - Jane.

Bloody VIM

Thu, 10 Aug 2023 00:00:00 +0000

Vim is a highly configurable text editor built to make creating and changing any kind of text very efficient. It is included as “vi” with most UNIX systems and with Apple OS X.

vim.org

You will encounter vi/vim as the incomprehensible text editor that pops up by default when you need to edit something in your sysadmin journey. Perhaps you issued the command to edit your Ansible vault, perhaps you forgot to add a message to a commit. It’s going to be unavoidable.

Finding the host IP from inside a Docker container

Mon, 07 Aug 2023 00:00:00 +0000

Having successfully set up and tested my node.js api handling app behind nginx on a development VM in the homelab, I decided to move it to my VPS so I could start using it for real. I had a bit of trouble finding the nginx.conf files on the VPS, until I remembered I was running nginx in a docker container on this machine!

I got everything set up, I could hit the domain in a web browser and get served the static page, and I could <domain_name>:3000/api/gnp_temp.txt and get the file delivered by the node script, but if I tried <domain_name>/api/gnp_temp.txt - “Bad Gateway”.

nginx in Front of a node.js app

Fri, 04 Aug 2023 00:00:00 +0000

NGINX is a great webserver and reverse proxy - as in it can hand off requests to other web-servers. That’s the situation I want to have set up on my VPS. I want NGINX to handle incoming requests - some of them will just be sorted out by returning static HTML, others (like the weather api I’ve been playing with) need to be handed off to other services to respond to.

Where to go after Reddit

Tue, 01 Aug 2023 00:00:00 +0000

A big chunk of my mindless doomscrolling used to go to Reddit, but also, Reddit posts from the various communities were frequently the useful results when googling error messages. I lurked in many a sub-reddit, but only posted in a couple - usually r/self-hosted or r/Homelab.

The problematic treatment of the communities in the leadup to their IPO has been well publicised, and the short blackout by some subreddits seemed to have zero effect on the company’s approach to it’s users (which is in fact what they have to sell). Those subreddits, and many others are still working, but (and perhaps I’m imagining this) seem somehow thinner. Additionally, I feel like it’s a fragile arrangement - the company has shown how they will deal with their communities, so depending on them in the long term does not seem wise, or even, somehow, ethical - like I’m crossing a picket line.

ZFS Basics on Proxmox

Sat, 29 Jul 2023 00:00:00 +0000

I’m a keen listener of the 2.5 Admins podcast in which there’s frequent enumeration of the advantages of ZFS as a file system. So much so, that I’ve had occasional twinges or regret about the money I spent on the Synology - although it has been boringly reliable and does everything I need.

Proxmox has some built in support for ZFS, including through the web GUI. So I’ve been itching to give it a try.

First Ansible Playbook

Wed, 26 Jul 2023 00:00:00 +0000

In the previous post , we looked at getting up and running with Ansible, including using the ad-hoc mode to send commands to our servers. We had a inventory file called hosts that had groups of server IP addresses and a simple ansible.cfg file that pointed to our inventory file.

Playbooks

Ansible playbooks are used to collect together a description of the state we want in a server. When the playbook is executed, Ansible figures out what things need need changed, and changes them. If you’re used to the procedural nature of a bash script, where things proceed from one step to the next, and there might be decision branches, this requires an adjustment in your thinking. This is similar to the adjustment I had getting my head around SwiftUI , and moving from JS to React .

Proxmox 8.0 Install

Sun, 23 Jul 2023 00:00:00 +0000

I’m normally a x.1 release type of sysadmin, but the increasing temptation of installing Proxmox 8.0 while I’ve got some time off, and the fact that I’ve got a cluster, so I can just move the VM’s around all adds up to thinking I’ll do that today.

Here’s how my system works. It consists of three HP-800 mini G2’s. pve-prod1 is a bit fancier - i7 6700T and 32GB, the other two are i5 6500T and 16GB. The production VM’s use the local SSD but backups go to the NAS. All the machines are currently running Proxmox 7.4. They are not clustered in the proper sense - I don’t need high availability, and I don’t want to run them all the time. pve-prod1 runs 24/7 and I just power up pve-dev1 when I’m working on something.

Getting Started with Ansible

Wed, 19 Jul 2023 00:00:00 +0000

Ansible is a system for executing commands on remote systems. It allows a declarative approach - so if you run a playbook (the system configuration files are called playbooks) that says a system has a Docker container running Jellyfin, Ansible will check if that’s true, and if not, make it so. Ansible is best used when you have a large number of systems to maintain, but even with a small number, it serves to document systems as well as to automate their creation.

How to recover a docker run command

Sun, 16 Jul 2023 00:00:00 +0000

Imagine if, lets say hypothetically, you’d set up an application months ago with a docker run command. Then you’d heard there had been an update to the app because of a security update. So you need to stop/remove the container, pull a new image and restart it, trouble is, you don’t remember the exact run command you used to start it.

This didn’t happen to me, since all my vm setups are in git as markdown (I’m pre-Ansible), but I did google how to do this thinking that there would be an easy way before I bothered to look through my config files.

How to deploy a Node.js app

Wed, 05 Jul 2023 00:00:00 +0000

This is one of those things that is simple once you know it. I had my tiny Node service working on my MacBook, but how do I run it on the server?

Native or Container

Obviously I need Node.js installed on the server, should I have it in a Docker container, or native on the machine. There’s no clear answer here - in a container set up with Docker Compose might be more in line with my ideology of treating machines as disposable, but a native install is simpler, and I probably want to make life simpler at this stage when I’m learning everything.

Complicating the Temperature API

Wed, 28 Jun 2023 00:00:00 +0000

I’ve been slammed with other work, so my web dev learning has fallen well behind. Luckily, the YouTube procrastination algorithm noticed this and suggested I watch a video from CodeWithCon titled Learn Backend in 10 MINUTES .

Since I was watching a video of a guy learning to land a C152 at St Baths (a skill I do not need) at the time, it was hard to argue with myself that I didn’t have ten minutes to learn all of backend programming.

Outside Temperature From an API in a Shell Script

Wed, 03 May 2023 00:00:00 +0000

I’m interested in collecting some internal temperature data from my servers to look at the effect of adding an NMVe drive. Last week we had a couple of warm days immediately followed by a couple of cool ones. I imagine a 20° ambient temperature change could effect the server temperatures, so I thought it would be good to add that to my temperature logs.

I don’t have a weather station or other automated system for collecting the temperature, but there are several commercial sources for this data which, while probably not as good as a sensor in the server room, will be fine for our purposes.

Running a Browser Remotely - n.eko

Tue, 02 May 2023 00:00:00 +0000

When I installed the backup NAS and a media server at the remote site, one of the jobs on my list was to reserve the IP addresses for the NAS, node, and the VM in the local router. I carefully did that, but when I got home (200 km later) and opened my laptop, the browser page was open on the DHCP settings with a table of mac addresses I’d added, and the reserved IP’s, and at the bottom of the page, a large blue “Apply Changes” button. Had I pressed that button to save my changes correctly? I’m not sure.

ISO wrangling - Etcher and Ventoy

Mon, 01 May 2023 00:00:00 +0000

If you fiddle around with computers, and especially with Linux drives, you’ll often find yourself with an ISO file you need to boot a device from. These can’t just be copied onto an existing USB or SD card - they need to be bootable, so you’ll need a special program to write the ISO to the storage device.

Previously I’ve been a big fan of Balena Etcher . It couldn’t be much more simple - you chose the ISO file you’ve downloaded from somewhere, chose your removable drive (it intelligently hides the non-removable drives to prevent you from accidentally wiping your hard disk), then tell it to do it’s thing.

Linux Shell Script for Temperature Logging

Thu, 27 Apr 2023 00:00:00 +0000

A potential solution to my concern about the either perfect, or nearly dead, SSD would be to add a NVMe disk to the M.2 slot in the HP Elitedesk 800 G2’s. I’d use those to boot from and run Proxmox, then the existing SSD’s on each node in the cluster would just be part of the CephFS pool that has some redundancy built into it and hosts the VMs that are not using the NAS for their storage.

SDD Wearout numbers

Tue, 25 Apr 2023 00:00:00 +0000

I didn’t understand why the default Proxmox install sets up the storage the way it does - with the available disk split up into an LVM and an LVM thin storage - so I’ve been reading this excellent Proxmox Storage Guide by Programster (spoiler - the LVM thin makes VM snapshots easier).

At one point in the post they mention that you can see the “Wearout” percentage for any SSD drives in the Proxmox GUI, so of course, since I now own five second hand HP Elitedesk 800 G1/G2’s all with SSD drives, I dived in to have a look at each drive and found this.

Why use './' in front of filenames?

Sun, 23 Apr 2023 00:00:00 +0000

In Linux (and MS-DOS I guess) the period signifies the current directory, so if I have a file in the current directory called test.txt, I can refer to it as test.txt or ./test.txt

ian@enrico-rider:~$ cat test.txt
test
ian@enrico-rider:~$ cat ./test.txt
test

I mostly see this in references to files in HTML and have often wondered why. Here it is being used in a Udemy course I’m following.

It’s one of those things that’s difficult to Google, so these days my reflex is to ask ChatGPT such questions.

Mounting NFS shares into LXC containers

Fri, 21 Apr 2023 00:00:00 +0000

I’m playing with Syncthing with the idea that it might be a good replacement for Dropbox. There wasn’t a Docker container listed in the install options, so I thought this might be a good app to run in an LXC.

I’m going to use a share from the NAS, and I’m assuming I’ll need it mount it into the container for Syncthing to access. I’m experienced enough to know that I’m going to want a privileged container, and I thought I’d done all the NFS sharing correctly, but when I tried to mount the NFS share, I was getting an error.

Running Multiple Linux Commands in One Line

Wed, 19 Apr 2023 00:00:00 +0000

Since I’m constantly standing up Linux virtual machines and containers - almost always of the apt variety, I’m constantly typing in:

apt update
apt upgrade

Then hitting enter again to allow whatever installation is needed to proceed. I’ve noticed in some of the commands I’ve been pasting in from installation instructions or StackExchange solutions have been separated by characters that look like it allows several commands to be run one after the other. To cut a long story short, the commands above could be entered like this with double ampersands:

Linux on HP Mini 110

Mon, 17 Apr 2023 00:00:00 +0000

I’ve been furthering my Linux education by playing with some desktop distros in VMs, but it’s not a great experience accessing them through the Proxmox web GUI. The alternative to this is to use a good SPICE client on the remote desktop, but there is not a simple good solution for this for MacOS.

I’ve been playing with the idea of picking up an old i3/i5 Thinkpad - these are around the AUD130 mark on eBay, to run a Linux distro with the main idea being to use it to SPICE into my VMs.

Recursively Deleting Files in Linux

Fri, 14 Apr 2023 00:00:00 +0000

I’ve been using this rsync command to backup files from my NAS to a USB drive. The –excludes are to avoid copying over some junk hidden files - some created by MacOS and some by Synology.

sudo rsync -rvit --exclude '*@eaDir*' --exclude '.DS_Store' /volume1/media/ /volumeUSB1/usbshare1-2/media --del

The .DS_Store files seem to be dropped by MacOS every time I view a directory on the NAS from my MacBook. They’re not doing any harm, and they presumably do something handy for the Mac - remembering the view settings for that folder or some such. Nevertheless, they annoy me. It makes sense to not back them up - they don’t serve any useful purpose in that context.

Proxmox LXC backup to NFS share failing

Wed, 12 Apr 2023 00:00:00 +0000

I was doing updates on all my nodes and VM’s today, and backing up the VMs that aren’t already on a backup schedule. On my dev machine I have a Debian LXC container that I mostly just use for trying out Linux commands and playing around. I used to have a backup of it that I used a lot - after playing around I like to set it back to a fresh install plus my ssh keys - but I lost it somehow when moving the VM to new metal.

Using NAS for Proxmox backups

Mon, 10 Apr 2023 00:00:00 +0000

A few weeks ago , I was very excited to be able to take a snapshot of a virtual machine, copy it across the network from that Proxmox node, copy it back across the network to a different Proxmox node, start it there, and have it up and running, without it noticing it was actually on different hardware.

Backing up a VM is pretty simple, you just click on the node, choose Backup and click the Backup Now button. The ease, and completeness of backing up a VM is one of the main reasons I’m using Proxmox for my systems.

Where Do Docker Container Logs Go?

Sat, 08 Apr 2023 00:00:00 +0000

I’m still loving the Docker “just works” magic, despite their terrible PR skills , but sometimes I start a container, then the docker ps -a shows it exited almost immediately. Clearly I’ve made a mistake, but there’s no stdout error message to tell me what I’ve done wrong, where is it.

Let’s look at an example from today. I’m testing Filebrowser on a dev machine before I deploy it to the remote backup machine I’m assembling. And instead of following the official instructions , I’m following a blog post which has a few more details, but unfortunately also a small error.

Allowing Proxmox to use a Dynamic IP

Thu, 06 Apr 2023 00:00:00 +0000

I’ve discussed before , that when you first install Proxmox, it grabs an IP address from your DHCP server (this usually runs in your ISP modem if you haven’t created a better setup), but then it stores it as a static ip. This is a sort of compromise that makes sense and works for most circumstances.

As soon as I’ve provisioned a new Proxmox server, I then usually tell the DHCP server, to always serve that address to the MAC address of the new Proxmox server. Since Proxmox does not use the DHCP server on subsequent boots, all that really does is prevent the DHCP server give the same IP address out to another device - which had happened to me prompting the earlier post. The DHCP server had given the address to a wifi lightbulb while the server was off, then when the Proxmox server booted up, the netwrok access was all messed up.

RAID Rescue

Tue, 04 Apr 2023 00:00:00 +0000

I’m in the process of shuffling disks around as I move towards my 3-2-1 storage arrangements. I thought after my extensive rsync adventures I’d mirrored everything everywhere, but then realised, with a sinking (no pun) feeling, after I’d repurposed a drive out of the 2 drive Synology as a USB caddy drive and wiped it, that I’d forgotten my audio book directory. All my rsync fiddling around had been on the video subdirectory of the media folder, not the whole media directory that included my audiobooks.

HP EliteDesk 800 G2 Memory Upgrade

Sun, 02 Apr 2023 00:00:00 +0000

The hardware engineering of these corporate world mini-PCs is really nice. I swapped out the RAM today to bump my main machine up to 32GB from 16GB. It was a straightforward task - no screwdrivers, no drama.

To open the machine up, there is a single large screw on the back that can be undone with your fingers - it’s a captive screw, as in it doesn’t fall out - just another nice engineering thought.

Proxmox Backup Files

Fri, 31 Mar 2023 00:00:00 +0000

I’ve got some extra RAM to drop into the HP 800 G2 mini that I use as my production server. I feel like that’s a low risk change, but since it’s easy to take VM snapshots I shutdown the VM’s and did that, and wanted to just copy them off the local storage.

I’m moving towards having these backups (and the ISOs) on the NAS rather than locally, but have not implemented that. So to get my backups I need to SSH in and find them.

rsync episode IV - a sudo hope

Thu, 30 Mar 2023 00:00:00 +0000

With all those earlier rsync bumps out of the way, I was ready to try my first rsync backup at the command line to sync my movies directory on the NAS to a (NTFS formatted) USB drive plugged into the same NAS. This is to be one of the simplest since there’s no remote server involved, just copying from mount point directory to another - so no drama with remote permissions.

rsync / Synology / @eaDir

Tue, 28 Mar 2023 00:00:00 +0000

The reason I’ve been figuring out rsync is to setup my backup strategy. Eventually this will partly be managed with scheduled tasks (ie cron jobs) running rsync. I wanted the SSH in and try this out, since I didn’t know some basic things like the mount points of the shares.

Mount points

My first issue was to find the paths to all my data. This turned out not to be a drama. Each of the volumes you create when the NAS is set up are just in the root directory. This includes any USB drives plugged in.

SSH with Keys to Synology

Mon, 27 Mar 2023 00:00:00 +0000

The Synology operating system DSM (I’m on DSM 7.1.1) is Linux, but its highly customised for the purpose of making running a complicated Linux NAS doable for less technical users.

Due to that, some things that are routine in a regular distro, require a few more steps to jump through to get them to work. SSH-ing in to a Synology with keys is one of those things.

Should you?

Before you do start fiddling around, it’s probably worth mentioning that almost all the things you might want to do on the Synology can be accomplished through their web interface, or by installing a ‘package’ from the Package Center. For example, if you need to run a cron job, that’s done through the Control Panel ‘Task Scheduler’. If you need TailScale installed to easily access it over Wireguard, there’s a TailScale package. In general it’s probably easier and safer to do things their way.

rsync basics

Sun, 26 Mar 2023 00:00:00 +0000

I’ve started down the path of improved storage management, including embracing the 3-2-1 mantra. I’ve settled on a RAID6 NAS for local, mirrored to an off-site NAS, and an offline local USB drive.

While I’ve been setting those up, my services have been live, so files have been changing on my main storage, which I’ve then switched to the bigger NAS, and I’ve been trying to keep data in sync by remembering what changes have been made where, and manually replicating them. That’s not sustainable and not the plan.

CPU Comparisons

Fri, 24 Mar 2023 00:00:00 +0000

When I was a young whipper-snapper, working at the “data processing” centre, you could see if one CPU was better than another one by the CPU name/number. No one wanted an 8086 once the 286’s came out. Then a 386 was what you wanted for the latest multitasking support, but only till the 486 was available, then you wanted that for the gargantuan memory addressing.

With that idea firmly in mind, I’ wanted an i5 to be better than an i3, and an i7 better than all of them, but it’s apparently not that simple . I do come across people in forums talking about ‘generations’ of Intel processors - so all this is probably decodable, but I’m not exactly sure how.

HP Secure Boot Pain

Thu, 23 Mar 2023 00:00:00 +0000

Since the HP EliteDesk 800 G1 I’m using as a dev/homelab machine is going to be re-purposed as a media/backup server elsewhere, I’ve grabbed another G2 to use as a second box. The homelab machine serves as a backup device for the production server that runs my self-hosted services, but also is the machine I play with - testing my software, but also trying out any new self-hosted software I’m having a look out or configurarions I’m thinking about for the ‘production’ server.

Mounting one Synology NAS to another one

Tue, 21 Mar 2023 00:00:00 +0000

I went over mounting a Synology NAS share on a Mac or Linux host a while ago . Now I’ve populated a new NAS, and I want to copy my data over to it. I could mount them both to my laptop, and the data flow would look like this:

NAS1 - switch - wifi - laptop - wifi - switch - NAS2

Since I’m copying 4TB, it will take a few hours, and if I forget what’s going on and close the laptop, or take it outside of my wifi the transfer will die, and I won’t be sure which files are patent. What might be better would be something like this:

Proxmox VM Memory Upgrade

Sun, 19 Mar 2023 00:00:00 +0000

I ordered some RAM this week for my production server - it’s quickly becoming clear that memory is the limiting factor when running lots of services and VM’s that don’t get much use - rather than processing power. I’m not really a hardware guy, so figuring out exactly what RAM I need is a slightly fraught process - I won’t be fully confident I’ve ordered the right thing until I install it, boot up, and see my G2 800 come to life maxed out at 32GB.

No DNS on Proxmox machine

Fri, 17 Mar 2023 00:00:00 +0000

I had some more network weirdness setting up this new Proxmox machine. When I went to run the updates it couldn’t resolve any of the addresses:

root@pve-kr01:~# apt update
Err:1 http://ftp.au.debian.org/debian bullseye InRelease
 Temporary failure resolving 'ftp.au.debian.org'
Err:2 http://download.proxmox.com/debian/pve bullseye InRelease
 Temporary failure resolving 'download.proxmox.com'
Err:3 http://security.debian.org bullseye-security InRelease
 Temporary failure resolving 'security.debian.org'
Err:4 https://enterprise.proxmox.com/debian/pve bullseye InRelease
 Temporary failure resolving 'enterprise.proxmox.com'
Err:5 http://ftp.au.debian.org/debian bullseye-updates InRelease
 Temporary failure resolving 'ftp.au.debian.org'
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://ftp.au.debian.org/debian/dists/bullseye/InRelease Temporary failure resolving 'ftp.au.debian.org'
W: Failed to fetch http://ftp.au.debian.org/debian/dists/bullseye-updates/InRelease Temporary failure resolving 'ftp.au.debian.org'
W: Failed to fetch http://download.proxmox.com/debian/pve/dists/bullseye/InRelease Temporary failure resolving 'download.proxmox.com'
W: Failed to fetch http://security.debian.org/dists/bullseye-security/InRelease Temporary failure resolving 'security.debian.org'
W: Failed to fetch https://enterprise.proxmox.com/debian/pve/dists/bullseye/InRelease Temporary failure resolving 'enterprise.proxmox.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.

So some sort of DNS problem. The entry for the DNS is in /etc/resolv.conf when I looked in there, it said:

Proxmox Dynamic IP

Thu, 16 Mar 2023 00:00:00 +0000

I ran into a little hiccup today. I’m building out a Jellyfin media server in a little HP G2 Mini PC. The config was going to be a Debian server inside Proxmox (because I love VM snapshots for backups) running Jellyfin in a container. There’ll be an external USB3 hard drive for the media storage.

I was intending to build it all out and test it, then ship it to it’s final home.

NAS Storage Calculations

Sat, 11 Mar 2023 00:00:00 +0000

I’ve been really happy with my two bay Synology NAS - a DS216j. The Synology’s seem to have great reputation for just pushing on. Mine is loaded up with two 8TB Seagate Barracudas in RAID 1 leaving me with a one drive failure redundancy.

I guess a more hard-core host-er than me would be building their own array and using Unraid or ZFS or something. I’m pretty comfortable with the Synology off the shelf system; it’s a good match for my (low) level of expertise, and more robust than my previous storage system of a USB external drive.

Recursive list of files in Linux

Wed, 08 Mar 2023 00:00:00 +0000

I’ve spent a few hours over the weekend migrating a media library from an external USB drive to the NAS, and in the process reorganised it, and in many cases bulk changed file names. I’ve also added a heap of metadata.

I’d like to check that I haven’t missed any files, but a side by side listing of each data source won’t do the trick, so I’ll probably end up pulling the data into a spreadsheet, but I’d like to get as close as possible with Linux-fu first.

Sudoers' file not working

Mon, 27 Feb 2023 00:00:00 +0000

A couple of weeks ago, I posted about the sudoers’ file , and how there was a special tool for editing it since breaking it is a bad idea, and that in fact I needn’t bother, since I can just add my user to the sudoers’ group with:

usermod -a -G sudo ian

That worked (on Unbuntu) since /etc/sudoers contained a line saying:

# Allow members of group sudo to execute any command
%sudo	ALL=(ALL:ALL) ALL

I tried the same trick on a fresh Debian install today, and no dice:

Folder ownership problems with Jellyfin

Wed, 22 Feb 2023 00:00:00 +0000

After being so blase about the file permissions when mounting the share to the Linux file system, and testing that root could read and write to the share, I ran into problems immediately when trying to add the media folder as a library in Jellyfin - getting the error “The path could not be found. Please ensure the path is valid and try again.”

I definitely had the path correct - I could copy it from the dialog and cd to it at the CLI. So I suspected it was a permissions thing. The app might not have read permissions for the directory.

Accessing a Synology NAS from Linux

Mon, 20 Feb 2023 00:00:00 +0000

I picked up a Synology DS216j NAS from eBay to use for storage for the rapidly growing home lab. The eventual plan is that as well as my VM backups, it will host the media library, and eventually (when this has all proved itself reasonably bullet-proof) my current DropBox contents. That won’t all fit on the 2x2TB drives that the DS216j came with, and I have a pair of 8TBs on hand, but I wanted to set it up and checked it all worked.

Configuring Proxmox for Free Use

Thu, 16 Feb 2023 00:00:00 +0000

I installed Proxmox on my second server last night, and tonight when I ran apt update I ran into the error you get when you haven’t bought a license.

Err:5 https://enterprise.proxmox.com/debian/pve bullseye InRelease 
 401 Unauthorized [IP: 103.67.14.50 443]
Reading package lists... Done 
E: Failed to fetch https://enterprise.proxmox.com/debian/pve/dists/bullseye/InRelease 401 Unauthorized [IP: 103.67.14.50 443]
E: The repository 'https://enterprise.proxmox.com/debian/pve bullseye InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Even though I guess it was only a month ago (let that sink in people who think the raspberry Pi they just bought is going to be the last homelab hardware they buy 😊) since I set up my first Proxmox server, I’d already forgotten there’s a step to enable it to get updates without a subscription.

Moving a VM between two Proxmox hosts

Thu, 16 Feb 2023 00:00:00 +0000

So, the very small datacentre has undergone a major hardware upgrade today. The HP 800 G1 is joined by an HP 800 G2. Four core i7 vs the old two core i5. Double the RAM to 16GB, four times the disk. The old machine will become a dev/play machine - still virtualised, and the new machine will run the production apps, mostly in Docker containers.

Since everything is containerised, I did consider running Unbuntu Server on the bare metal of the new machine, but running it on Proxmox will give me some flexibility, and since we’ve stepped up the underlying hardware resource so substantially, performance will be well in front anyway. Plus it will give me some flexibility if needed in the future.

Uptime Kuma & NFTY

Wed, 15 Feb 2023 00:00:00 +0000

Uptime Kuma is a monitoring tool suitable for self-hosting, and as well as being a good tool for monitoring the status of your network and applications, it’s a nice smallish app to get started on Docker containers.

Since it’s in a container, you need to create a volume for it and pass it in to persist your settings. Then it’s just a matter of adding each item you want to monitor. There’s a heap of fancy options for this, the only three I’ve used are ping - just pings an address, http(s) - requests a page and checks the header for a 200, and http(s) keyword - looks at the returned page for a keyword in the html.

Netgear GS108E switch problem

Tue, 14 Feb 2023 00:00:00 +0000

I had a weird issue today that I wouldn’t have known about if I didn’t have an over-engineered home network monitoring system.

I’ve got a new GS108E managed switch , purchased in anticipation of connecting a NAS to the homelab - I want to have a solid 1Gb connection between the NAS and the servers, and also in anticipation of moving to VLANs before I start to expose self-hosted services to the internet.

Local host names with Pi-hole

Mon, 13 Feb 2023 00:00:00 +0000

I run an instance of Pi-hole as a network-wide advert and surveillance blocker. It also has a setting to block individual domain which I use to force myself to really consider if 30 minutes of Reddit is a good idea when I should probably just be going to bed.

As I’ve increased the number of real and virtual devices on my network, it’s getting to be a pain remembering all of their IP addresses. So I’d like to have DNS entries for them, for example I’d much rather:

ssh key login on VPS

Sun, 12 Feb 2023 00:00:00 +0000

Due to potential brute force attacks , it’s a good idea to turn off password access via shh and instead rely on ssh keys. In this post, I’ll run through that process.

Generating your key

On a mac (or actually most *ix systems), your ssh keys live in the .ssh directory inside the users home directory. Since it starts with a period, it’s a ‘hidden’ directory. To see it in Finder press

Save Proxmox password in Chrome

Sat, 11 Feb 2023 00:00:00 +0000

When I installed Proxmox, I’d used a secure, and therefore absurdly long and complicated root password. I do use a password manager, but don’t have it integrated into Chrome, so it was buggging me having to find it and paste it in each time - why wasn’t Chrome offering to save it for me?

Well, you’d guess it was something to do with this. I feel like Chrome is trying to tell me something here:

Saved by the qemu_guest_agent

Fri, 10 Feb 2023 00:00:00 +0000

Literally an hour after I wrote the post about installing the qemu guest agent in a VM and explaining how it can be used to inject root level commands into a VM, I had use of it due to a mistake.

I’d decided to add myself to the sudoers file. Since the last line in that file is a directive to include all the files in the /etc/sudoers.d directory, the accepted way to do that for local changes is to create a file in that directory with the necessary commands.

Proxmox - Qemu-guest-agent

Thu, 09 Feb 2023 00:00:00 +0000

One of the strengths of having virtual machines (VMs) running inside a hypervisor like Proxmox is how they are isolated from each other and their host. This is a strength - if there is a problem with a particular VM nothing else should be affected by it.

But this can also be a pain if the hypervisor needs access to a VM to control or monitor it in some way that’s only possible from inside the VM. Proxmox can use the Qemu Guest Agent for this purpose. To over simplify, this is a deamon that runs in the VM and opens a unix socket/virtual serial port to the hypervisor, and listens for commands on it. With Proxmox, the main use of this is to aid in orderly shutdowns and backups, but it also allows us to run commands in the VM from Proxmox - an obvious security compromise. You definitely would not want to install this daemon on a hosted VPS.

SSH & the scary warning

Wed, 08 Feb 2023 00:00:00 +0000

The first time you connect to a new server with ssh, it asks you something like:

➜ ~ > ssh ian@192.168.100.20 
The authenticity of host '192.168.100.20 (192.168.100.20)' can't be established.
ED25519 key fingerprint is SHA256:ZcNTcOjO/0fOLC5iNChf8Q8MHN7z2d+VV0qz7XqH1g4.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.100.20' (ED25519) to the list of known hosts.

Once you’ve said yes, it adds the server ‘fingerprint’ to the known hosts file, then next time you ssh there, it feels safe - we know this server.

Proxmox - Installing a Virtual Machine

Tue, 07 Feb 2023 00:00:00 +0000

Installing your first virtual machine (VM) in the Proxmox hypervisor is pretty straightforward. This post runs through those steps using Proxmox 7.3.

You need an operating system for your virtual machine, I’m going to use Ubuntu server in this example, but it could just as easily be Windows server , or regular windows, or one of the desktop Linux distributions. Whichever you decide, you’ll need to find and download the ISO for it. The ISO is a (usually quite large) file needed to install the operating system.

Chinese Hackers Want to steal my Hello World container

Mon, 06 Feb 2023 00:00:00 +0000

A smart thing to do after setting up a server on the internet, is to set up SSH keys and then turn passwords off for SSH. The reason for this is that scanning for open port 22 on IP addresses, then brute forcing password files on them is pretty much hacker 101. So if you have passwords turned on, and especially if you have a weak password you are really inviting someone to take over your server as root and add it to their botnet army for liking Putin’s twitter posts or whatever.

Your own Aussie server on BinaryLane

Sun, 05 Feb 2023 00:00:00 +0000

Listening to podcasts, I’ve been jealous of US developers who seem to have masses of $5/month VPS (Virtual Private Server) options. When I looked for similar Australian offerings a few months ago, they all seem to start at around $35 which is outside of my ‘have a play with something’ budget range.

I could of course use one of the international options, but one of the main apps on my app ideas list needs to be hosted in Australia and work under Australian data privacy rules. That might be the case for Digital Ocean (or other US companies) if you select an AU server, but I’m not a lawyer. For the imaginary clients of my imaginary app, me being able to say that the hosting is with an Australian company in Australia would be a plus.

sudo Incident Reports - where do they go?

Sat, 04 Feb 2023 00:00:00 +0000

Even though it’s my server, I still have a pang of guilt when this happens.

I always imagine Richard Stallman (or someone with a similar 2000’s database administrator beard) looking at me disappointedly and shaking his head slowly.

It does raise the question though - since it’s my server, shouldn’t I be getting a text message from CERN or something?

Where is this report?

(Relevant xkcd )

Like everything, the answer is ‘it’s logged’. We can use the journalctl command to look at the logs, on this server that’s been running less than 20 hours, there’s already several thousand lines to look through if you just enter journalctl, so I’m going to just send all the high priority logs to a file:

Proxmox - Storage Basics

Fri, 03 Feb 2023 00:00:00 +0000

Once you’ve got Proxmox installed, you can point your web browser at the IP for the physical server, and use the port 8006. Log in as root using the password you entered during the install. If you just accepted all the defaults during the install it will look something like this:

Let’s discuss what you’re seeing in that ‘Server View’ on the left there. pve is the name of my node - this installation of Proxmox on my physical server. If you named your server something different during the install, it will be show that name here.

Upgrade Cycle

Thu, 02 Feb 2023 00:00:00 +0000

Now that I’ve seen I can easily stand up VM’s on this baby server, it’s apparent the first limitation I’ll run into is RAM. It has two laptop sized memory slots that can take up to 8GB apiece. So it could easily be doubled, but at a cost of around $70.

While I’m looking on eBay for RAM, the algorithm thinks I might be interested in this.

While I’m looking at the specs (4 cores - the current one has 2, double the RAM, bigger disk), eBay is like “Hey, how about this 20% off discount code - is thAt soMetHing ThAt miGHt HeLp yoU deCiDe?”

Proxmox Hypervisor

Wed, 01 Feb 2023 00:00:00 +0000

I mentioned a while ago that the price of the Raspberry Pi4 was getting such that it’s smarter to purchase one of the little business workstations instead. Depsite having little need for such a thing, I went ahead and bought an HP Elitedesk 800 G1 “mini” PC. It has 8GB RAM (which is the max for the Pi4) as well as a 128GB SDD, the processor is an Intel i5.

This compares pretty well with the 8GB Pi4 which only has a fraction of the storage (on an SD card) at around $400. One area where the Pi would have an edge might be in power consumption - I expect it would be a bit less. One possible catch for young players is that the HP has a ‘display port’ rather than HDMI for the screen connection, so pick up a $5 adapter if you’re getting one. The metal case and nice finishing on the HP actually looks really great in my office compared with my Pi 3b+ dev server that’s sort of hanging on the end of a cat5 cable.

Pi Server

Sun, 04 Dec 2022 00:00:00 +0000

I have a a couple of Raspberry Pi’s on my home network. One is a radio interface on the AllStar network , and the other is just a toy server - I can’t actually recall why I bought it. Both of them are Model 3B’s - I’d love a 4, but they are scarce and expensive.

This doesn’t have much to do with Swift, although it’s possible to run Swift on a Pi , or even Vapor . Mine is set up as a generic web server that I use as the back end for my tiny projects. It runs Node.js , apache and lighttpd webservers, PHP , MySQL , SQLite , and, when I get to that stage of my programmming, Postgres . I could do all that on my MacBook, but it’s somehow more fun on the Pi.