I’ve been meaning to write this for a couple of weeks, so let’s get to it - things are moving to fast to reflect too long; which is it’s own risk.
In March, I wrote about how I was using AI in coding , which was Codeium (now Windsurf) in VS Code for completions, and ChatGPT and Claude online for architecture questions and code gen that was more than half a function.
When you’ve made a change to your web-app, do you run it then click around the new bits to check it works? Good start, but instead of doing that yourself, do it in a faster, more comprehensive and automated way with an end-to-end (E2E) testing setup using Cypress . Here’s how.
End to End testing is testing your app as a user might - by clicking links, entering data, looking at the screen and checking everything is okay, but it’s scripted like a unit test and the results are checked with assertions. Like unit testing this allows you to build up a collection of comprehensive tests that easily detect for unexpected behaviours - not just in the results of functions in your app, but in the user experience of the app.
I’ve been whipping up a little mock-database unit that has a few access functions but actually stores the data as arrays for a demo project for a post I’m writing. In the process I wrote this gem:
export function dbOrdersAdd(order) {
const orderCopy = { ...order };
// since id is a stringified number, finding the max is a bit of a mess
const maxId = orders.reduce((max, o) => Math.max(max, parseInt(o.id)), 0);
orderCopy.id = String(maxId + 1);
orders.push(orderCopy);
return { ...orderCopy };
}
In the comment I’m claiming the code is a bit of a mess (and from a readability point that’s true) but actually I love the elegance of using the reduce() method here.
For the longest time, I’ve been using Mocha (test runner) and Chai (assertion library) for my JS testing. They are reliable old friends.
One of the effects of the existence of Bun and Deno has been to spur Node onto adding some new features, so after appearing as an experimental feature in 18, the Node test runner dropped in Node 20.
I’m not sure if the familiar unit test layout of Mocha and Node is inherited from Jest, or comes from older testing frameworks of which JUnit and NUnit were the first ones I’d ever used. Before that I just used to write tests as lumps of assertions in regular code - which worked but wasn’t as pleasant to use as a proper unit test setup. Regardless, the system of bundling a few tests together and having them all run and spit out green ticks is not a new one.
There’s still plenty of controversy about LLMs for coding, and not without reason. But I thought I’d run through what I’ve tried, and where I’ve landed for using AI. Also what the pitfalls are, where it’s useful and how it’s changed my practice.
The training data for large language models generally is problematic. There’s no doubt that they have been trained on copyright material. With code it’s slightly less murky since there is a high availability of good quality open source data with attached licenses to train models on. No doubt this include code written by people who don’t approve of it being used by AI, but I think the popular reading of most open source licenses is that using it for training is fine.
I had an idea for a little holiday project that required a list of episodes from The Rest Is History podcast. On their ‘Episodes’ page, they have a player, and a list of post entries for the most recent eighteen podcasts. There is a ‘show all’ button, but it doesn’t work.
The player does contain the full list of episodes (about 600) including a number of duplicates, so I expected if I inspected the network calls that I’d see a JSON package arriving with what I wanted. This is what I almost always find these days so I’ve had very little call to do any real web scraping - it’s normally just a matter of locating the endpoint and perhaps extracting an API key from a header.
I’ve been containerising my websites, with their servers to make deployment simple and robust, and to move to a CI/CD workflow. Since an install of a production web server is large, I would be running about ten of these containers, and there’s already a good server facing the net and doing the reverse-proxying (NGINX Proxy Manager), I chose to bundle the Busy-Box httpd server with my sites inside the Docker containers.
I love the convenience of a hosted blog on wordpress.com, but one of the justifications for my ‘investment’ in homelab hardware and learning time was that I’d reduce my spend on hosted platforms by self-hosting them. I’ve already quit Evernote and dropped back to the free plan on Dropbox by building systems to replace them for less money and more data sovereignty. And now, the recent Wordpress drama has made me uneasy about Matt having control of domains I’ve got registered with wordpress.
I’ve had my external UptimeKuma chugging away on fly.io , for free, for months now, and the container image it was based on was a bit out of date, so I wanted to update it. I hadn’t looked at fly.io for months, and couldn’t really recall what I’d done to create it.
The way this works is that that you create a fly.toml file that sets out the details of your app. From memory I think I used the one from the docs and gave it a unique name, the name of the Docker image, the port, the datacentre location, and the directory for the persisted data. The you run fly deploy from the directory with the toml file (having already installed the CLI tool and logged in) and you’re in business.
I’ve been containerising my static websites with BusyBox (because it’s small), and in an earlier post showed how to even get the container to update parts of the site by reaching out with wget to download resources from elsewhere and saving them inside the container where we are serving the ‘static’ site from. I’d done this by including a bash script in the container with the wget in a loop with a sleep. Then started the script and the httpd server in the CMD line of the dockerfile.
The previous post went over how to bundle a static website into a Docker container. That’s a neat little trick - keeping the entire website and making it trivial to install on a VPS behind Nginx Proxy Manager. It worked great for most of my little websites.
A couple of my websites had very minor ‘dynamic’ content. One was pulling down the local temperature from OpenWeather, then exposing a cut-down version of that as a REST endpoint so all my servers could grab it without me being rate-limited by OpenWeather for abusing my free API key. Another one re-hosted an image that changes a couple of times a day from an unreliable service.
Having figured out how to use the GitHub package registry, I was a bit inspired by this blog post from Florin Lipan to deliver all my little static websites as Docker containers. I’m not as focused as he is about making them tiny, but I did steal the idea of using BusyBox httpd for serving them, resulting in about 4MB containers. That’s small enough for me, and since they are all very similar, there’s a fair bit of layer reuse going on.
I quite like GitHub scanning all my code and sending me security advisories. Here’s today’s:
With these, and my dependabot alerts, fixing them is usually just a matter of pulling down the project, running an npm update, building any artifacts, then pushing it back up. But today, not so:
It’s probably worth revisiting what the package-lock.json does. It contains all the versions of any packages you’ve imported, and their dependencies. The idea is that this will make the build reproducible. We don’t commit the node_modules folder (that actually contains all that package code), but npm can reproduce it exactly by using the version information in the package-lock.json file. Here’s a snippet where you can see all those versions:
If you find yourself copying over a source file from one Node project to another because it’s a handy utility you wrote and are used to using, you’re only doing it half right. A better way to do this is to publish your utility to the Node Package Manager (NPM). That way you can just import your utility where ever you need it, it will live in the node_modules of any project that uses it, and most importantly, updates are sorted out automatically - because that’s what package managers are good at.
My default approach to web apps at the moment is Node/Express SSR. I needed to have users be able to upload files this week, and as usual there’s an express middleware that makes it trivial. This post just steps through using multer to make it simple to enable file uploads on your website.
Before we look at file uploading, it’s worth just explaining how it fits with the other tools we’re using:
Pretty much every serious web app needs to include a way for users to log in securely and to be served their content. Since there’s a lot of complexity in this, it’s highly advisable to use good libraries to support this. In a future post we’re going to use those libraries, but first I want to explain what’s happening at the lower level and tease out some of the concepts as we build a secure system from the ground up.
This post looks at the context for some of my thinking about AI for supporting software development, and where I’ve landed on it for the time being.
I briefly wrote about ChatGPT’s coding ability at the end of 2022. The wide availability of this tool marked the beginning of what I think can fairly be described as a revolution. The controversies that have crystalised since have not dampened my amazement of this step forward in what compute can do, especially around natural language processing.
A ‘dockerfile’ contains all the instructions to build a Docker image. Here’s my first draft for a project I’m working on:
FROM node:20
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 3000
CMD ["node", "server.js"]
COPY . . is copying all of the files in my project into the working directory of the image so they can be run. Of course we don’t need them all for the app - for example the node_modules directory will be created when we npm install so no need to copy that, and I don’t need all my dot files in the container.
I’ve mentioned using NGINX as an interface between the internet and a service a while ago. This works by all incoming traffic coming to NGINX, and NGINX determining which service that traffic should go (from the NGINX config files) then acting as a middleman. This functionality is generally referred to as a ‘reverse proxy’.
This is nice for a few reasons:
When I wrote the install instructions for mdserver (little Markdown server Node app) on it’s github page it was something like:
npm startWhich is great if you know how to do those things (they are bread and butter to a web dev) but not if you’re a self-hoster who just wants a web server that converts markdown to HTML on the fly. For any situation where you just want to use the app, what you probably want is a Docker image of the app.