How to downsize a tiny web server and the services on it

May 9, 2023

Until yesterday, I hosted this website on DigitalOcean. Now it’s on EC2 instead. These are a few notes about how and why.

The old setup

For nine years, I’ve hosted this site on the same virtual machine on DigitalOcean. It was originally Ubuntu 14, and later upgraded to Ubuntu 18, which is now EOL.

It’s always been a tiny system running NGINX. Behind NGINX, I’ve hosted a bunch of different web services. There were several WordPress sites, a short-lived Drupal site, and even some raw PHP scripts. There were a couple of toy Ruby on Rails projects. They all had hosting configuration, SSL certs, their own Linux users, their own databases (MySQL, SQLite), their own server processes (Unicorn, PHP-FPM). Most of them have been decommissioned, and along the way, the system got full of the debris of old projects.

(It’s great to have a sandbox. Highly recommended.)

The server itself was originally the classic $5 DigitalOcean droplet. It has 1gb of RAM and enough disk space for my projects. I also paid an extra 20% for automatic backups, just in case.

But recently, DigitalOcean raised all the prices by 20%, so I started to wonder: Is $7.20/month really the best I can do for basic Linux web hosting?

Tiny web servers on AWS EC2

I like having a Linux web server to play with. I don’t want to go serverless. I don’t want to move all my static sites to S3 and my remaining back end services to AWS Lambda. And while DigitalOcean isn’t the absolute cheapest option for a cheap linux VPS, most of the competition is not a lot cheaper. (And DigitalOcean has been flawless for my use case, with a nice user interface and excellent service, so that’s worth something too.)

But then I noticed that you can get a t4g.nano EC2 instance for $22/year, which is 69% cheaper than DigitalOcean. The big tradeoff is that it’s 500mb of RAM (and an expectation of very low average load, which is probably fine for me 🤞🏼).

So I thought it over and decided that the cost savings was worthwhile. And maybe I could clean up the cruft from my old server while I was at it.

Notes on Amazon Linux 2023

I spun up a new EC2 instance with Amazon Linux and moved all my static sites there. Here are a few notes on that experience.

It does not have Cron. I was kind of amazed by this, since Cron seems like a bedrock part of Unix-like systems. The Amazon Linux devs feel that you can just use systemd timers instead. I only need this functionality for renewing letsencrypt certs, so I followed Steven Westmoreland’s handy instructions to set that up.
The package manager is annoying compared to Ubuntu. There’s churn from one Amazon Linux version to the next, so some of the docs are useless, and some basic packages are unavailable. In particular, Certbot seems like it probably should be present in every Linux package manager by this point. I had to install it with pip, which is a barely-supported official installation approach. I guess Amazon just doesn’t want to encourage using Letsencrypt.

Other than those minor details, it was pretty straightforward to get a new system set up and running. It isn’t exactly the same as Ubuntu, but the differences aren’t consequential.

From Ruby to Go

It was easy to migrate the static HTML sites from one server to another. You just create the right web root directories, copy the NGINX vhost configuration from the old server (with a few updates), and point the deploy scripts to a new place.

My static sites are all built by Middleman, so the deploy scripts are very simple, like this:

#!/bin/bash

bundle exec middleman build
rsync -rzvu build/ webserver:/path/to/webroot

But I do have one project left over that isn’t a static site. It’s a tiny Rails app that shows a little art project about leaving academia. It did some server side rendering for a few templates, accepted user input, and updated records in a SQLite database. Using Rails made it very quick to put together (it was the classic “optimize for dev time” approach).

The problem is, even a tiny Rails app takes a couple of hundred megabytes of RAM to run. It’s not a good choice for a very tiny, low resource web server.

So I decided to rewrite my Rails app like this:

All the front end code (HTML/CSS/Javascript) was moved into a new React app. I’ve never used React before, but it was easy to get going and spin up some basic components. I didn’t try to write a single page app; I just used React to render components on top of static HTML files served by NGINX. The HTML/CSS mostly stayed the same; the Javascript had to get rewritten, but it was fun to do that.
All the back end code was rewritten in Go, which is a language I have never touched before. I was looking for a language that could offer good performance and low resource use, but something with a better developer experience than writing a web service in C. I looked a little bit at Rust and Go; Go was the obvious winner for my use case. It turned out to take less than 500 lines of Go to spin up a basic web service that could accept form submissions, update a SQLite database, and write a JSON file that provided data to the React user interface.

I felt pleased with myself for figuring out how to write a Systemd service that would deploy my app properly, complete with logging and configuration in env vars. I even wrote a handy deployment script that builds for ARM64 (because my EC2 instance runs on ARM), copies the compiled application to the server, stops the service, copies the compiled application to the right place on the filesystem, and restarts the service.

The new Go back end service runs in 13 MB of memory instead of 97MB, and is easier to manage (because it has fewer moving parts and a simpler deployment process).

Migrating DNS

One of the worst parts about leaving DigitalOcean was leaving behind their excellent interface for updating DNS records.

I changed nameservers for my domain, and I moved all my DNS records over to the DNS system provided by my domain registrar (currently NameCheap). But I was pretty frustrated with how much less nice their interface was compared to DO. It required some downtime for the site as well, basically because of the bad UI choices (no way to provide structured data, or to provide configuration before changing the nameserver setting).

This is just my private website, so it was OK. But in a medium-sized org, I’d think you would probably not want to ever switch nameservers if you could possibly help it, or at least you would use DNS services from someone who provided a much better admin experience.

Was this worth it?

If I’m being honest, probably the cost savings ($50/year) was not worth the several evenings I spent moving everything around and writing a new Go application. If I really counted up the hours to move everything over and write two new projects (golang/React), even though it was a relatively quick and straightforward project, I would probably bill someone at least a few thousand dollars at professional software rates. Probably a lot more if it were a full fledged consulting project.

But money aside, it feels excellent to learn new things. It’s great to explore new hosting options, new infrastructure, new programming languages. So … yes it was worth it; but the value was more intellectual than economic.