Eli Thorkelson

Bike safety rules in cities

2026-04-03T20:07:00+00:00

Outside of work, I really like to ride my bike. I’m not really interested in racing or the competitive side of cycling; but I love the experience of it, which always feels a little like flying, and I love how much you can see a city from a bike. It’s like a whole other way of seeing the world that you can’t get from walking or any other kind of transportation. I moved to Chicago when I was 23 and my bike was my main form of transportation for years. I never owned a car in those days; I just rode my bike everywhere. There was a year when I had a 13 mile daily bike commute — I don’t think I have ever been in such good shape. It’s world-changing, somehow, to spend so much time doing this activity.

Unfortunately, riding a bicycle in a city can also lead to crashes. It is objectively somewhat risky.

So every time I have any crash or incident, I’ve started to do a risk analysis afterwards, to see what I could do differently in the future. (Here, it starts to intersect with professional software development just a little bit, inasmuch as formal risk modeling is part of software engineering.)

Ideally, if you find out what scenarios are likely to cause bike accidents, you can avoid putting yourself in those situations again.

Over time, I’ve formed a list of rules for bike riding.

Basic rules

Reduce speed. Speed is one of the biggest risk factors while biking, because it makes accidents more likely and it also makes them so much worse. Kinetic energy is proportional to the square of your velocity, so every time you double your velocity, a crash will have 4x as much energy to dissipate into your body. Also, speed reduces your reaction time; the faster you go, the less time you will have to react to any approaching risks. So, Rule 1: don’t go super fast. (I enforce this for myself on my road bikes, which have front and rear gears, by almost never shifting the front shifter into the biggest (outer) ring - I figure that the increased safety is worth going a little slower.)
Scan the road surface carefully and continuously. Mentally note any bumps, dips, cracks, grooves, potholes, drains, grates, speed bumps, or any other imperfections in your path, so that you have a detailed picture of exactly what risks are approaching on the roadway. Over time, you can develop intuitions about how your bike will interact with any given obstacle (“this small bump will be fine,” “this sudden dip in the road looks dangerous,” or “this is a huge bump and it’s too late to avoid it, eep, hold on tight”).
Constantly scan the environment for all other vehicles, bikes, pedestrians, or any other moving objects. Estimate when or if they might intersect your bike’s future trajectory, or, worse, if they might hit you based on their current path.
Scan behind yourself for any approaching vehicles. Especially check when you hear noises, or if you are considering moving farther left into the travel lane.
When possible, make eye contact with approaching people or vehicles, or wave at them. You want to communicate with people so you can establish mutual awareness and (ideally) respect. You want to be sure that they know you are there, and if possible, ensure that they are planning not to hit you. I usually try to make eye contact first. If I’m not sure that eye contact is working, I might wave at them too. People’s eyes tend to notice motion.
Choose your trajectory carefully and precisely on the road. Choose your path to avoid obstacles and minimize risks, given everything you know about the situation around you, and given your state of communication with other people and vehicles. (The longer I ride my bike, the easier it is to place it precisely where I want it to go on the road.) You might want to ride closer or farther to the curb; you might want to avoid obstacles or go over them; you might want to slow down or speed up. The point is, you are making all these choices in view of the situation.
There is a flip side to the last point: Don’t change your trajectory without knowing what’s happening around you, especially what’s behind you. Just because you see an obstacle in your path, this does not mean it is safe to swerve out of the way - you could be swerving into a new problem. (In emergencies, you might need to swerve without being absolutely sure of what’s behind you; this is a risky choice, even when it’s necessary.)
If you spot a dangerous situation like a possible collision approaching, remember that you have more than one option for how to avoid it. Obviously, you can hit the brakes and slow down. But you can also steer out of the way or, unintuitively, you can also speed up. (Speeding up is a perfectly valid strategy for avoiding collisions.) Obviously in a real emergency you won’t have much time to think about your options, but it’s good to know in theory that you have more options than just hitting the brakes.
It is OK to just pull over and stop momentarily if you need to. For instance, sometimes when an obstacle is approaching (a parked car in an otherwise empty street), instead of moving farther out into the road and going around it, I pull over and stop for a few seconds to let traffic pass me, and then continue.

Intersections and traffic

Respect traffic signs and signals the same as a car, especially if there are any cars around whatsoever. It makes drivers mad at you if you openly ignore the traffic signals and stop signs. It is also just safer and easier for everyone if you respect the traffic rules. This is one way we can help to create a consistent, predictable environment.
Intersections are generally high risk places. Always assume that car drivers don’t see you until they have proven otherwise. Assume people might ignore the traffic signals (for example, they might keep going after the stoplight turns red). Don’t enter an intersection until you are sure it is safe.
It’s often safer at intersections to join the vehicle traffic and wait your turn than to try to dodge along the side of a line of stopped cars. You do have a legal right to do this as well, for what that’s worth.
Driveways are also dangerous, though not as dangerous as intersections, because people entering or leaving a driveway don’t always see you. Check driveways as you pass for signs of motion. Slow down if you are approaching a driveway and can’t see what’s inside, like if it is behind a construction fence or a gate.
Never ride on sidewalks unless you are accompanying kids on a bike ride.
Bike lanes at the edge of the road are a missed blessing. Cars don’t always pay attention to them. People tend to park across them, meaning you will have to go back out into the vehicle lane. Bike lanes tend to have more debris, fallen leaves, and broken glass. Don’t avoid bike lanes, but use caution.
Especially beware of bike lanes at intersections, because any traffic turning will have to cross the bike lane and may not see you.
Consider traffic patterns when monitoring the road. For example, during rush hour, there will not only be more traffic on the roads, there will also be a higher rate of people entering and exiting driveways and parking lots, so you have to be extra cautious on both sides.

Relationships with drivers

More communication is usually better. Don’t hesitate to communicate.
Make noise as needed for safety reasons. Yell something at drivers to help them notice you, shout at pedestrians to watch out, whatever makes things safer. It’s better to be a little bit rude than for someone to get hurt.
It’s usually better to be extra nice to people (like let them go first, let them pass you, generally try to help them) than otherwise, and it probably makes you a little safer.
Signal your left or right turns with your hands. It’s safer for you if other people understand your trajectory.
Drivers may try to pass you in dangerous ways. You can try to help avert catastrophes. For instance, if a car is trying to pass you from behind, and you’re ahead of them on a curve, it’s smart to signal the car with hand signals whether it’s safe to proceed. (I have seen some near misses because drivers moved into the oncoming traffic lane to pass me, but someone was coming.)
Think about what messages you are sending to other vehicles because of how you ride. Sometimes I deliberately wobble just a little bit around the road, to make approaching vehicles think I might be a little bit incompetent and they should give me a little more space. Some drivers seem to pass a little closer if you are going in an even, straight line.

Failures of communication

Don’t assume that drivers follow the rules about how to take turns at stop signs. A lot of people will just go once they have a brief stop, even if it isn’t supposed to be their turn.
Any driver who is looking at their phone is an existential risk to your life and should be monitored with extreme caution. Such drivers probably don’t see you, their performance is impaired, and you probably can’t communicate with them at all.
I am scared of self-driving cars, as a cyclist, because I don’t know how to communicate with them, or if it is even possible to communicate with them.

Parked cars

Never ride close to parked cars, because their doors can open in front of you at any time, potentially causing a horrible crash.
If in spite of the previous rule, you do have to ride close to parked cars, try to watch them to see if anyone is inside them. You can sometimes spot people’s heads through the rear windshields or windows, giving some clues about whether cars are occupied.
If you do have to ride close to parked cars, it is also wise to slow down.

Road shapes

Always leave space between yourself and the very edges of the road. You can crash if you rub up against a curb. You can also crash by slipping off the side of a roadway that is built up above the surroundings (this is more common in rural environments with thick layers of asphalt).
Always cross railroad tracks at a right angle to the rails. If you approach at an oblique (near-parallel) angle, your wheel can get trapped in the groove between the rail and the roadway, causing an instant crash. (Ask me how I know.)
Always be very cautious about anything in the road whose surface is nearly parallel to your direction of travel, because bike wheels are not able to safely negotiate such surfaces. For example, if there is a groove in the road that is nearly parallel to the way you are going, avoid it; your wheels can slip into it and tip you over. Cross grooved or lipped shapes at perpendicular angles if at all possible.

Relationships with pedestrians

Always defer to pedestrians, but especially if they are kids, parents with strollers, or otherwise more vulnerable people.
Anyone running or jogging is a special danger to bikes, because they tend to run out into the road without looking, and they move fast compared to other pedestrians.
If small kids say hi to you, always say hi back to them. (This one isn’t about safety, just about humanity.)
In general, bike riding can be a nice way to have some tiny, fleeting encounters with strangers, even if it’s just a smile or two in passing.

Miscellaneous risks

Beware darkness. It’s easier to spot vehicles in the dark (because of their headlights), but it makes it hard to see the surface of the road in detail.
Beware slippery conditions. Slippery road surfaces are probably the second most important risk factor, especially if you ride a road bike with skinny tires, since traction from the bike wheels is greatly reduced when it is raining. So, increase caution and reduce speed whenever the roads are wet or snowy. Increase caution even more when it’s cold if there is any ice on the road.
Don’t carry anything in your hands while you bike. This can make it harder to handle the bike in complex moments, leading to crashes.

Gear

Always wear a bike helmet (even though they are kind of annoying). My cousin died because he didn’t wear a bike helmet and someone opened their car door in his path.
Get a really bright headlight. I once asked some Chicago bicycle cops what brand they used (it was ludicrously bright) and then I bought that. (For the record, it was Magic Shine).
Always have a back light. Probably have it flash for extra visibility.
It is strongly encouraged to have a bell, horn, or other audible signal.
For extra visibility, decorate your helmet, backpack, etc, with extra reflectors, shiny patches or extra lights.
If possible, wear bright or even fluorescent/high viz clothes. (I don’t always do this, but it is definitely a safety factor.)
Consider environmental issues, especially on longer rides - like bring enough (or any) water if it is warm, consider sunscreen and UV protection on exposed skin, always have gloves in cool temps to keep your hands warm enough, all this. Being comfortable can become a safety factor, because discomfort can make you more agitated or inattentive.

Accidents

Don’t expect that anyone will necessarily help you if you crash. People have this horrible tendency to just keep going and not… do anything to help.
Maybe carry some really basic first aid stuff with you, like some bandaids in case you have minor scrapes that need covering
Don’t be afraid to ask for help if you need it, especially in emergencies. One time I crashed my bike because of a sudden dip in a road at night, and I ended up staggering to the first house nearby with a light on the porch. They ended up driving me home with my bike. People can be really kind.
However, sometimes people are awful and will turn you away even if you’re hurt and ask them for help. I don’t even know what to say about those people, just know that unfortunately they are out there.

Caveats:

These rules work for me, but they are not intended to be universal.
They probably don’t make sense if you are in very different circumstances from me.
I might edit all these again in the future.
For transparency, this was last edited April 3, 2026.

Against the Five Whys

2026-02-28T13:51:00+00:00

My org has recently started using the Toyota Five Whys to analyze incidents.

The way we use it, it looks something like this:

Q1: Why did we see A?
A1: Because B.
Q2: Why did B?
A2: Because C.
Q3: Why did C?
A3: Because D.
Q4: Why did D?
A4: Because E.
Q5: Why did E?
A5: Because F.

The idea is that you get deeper into the process failures that lie behind a problem when you dig in like this.

But every time I write an RCA document, I find that the Five Whys is inadequate.

Why the Five Whys is inadequate

It is inadequate because it presumes a basically linear causality.

Almost all interesting incidents have multiple causality. Meaning that there isn’t just a straight line between A and F. There is a cluster of issues.

When I have to fill out this document, I usually hack the format by listing several causes under A2. And I put several causes under A3. And so on.

In this way, you can respect the form of the Five Whys while also having the intellectual integrity to acknowledge that causality is almost always multiple and incidents happen at the junction between multiple causal flows.

Alternative approaches

I’m not the first to criticize the Five Whys; see the criticism section in the Wikipedia article.

It turns out that researchers have already invented better forms of analysis that represent multiple forces better.

One approach is the “fishbone diagram”: it shows multiple forces that collectively caused an incident.

The idea is that lots of things had to collectively go wrong to cause an incident, and you diagram them in layers.

An even better approach is described in Alan J. Card’s 2013 paper, A new tool for hazard analysis and force field analysis: The Lovebug Diagram. A “lovebug diagram” is basically two fishbone diagrams that represent opposing directions. The name comes from “a mating pair of Plecia nearctica, commonly known as lovebugs.”

(Published version here, but I don’t have access anymore because I’m not in academia :/)

It’s a great visualization because it shows multiple forces that caused an incident and also multiple forces that tried to prevent an incident. And this, in my mind, is a more realistic image of how things actually happen. We do have safety controls. In an incident, it’s just that they weren’t enough.

I wish I could use these diagrams in our RCAs instead of Five Whys. But I guess the Five Whys is better than no methodology whatsoever, or just letting people write down that an incident happened because of one single reason. Methodology is better when it provides a lower bound for our inquiries, but it doesn’t have to be an upper bound.

Anyway, causality is always multiple in complex systems. I won’t stop insisting on this.

Ruby 'require' can deadlock

2026-02-15T17:19:00+00:00

Another short post with a surprising thing to learn about Ruby.

We had a threaded service that called require on a certain directory at runtime. It was expected to run the service only on one thread at a time.

My mental model of this was:

require is basically a no op once a file is already loaded
so it doesn’t cost much to run require each time the threaded service executes.

One day, due to some weirdness in a testing environment, this threaded service ran on multiple threads simultaneously.

Our Sidekiq worker came to a halt.

My colleague Dmytro found out that, if you try to require the same directory from multiple threads simultaneously, the process can deadlock.

Reproducing

A simple reproduction looks like this:

# alpha.rb
puts "[Thread A] Inside alpha.rb, sleeping before requiring beta..."
sleep 1
require_relative 'beta'
puts "[Thread A] alpha.rb finished."

# beta.rb
puts "[Thread B] Inside beta.rb, sleeping before requiring alpha..."
sleep 1
require_relative 'alpha'
puts "[Thread B] beta.rb finished."

# experiment.rb
puts "--- Starting Deadlock Reproduction ---"

t1 = Thread.new do
  puts "Thread 1: Requiring 'alpha'..."
  require_relative 'alpha'
end

t2 = Thread.new do
  puts "Thread 2: Requiring 'beta'..."
  require_relative 'beta'
end

[t1, t2].each(&:join)
puts "This will never print because of the deadlock."

This prints the following when I test it:

--- Starting Deadlock Reproduction ---
Thread 1: Requiring 'alpha'...
[Thread A] Inside alpha.rb, sleeping before requiring beta...
Thread 2: Requiring 'beta'...
[Thread B] Inside beta.rb, sleeping before requiring alpha...
experiment.rb:13:in 'Thread#join': No live threads left. Deadlock? (fatal)
3 threads, 3 sleeps current:0x0000000aa933b400 main thread:0x00000001030b86e0
* #<Thread:0x00000001023e8d18 sleep_forever>
   rb_thread_t:0x00000001030b86e0 native:0x0000000200c23080 int:0
   experiment.rb:13:in 'Thread#join'
   experiment.rb:13:in 'Array#each'
   experiment.rb:13:in '<main>'
* #<Thread:0x0000000121df67c8 experiment.rb:3 sleep_forever>
   rb_thread_t:0x0000000aa933b200 native:0x000000016dcbf000 int:0 mutex:3 cond:1
    depended by: tb_thread_id:0x00000001030b86e0
   /Users/eli/scratch/require-deadlock/alpha.rb:3:in 'Kernel#require_relative'
   /Users/eli/scratch/require-deadlock/alpha.rb:3:in '<top (required)>'
   experiment.rb:5:in 'Kernel#require_relative'
   experiment.rb:5:in 'block in <main>'
* #<Thread:0x0000000121df66b0 experiment.rb:8 sleep_forever>
   rb_thread_t:0x0000000aa933b400 native:0x000000016ddcb000 int:0
   /Users/eli/scratch/require-deadlock/beta.rb:3:in 'Kernel#require_relative'
   /Users/eli/scratch/require-deadlock/beta.rb:3:in '<top (required)>'
   experiment.rb:10:in 'Kernel#require_relative'
   experiment.rb:10:in 'block in <main>'

    from experiment.rb:13:in 'Array#each'
    from experiment.rb:13:in '<main>'

Why does this happen

Well, short answer, there is a lock inside require and multiple threads can deadlock when they contend for requiring the same resources.

Long answer — I have not had time to look into the Ruby internals around this, but the Ruby Hacking Guide does describe some of the mechanisms.

From Chapter 18, Loading:

The problem comes after. Like the comment says “the loading of Ruby programs is serialised”. In other words, a file can only be loaded from one thread, and if during the loading another thread tries to load the same file, that thread will wait for the first loading to be finished […]

The process to enter the waiting state is simple. A st_table is created in loading_tbl, the association “feature=>waiting thread” is recorded in it. curr_thread is in eval.c’s functions, its value is the current running thread.

The mechanism to enter the waiting state is very simple. A st_table is created in the loading_tbl global variable, and a “feature=>loading thread” association is created. curr_thread is a variable from eval.c, and its value is the currently running thread. That makes an exclusive lock. And in rb_feature_p(), we wait for the loading thread to end like the following.

When rb_thread_schedule() is called, the control is transferred to an other thread, and this function only returns after the control returned back to the thread where it was called. When the file name disappears from loading_tbl, the loading is finished so the function can end. The curr_thread check is not to lock itself (figure 1).

(I have not checked if this is still 100% accurate; the Ruby Hacking Guide is old.)

Fix

We changed our code to avoid dynamic require calls at runtime. Instead we loaded the necessary code at start time.

(It is code that does not need to be loaded in all scenarios, so that’s why it was not loaded by default in the first place. It is pointless to load code when it isn’t needed; but then the hard part is to reliably know when it is needed.)

Anyway, the point is, I just had no idea that require had an underlying lock. It’s hard to plan around things you don’t know exist.

Raise isn't a Ruby reserved word

2026-02-07T11:48:00+00:00

I was surprised to read this in the Ruby Hacking Guide:

In Ruby exceptions come in the form of the function style method raise. raise is not a reserved word.

Wait, what?

… raise is not a reserved word in Ruby?

It sure seems like part of the core structure of the language…

Let’s check the official list of Ruby keywords.

The following keywords start with R:

redo - Restarts execution in the current block. See control expressions

rescue - Starts an exception section of code in a begin block. See exception handling

retry - Retries an exception block. See exception handling

return - Exits a method. See methods. If met in top-level scope, immediately stops interpretation of the current file.

This is so weird. raise isn’t there.

Let’s test:

irb(main):001* def raise(beep)
irb(main):002*   puts "boop"
irb(main):003> end
=> :raise
irb(main):004> boop
boop
boop
boop
               rbboop
boop           ri                      █
boop           raise                   █
boop           rand
               raiboope
boop           raise
boop
irb(main):004> raise 2
boop
=> nil
irb(main):005> boop
boop
boop
boop
               rbboop
boop           ri                      █
boop           raise                   █
boop           rand
               raiboope
boop           raise
boop
irb(main):005> raise 6
boop
=> nil

So yes, raise is just a method, but redefine it at your own risk, because my simple definition here gets us deeply into Zalgo text behavior. It looks like IRB uses raise internally as part of its autocomplete functionality. It’s fun to see the side effect happening constantly while entering text.

If we redefine raise without any arguments, then the effect doesn’t occur. Presumably there is an internal ArgumentError before it enters the method body.

irb(main):001* def raise
irb(main):002*   puts "zalgo"
irb(main):003> end
=> :raise
irb(main):004> puts "test"
test
=> nil

OK, what else can you define here?

irb(main):001* def rescue
irb(main):002*   puts "help!"
irb(main):003> end
=> :rescue
irb(main):004> begin; raise "problem"; rescue => e; puts e.inspect; end
#<RuntimeError: problem>
=> nil
irb(main):005> send :rescue
help!
=> nil

So rescue is a valid method name, but, reasonably enough, the rescue keyword takes precedence over it if you do define it.

The same is true for other keywords - you can define them as methods if you want, but then you can’t call them normally.

irb(main):034* def while(condition)
irb(main):035*   loop do
irb(main):037*     break unless condition.call
irb(main):036*     yield
irb(main):038*   end
irb(main):039> end
=> :while
irb(main):040* send :while, -> { Kernel.rand < 0.5 } do
irb(main):041*   print "."
irb(main):042> end
# This prints a random number of dots and then halts.

Congratulations, we just reimplemented while without using while. Admittedly, it is useless, unintuitive, and dangerous if you do this in real application code. But in any case, there is no particular rule against using a reserved word for a method name.

Per the docs:

Method names may be one of the operators or must start a letter or a character with the eighth bit set. It may contain letters, numbers, an _ (underscore or low line) or a character with the eighth bit set. The convention is to use underscores to separate words in a multiword method name:

I’m imagining that in the Ruby parser implementation, whatever you type after def is parsed only as a Ruby method name and not interpreted as a possible keyword.

Meanwhile, raise is implemented as a method on the Kernel module that is included by Object, so that’s why it is both ubiquitous and stupid to redefine.

// ruby/eval.c

rb_define_global_function("raise", f_raise, -1);

// then f_raise delegates to rb_f_raise

static VALUE
f_raise(int c, VALUE *v, VALUE _)
{
    return rb_f_raise(c, v);
}

// which is implemented like this:
VALUE
rb_f_raise(int argc, VALUE *argv)
{
    VALUE cause = Qundef;
    argc = extract_raise_options(argc, argv, &cause);

    VALUE exception;

    // Bare re-raise case:
    if (argc == 0) {
        // Cause was extracted, but no arguments were provided:
        if (!UNDEF_P(cause)) {
            rb_raise(rb_eArgError, "only cause is given with no arguments");
        }

        // Otherwise, re-raise the current exception:
        exception = get_errinfo();
        if (!NIL_P(exception)) {
            argc = 1;
            argv = &exception;
        }
    }

    rb_raise_jump(rb_make_exception(argc, argv), cause);

    UNREACHABLE_RETURN(Qnil);
}

(It looks like it then uses rb_raise_jump to do the call stack mechanics of exception handling, since all we really see here is some argument parsing, but this isn’t the place to dig deeper into how it works.)

I do wonder why raise isn’t just a reserved word in the first place. I guess there must be a scenario where you would want to redefine it for some metaprogramming project, but I can’t think of a very sane use case.

The risks of looking too closely at things

2025-11-06T23:26:00+00:00

Sometimes it seems like, the closer you look at a running system, the more problems you find with it.

It can feel like the very act of looking creates weird anomalies that weren’t there if you didn’t look.

Of course, they were there already. You just weren’t noticing them.

This seems to be true across many domains of life.

Anomalies in a human body

Indulge me in a personal story. I often have funny results when I get my bloodwork done at my primary care clinic. It’s common in my family to have a slightly low platelet count, and other randomness somewhat outside the norms.

One year, my primary care provider got worried about my lab results and sent me to a hematologist.

The hematologist was intrigued by my case. Specialists love mysteries. He ran a lot of blood tests. He started to speculate about numerous obscure syndromes that I could hypothetically have had. He found nothing with the easy tests, so he moved on to the complicated, obscure tests. Some of them involved PCR gene sequencing. Before I knew it, he had ordered more than $30,000 of medical tests.

He never found any actual disease or any other problem that needed treatment. He sent me home with instructions to take more Vitamin B12 and come back again to be re-tested later.

What I have taken from this is: If you look closely enough at something as complicated as a human body, you will usually find anomalies. And these anomalies don’t necessarily represent symptoms or illnesses. They might just… exist.

Eventually, doctors started to use the word “idiopathic” to describe my case. Idiopathic just means “we don’t know why it happens, but no need to look into it further.”

Medical analysis, which is all encoded in software these days, typically relies on population-level benchmarks for anomaly detection, like “platelet count below 120 is not normal.” But in fact, the same reporting thresholds don’t always produce useful data for everyone. Context-free rules frequently produce false positives.

Beware of letting specialists go down rabbit holes that they can bill you for.

Anomalies in a building

The same effect applies in other areas too. Enough about bodies. Let’s talk about houses.

We live in a relatively new house; it is about ten years old. If I look at it from a distance, it seems to be in pretty good shape. When people visit, they often say it looks good.

But if I look at it more closely, I start to see lots of issues.

The paint is peeling on a drawer. There is a smudge on the doorframe where our kids frequently touch. There is a small crack between a concrete surface and a brick surface. Little vines are climbing up the foundation. The drainage pipe in the yard looks clogged by leaves. Dust accumulates on top of the light fixtures where you can’t see it. The caulk in the bathtub needs fixing. A post is starting to get rotted out where it gets wet. The windows need washing, even where it’s very hard to reach them.

Some of these are easy maintenance issues, once they are noticed. I go around the foundation a few times a year and remove the vines.

Others, I have no idea how to fix, since I am not a carpenter. The drainage pipe is a conundrum.

Other issues I don’t even know how to detect by myself, even if I try. That rotted post was detected by a professional; we missed it. Even though it was potentially a structural problem.

When I was younger, I naively imagined that houses mostly just keep existing once they were built, as long as the roof was intact to keep the rain out. Now I realize that all structures are constantly changing in their environment, and they always require attention. Permanently.

Nevertheless, some of the issues - like the smudged doorframe and the dusty light fixtures — are very, very minor. You can drive yourself crazy if you try to fix them all.

As with the human body, the issues show up as soon as you look for them. But when you aren’t looking, they seem to vanish again. Some anomalies are safe to ignore; others not so much.

Anomalies in software systems

Now back to software.

There’s a kind of iterative cycle in observing our systems at work. It goes like this:

Due to a problem or crisis, we start looking closely at our logs and stats.
While looking at the logs, numerous other issues jump out at us. These issues were usually there for a while, but no one noticed.
Some of the new issues turn out to be minor or just noise. Others require fixing.
We create some new alerts or metrics to filter for whatever signal we want to keep track of next time.
We stop looking at the raw logs.
New issues gradually appear in the logs, but no one notices them if they don’t hit the alert thresholds.
The whole cycle repeats.

One time during an incident call, we noticed an absolute flood of errors in our web request logs that had been happening for a long while. It turns out that some load balancer healthchecks were failing in a loop. It never caused any real problem, but it spammed us with errors.

Alert rules are not a solution to this general problem. In audio processing terms, one can see an alert rule as a bandpass filter against a signal. Unfortunately, all filters, and thus all alert rules, have signal/noise ratios, and thus they will pass noise. Especially for simple threshold-based alerts, there is a tradeoff between not enough filtering (then you are spamming yourself with too much noise) or too much filtering (then you are going to overlook real issues until they become incidents). Setting alert thresholds is informed guesswork (the polite word is “an empirical problem”).

In fact, it requires expertise to even figure out if something is signal or noise. It’s not an easy problem. It requires context and often demands a judgment call about each case: Does this matter?

My current bandaid for this problem is to randomly look at the raw data sometimes, just to see what I notice. Take random walks through your data. Randomly inspect at your house to see what you see.

There’s no real substitute for looking at a log stream and trying to make sense of it. I almost always find something interesting. I also look every so often at dashboards that don’t alert and check for any interesting trends in the graphs (see also reading diagrams).

It’s probably getting more viable to ask a LLM to watch a log stream and check for anomalies. That would be a huge help if it can be done at scale. But we do have large log streams and there can be policy concerns, so I’m not sure if we’re ready for this yet. Also, we don’t know how reliable LLM analysis is, at this point.

Anyway, what I take from this is: The act of looking closely at something can be dangerous. It can lead you to a lot of new places. Some of them are useless. But you have to figure out how to do it, because otherwise, you only find out about the issues when they do become incidents. It’s better to stay ahead of the risks, if you can.

Reading graphs and diagrams

2025-10-11T11:06:00+00:00

I’ve been looking at a lot of operational dashboards lately.

The question presents itself: How do you read anything meaningful from a graph?

Training the eye

The human eye has exquisite pattern recognition capabilities.

And yet by default they are not always enough. They might have to be trained.

When I was 17 years old, still in high school, I got interested in astronomy. I lived in a college town where you could take some classes at the local university, so first I took the undergraduate astronomy survey course, and then enrolled in an independent study with a physics professor. She was probably 65 years old and she was happy, I think, that some kid was so interested in her field.

So we worked through some chapters of an undergraduate textbook, R.C. Bless’ Discovering the Cosmos (1996), and covered various topics in stellar evolution. I remember looking at a photo of some stellar phenomenon in the textbook and being asked: “What do you see here?”

“I don’t see anything,” I said.

And my professor pointed out some little dots on the image and explained what they meant.

She could see something I just… couldn’t.

Even though it was a simple diagram that we were both looking at.

You have to train the eye to see diagrams in a given domain.

Diagrams tell stories

Enough about astronomy; let’s talk about web services.

Suppose you have a graph over time of 5xx (internal error) responses from a set of web services in different environments.

Someone is asking you, “Do we currently have an ongoing incidnt? Or is the incident over?”

You find some data. Let’s say it looks like this:

You can tell a story about this image that’s something like:

“There was a period of heightened errors on our production system earlier, lasting for a few hours. It is now done. There is also a more recent error spike in the training environment, but it is very low priority because the training environment is not a critical system.”

But if you had a different context, another story you could tell from this would be:

“There is always a non zero error rate for this system, which is normal. There are a few relative blips in the error rate, but they are all very tiny compared to the 10 million successful requests per hour that this service handles.”

You have to know what counts as “baseline” to tell a story from a diagram.

Yet another story you could tell about this diagram would be like:

“This is a graph based on application server data, with telemetry emitted after each request is complete. However, upstream of this system is a load balancer, which has recently started showing much bigger error spikes up to 15,000 errors per hour. These bursts don’t appear at all on the current graph, because these requests are overloading the application server and breaking the instrumentation system.”

This brings us to a second important point.

All diagrams lie

You can’t trust diagrams. You can learn from them, but you have to understand they are, by design, partial and limited.

Here are a few ways that diagrams can lie. (There are many others.)

Diagrams are probably missing data. For instance, maybe there are cases that are invisible to the telemetry system you use. As we just suggested, if your telemetry comes from back-end application servers, it might miss errors that are only apparent upstream at the load balancer. Or if your application process crashes or gets halted by Kubernetes, it probably can’t report its last state before halting.
Visualization requires aggregation functions. Aggregation functions are inevitably going to mislead you. Do you want to see the median, the mean, the 95th percentile, the 99th percentile? There is a particular limit case in my Prometheus dashboards where the aggregation query fails to count the first value in a new series. I think it’s because they are based on increase and increase doesn’t handle this nicely - my guess is that it tries to compute a difference between time A and time B, but the difference between “undefined” and “1” is represented as 0, not 1.
Timeslices matter hugely. For instance, if you are counting events over time, you might have to count them over a given timeslice. And the graph is almost always going to look bumpier, noisier, when the timeslice is smaller. However, for performance reasons, you can’t use small timeslices over large intervals.
Metrics may have a collection interval below which you can’t disambiguate. If you scrape Prometheus metrics data every 60 seconds, for instance, then you simply cannot see sub-minute spikes in the data series, no matter what you try. As with music, the sampling frequency matters.

Nonlinear behavior is common

Diagrams can lend themselves to the illusion of linearity. You can often identify linear-looking traffic growth over time, or even determine visually that traffic appears steady state.

At some point, the illusion (or assumption) of mostly linear behavior is going to break down. Databases will run out of resource limits and performance will plummet. Small fluctuations in a noisy baseline graph will suddenly resolve into huge spikes in an outage.

You have to train yourself not to expect linear behavior in complex systems, because they are fundamentally nonlinear, and will hit limits suddenly.

You have to learn, gradually, how to include adequate “safety factors” in your assessment. “My system has used 80% of its available resources, does this mean we can keep waiting until we use 95% of resources? Or should we increase resources now?” It depends on your forecast of possible linear or nonlinear growth in the future.

This implies one last point.

Your understanding of diagrams reflects your understanding of the system

If you are a visually inclined person, then graphs and diagrams can be powerful tools for building understanding of a system.

However, at the same time, your baseline understanding of a system feeds into what you can read out of a diagram based on that system.

Ideally, these two constraints are mutually reinforcing instead of being mutually disabling.

For example, if you always deploy new code every Tuesday morning, then you might know intuitively that a latency spike on Tuesday mornings is related to the deployment event, and not otherwise concerning.

However, if you just see a latency spike on the most recent Tuesday, and you don’t have any other context, then you might spend some time chasing down various nonsense theories and speculations before you eventually figure out that the same spike happens every Tuesday.

(Here is a version of this that I’ve lived through: Q: “Why is there a huge latency spike towards the end of the month? What’s broken?” A: “Oh wait… it’s the end of the fiscal quarter and usage is always heightened at this time.”)

You have to let yourself learn from graphs and diagrams (we should never throw away any source of useful knowledge), while also recalling that graphs and diagrams often lie, and almost always omit important context.

As with so many other things, experience isn’t everything, but it counts for a lot.

On incident fatigue

2025-08-02T17:22:00+00:00

Once upon a time, in a galaxy far away, we had a week where things just kept breaking.

Some of the broken things were in production environments. Some were in engineering environments.

Some of them were incidents. Some of them were merely… issues.

The broken things were roughly like the following:

Request timeouts being reached in critical flows, where we didn’t previously expect to see them being reached.
Health checks failing because of seemingly unrelated configuration changes.
Mysterious CPU spikes in certain database instances. (Well, they were mysterious before we knew the explanation, anyway.)
Background jobs not running as scheduled.
Networking problems caused by configuration problems caused ultimately by cross-team misalignments.
A series of other awkward things that we can’t even get into here.

The details aren’t what matters. The point is that there were a lot of technical issues in a short span of time.

This was exacerbated by a series of other factors:

Nontechnical stakeholders were clamoring for answers.
Key people were out of the office.
We were dealing with really complicated causal chains. The symptoms of things were very far removed from the causes. Every issue required analyzing a long, complex set of effects.
There are several different logging systems, and not all requests are tracked in all of them.
We faced jurisdictional ambiguities about which team owns what.
We were spread out across so many different time zones.
We have so many different Slack channels.

I started to think that our incident response strategies were fundamentally built to handle one issue at a time.

So: What do you do when there are more incidents than there are people on call?

What we did: Improvisation

We did not explicitly have a plan for this situation. We expect that there will basically be one incident at a time. If only incidents were so well behaved.

So we improvised. We entered a zone of undefined organizational behavior.

It was a group of experienced software engineers, relatively committed to their work. We leaned on each other for support. People not on call got pulled in.

Soon there were people losing sleep, staying up late out of concern, and just getting worn out.

Later, things did start to calm down.

What would I do, next time?

What can one do to make it better?

I think it starts with recognizing that incident fatigue is just its own thing, categorically distinct from other kinds of problems. It is a particular state one can, unfortunately, get into.

Of course, tech business culture gives us some general strategies about what to do.

Strategy 1: Prioritization. When there are a lot of issues, rank them by urgency and only handle the most critical ones first. (Problem: Too many issues are critical, and/or their criticality is hard to assess. Eventually, major issues are left unfixed.)
Strategy 2: Deflection. When there are a lot of issues, try to send them elsewhere, see who else can handle them. (Problem: Dealing with organizational jurisdiction issues can take as much time and effort as just fixing the issue yourself. And the latter actually delivers some practical value. Problem 2: Over time, too much deflection can erode that sense of commitment that we can call “ownership” of technical systems.)
Strategy 3: Self-care. A very American cultural theory, it seems to me - the idea that probably you should try to do something extra nice for yourself to make up for the extra stress. (Problem: It doesn’t really solve anything, it just distracts.)

I’m not against any of these strategies, in general terms, but I’m not sure they help enough. I keep thinking that there’s more we could do.

Maybe there could be some other division of labor, where we move beyond the “first responder” model, and try to solve issues more directly as a team.

Maybe there could be some way to acknowledge the fatigue and get some extra rest.

Maybe it would help to organize a retrospective, not for specific issues, but for the whole set of issues. A space to think about how we handled things overall.

A little bit of collective recognition seems like it would help.

Coda: What we did afterwards

We did end up having a retrospective session to discuss these issues, which was helpful. We were able to identify certain common root causes among our issues.

I came away thinking: I would do more to call attention to incident fatigue sooner, if we ever get there again.

I would do more to ensure people get enough rest and don’t overwork.

Naive optimizations of CI job allocations

2025-07-20T11:42:00+00:00

One time I had a job interview with a live-coding challenge. It went badly, because I was too anxious to do well. It asked a question about optimally allocating test jobs across CI workers. In essence, a bin packing problem.

OK, so, test job allocation.

Suppose you have a set of test jobs and you want to allocate them to a set of workers for execution. What is the optimal way to do this?

At moments like this I wish I had taken the CS class that covers queueing theory and bin packing problems. If you never took that class, there’s no good way to deduce it all from scratch in a random job interview.

Well anyway, that interview went badly. It did not produce any working implementation, and I was obviously not hired. But I went home and wrote the code I wished I had been relaxed enough to write. It compares naive allocation algorithms with the following preconditions:

We know the runtime duration of each test file at the start. (Obviously not true in practice in any significant CI system, but we can probably use expected duration or historical average duration as a proxy here.)
The durations range from 1-251 seconds.

What counts as optimal?

We want to allocate job load as evenly as possible among workers, such that each worker finishes at the same time. We don’t want to have worker 1 finish first and sit idle, while worker 2 keeps working twice as long.

As the jobs have randomly large durations, there is no absolute guarantee that all workers can finish at the same moment. If you have two workers and two jobs, and job J1 takes 1 minute while J2 takes 3 minutes, then one worker is going to finish 2 minutes sooner than the other, no matter how you allocate them.

But in principle, especially as the individual job durations approach zero relative to the total length of the queue, you can evenly divide the jobs among workers and they will all finish simultaneously. I like to think here of the analogy with dividing a large volume of water into multiple buckets: if we were dividing a continuous volume V liters into N buckets, then each bucket should ideally contain the exact same quantity of water, namely V/N liters.

Of course, jobs do have discrete individual durations, so the analogy with dividing a fluid is inexact. But it still gives us an excellent target to measure against. For any allocation of a finite set of jobs across a given set of workers, we can measure how closely our allocation approaches the ideal case of perfect division across a worker pool.

Say that the total duration of a set of jobs is D seconds and the number of workers is N. Ideally, each worker should finish processing after D/N seconds. Let’s call this the optimal target duration for each worker, or t(optimal) for short.

You can then trivially measure the variance from the optimal outcome for each worker with:

100 * (t(actual) - t(optimal)) / t(optimal)

(Multiply by 100 to get an output in percent, so you can say that a particular worker finished 1% slower than optimal, or 30% faster, etc.)

Some naive allocation algorithms

Let’s suppose that before allocating jobs, we sort the input job set into an list that’s sorted by historical job duration, so we can allocate larger jobs from the tip and small jobs from the tail of the list.

We want to explore algorithms for sorting jobs into N worker queues. For simplicity, let’s stipulate that N must be a power of 2.

I tested some simple allocation algorithms:

Allocate from ends: Go through each worker in round robin form. Push the longest available test job onto each queue. If there are any more test jobs, also push the shortest available test job onto each queue. My inspiration here was about trying to balance out the slowest and fastest jobs across our available workers (so each queue gets part of the short and long tails).
Allocate to one worker until full: Compute t(optimal) for a given set of workers. Allocate jobs to the first worker until it contains >= t(optimal) jobs. Then allocate jobs to the second worker until it reaches or exceeds t(optimal), and so on for N workers. (I believe this is fairly similar to the next fit bin packing algorithm.)
Recursive partitioning: Create two buckets of jobs, B1 and B2. Iterate through all the jobs, assigning the next job to whichever bucket is currently emptier. (This handles the case where you put a large job into B1 and then you put a number of smaller jobs into B2 until they converge.) Then recursively repeat the process, dividing each bucket into 2 more buckets via the same process. Stop when you have allocated jobs into N buckets, equivalent to the available number of workers.
Rebalanced recursive allocation. Follow the process for recursive allocation, but before recursing, also try to rebalance the buckets by swapping an item between queues in such a way as to make the total duration of each buckets closer to each other. Suppose that bucket B1 has duration 150 and bucket B2 has duration 130. We can rebalance them by looking for a job in B1 that has duration (150-130)/2 = 15 and moving it to B2. If we can’t find a job whose duration is exactly 15, we can just use the closest approximation we can find. We would not swap anything if there are no items we can move that improve the balance between buckets.

Intuitively, I expect the recursive algorithms to work better than the round-robin or next-fit approaches.

Measuring the results

As the total duration of all jobs t(total) grows very large compared to the maximum duration of any individual job t(individual_max), we should be able to converge on solutions that approach the optimal (continuous) target t(optimal) mentioned above.

I wrote a ruby script to generate random input sets and test each allocation algorithm against them at various sizes. The results were as follows:

Small scale: 4 workers, 40 jobs
Ran 150 rounds of allocate_from_ends with 40 jobs in 4 workers
      => Duration 3.1ms
      => Mean variance from optimal allocation: 0.912%
      => Max variance from optimal allocation: 4.831%
Ran 150 rounds of allocate_until_full with 40 jobs in 4 workers
      => Duration 3.5ms
      => Mean variance from optimal allocation: 2.237%
      => Max variance from optimal allocation: 7.23%
Ran 150 rounds of recursive_allocation with 40 jobs in 4 workers
      => Duration 4.1ms
      => Mean variance from optimal allocation: 4.999%
      => Max variance from optimal allocation: 9.178%
Ran 150 rounds of optimized_recursive_allocation with 40 jobs in 4 workers
      => Duration 7.8ms
      => Mean variance from optimal allocation: 0.568%
      => Max variance from optimal allocation: 2.807%
===================
Medium scale: 16 workers, 250 jobs
Ran 150 rounds of allocate_from_ends with 250 jobs in 16 workers
      => Duration 17.2ms
      => Mean variance from optimal allocation: 3.921%
      => Max variance from optimal allocation: 11.217%
Ran 150 rounds of allocate_until_full with 250 jobs in 16 workers
      => Duration 17.8ms
      => Mean variance from optimal allocation: 1.274%
      => Max variance from optimal allocation: 4.754%
Ran 150 rounds of recursive_allocation with 250 jobs in 16 workers
      => Duration 23.4ms
      => Mean variance from optimal allocation: 3.223%
      => Max variance from optimal allocation: 5.441%
Ran 150 rounds of optimized_recursive_allocation with 250 jobs in 16 workers
      => Duration 39.3ms
      => Mean variance from optimal allocation: 0.194%
      => Max variance from optimal allocation: 1.078%
===================
Large scale: 32 workers, 20000 jobs
Ran 30 rounds of allocate_from_ends with 20000 jobs in 32 workers
      => Duration 229.9ms
      => Mean variance from optimal allocation: 0.16%
      => Max variance from optimal allocation: 0.175%
Ran 30 rounds of allocate_until_full with 20000 jobs in 32 workers
      => Duration 263.5ms
      => Mean variance from optimal allocation: 0.006%
      => Max variance from optimal allocation: 0.048%
Ran 30 rounds of recursive_allocation with 20000 jobs in 32 workers
      => Duration 342.9ms
      => Mean variance from optimal allocation: 0.039%
      => Max variance from optimal allocation: 0.056%
Ran 30 rounds of optimized_recursive_allocation with 20000 jobs in 32 workers
      => Duration 350.6ms
      => Mean variance from optimal allocation: 0.0%
      => Max variance from optimal allocation: 0.001%

TLDR:

As expected, smaller input sizes correspond overall with higher variance from the optimal outcome.
The worst observed performance in any single experiment was 11.2% deviance from optimal, in the medium scale test of the “allocate from ends” strategy. Mean deviance for that strategy at that scale was lower, at 3.9%.
At large volumes, all these algorithms have well under 1% error. So our expectation that at large scale, we would converge on the optimal (continuous) division case, seemed to be proved true.
The optimized recursive strategy was the most successful across the board. Given a large input size of 20,000 jobs, it had at worst 0.001% deviance from the optimal solution over 30 experimental trials.

Takeaways

I would never try to implement an optimized allocation algorithm at work without having some quiet time to do research first. Kind of disliking that someone made me try to do it as a live-coding exercise.
Bin packing theory seems like an interesting thing to look into further. Probably a good thing to catch up on, given that I never got a CS degree.
In practice, at work, we just delegate this problem (CI job allocation) to a SaaS solution (Knapsack Pro). We don’t try to roll our own solutions here.

Coda

The experimental code was as follows.


############################
# Experiment - a class that tests a given allocation strategy for a given number of rounds,
# using a specified number of buckets and a specified volume of randomly generated tests
# After the experiment is done, it outputs its results to stdout.
############################
class Experiment
  def initialize(strategy, rounds:, buckets:, tests:)
    total_variance = 0
    max_variance = 0
    start = Time.now

    rounds.times do
      test_mean, test_max = run_test(strategy, buckets, sample(tests))
      total_variance += test_mean
      max_variance = test_max if test_max > max_variance
    end
    duration = (Time.now - start)*1000 # ms
    puts "Ran #{rounds} rounds of #{strategy} with #{tests} tests in #{buckets} buckets"
    puts "      => Duration #{duration.round(1)}ms"
    puts "      => Mean variance from optimal allocation: #{(total_variance/rounds).round(3)}%"
    puts "      => Max variance from optimal allocation: #{max_variance.round(3)}%"
  end

  def run_test(strategy, n, test_files)
    sorted_tests = test_files.to_a.sort_by(&:last)

    # optimally, we should have (sum(total_test_time)/n) in each bucket
    total_time = sorted_tests.map(&:last).inject(&:+)
    target = total_time.to_f / n

    buckets = (1..n).map { Bucket.new(target) }
    full_buckets = 0

    puts "Allocating #{test_files.count} tests into #{n} buckets, optimal bucket size: #{target}" if ENV['VERBOSE']

    buckets = Allocator.new.send strategy, sorted_tests, buckets, target

    # Report results
    puts buckets.map(&:status) if ENV['VERBOSE']

    # Return average variance
    mean_variance = buckets.map {|b| b.variance.abs}.sum / buckets.size
    max_variance = buckets.map {|b| b.variance.abs}.max
    puts "  ... mean variance #{mean_variance}, max variance #{max_variance}" if ENV['VERBOSE']
    # Return mean variance
    [mean_variance, max_variance]
  end

  # Generates a random test data set
  def sample(size)
    (1..size).map { |i| ["test-#{i}.rb".to_sym, Integer(rand * 250) + 1]}.to_h
  end
end

############################
# Allocator - a class implementing several possible algorithms for allocating input
#   items to buckets.
# The inputs to these methods are called `sorted_tests` because the original spec was for
#   each item to represent a test file with an integer duration in seconds, and we expect
#   the Experiment class to start out by sorting the inputs before invoking these methods.
############################
class Allocator
  # Allocate sorted tests into buckets, pulling from the ends of the array at each iteration
  def allocate_from_ends(sorted_tests, buckets, target)
    current_bucket = 0
    while sorted_tests.count > 0
      current = buckets[current_bucket % buckets.size]
      current.add sorted_tests.pop
      current.add(sorted_tests.shift) if sorted_tests.size > 0 # we might run out of tests
      current_bucket += 1
    end
    buckets
  end

  # Allocate sorted tests into buckets, removing buckets from rotation once they reach capacity
  def allocate_until_full(sorted_tests, buckets, target)
    current_bucket = 0
    full_buckets = 0
    while sorted_tests.count > 0 && full_buckets < buckets.count
      current = buckets[current_bucket % buckets.size]
      if current.total > target # don't add to already full buckets
        if !current.full
          current.full = true
          full_buckets += 1

          puts " ... just filled up a bucket: #{current.status}" if ENV['VERBOSE']
        end
      else
        current.add sorted_tests.pop
      end
      current_bucket += 1
    end
    buckets
  end

  # Repeatedly divide the list into two as-close-to-equal-halves as we can
  def recursive_allocation(sorted_tests, buckets, target)
    recursion_levels = Math.log(buckets.count, 2)
    raise "Must pass number of buckets that are a power of 2" unless recursion_levels % 1 == 0

    # Don't reuse the buckets we were given, we'll regenerate them in the recursion process
    buckets.clear
    buckets.concat recursive_list_division(sorted_tests, recursion_levels.to_i, target, optimize: false).flatten
  end

  # Same as recursive_allocation, but also make an effort to balance out imbalances between left and right halves
  def optimized_recursive_allocation(sorted_tests, buckets, target)
    recursion_levels = Math.log(buckets.count, 2)
    raise "Must pass number of buckets that are a power of 2" unless recursion_levels % 1 == 0

    # Don't reuse the buckets we were given, we'll regenerate them in the recursion process
    buckets.clear
    buckets.concat recursive_list_division(sorted_tests, recursion_levels.to_i, target, optimize: true).flatten
  end

  # Actually does the work for the recursive strategy
  def recursive_list_division(items, levels_remaining, target, optimize: false)
    left = Bucket.new(target)
    right = Bucket.new(target)

    # Allocate the next item to whichever list is currently emptier
    while items.size > 0
      if left.total <= right.total
        left.add items.pop
      else
        right.add items.pop
      end
    end
    puts "Balanced two lists: #{left.total} vs #{right.total}" if ENV['VERBOSE']

    # Try to balance the two lists if optimization is enabled
    if optimize
      25.times do
        left, right = recursive_list_balance(left, right)
      end
    end

    if levels_remaining > 1
      [ recursive_list_division(left.items, levels_remaining - 1, target, optimize: optimize),
        recursive_list_division(right.items, levels_remaining - 1, target, optimize: optimize)]
    else
      [left, right]
    end
  end

  # Attempts to balance out gaps between two unequal lists
  def recursive_list_balance(left, right)
    # Compute the gap between left and right -- that's what we hope to improve
    difference = (left.total - right.total).abs
    # Don't bother doing anything if the two lists happen to be identical
    return [left, right] if difference == 0

    # Now search for an element in the larger list
    # that is about half the size of the difference between lists:
    larger = left.total > right.total ? left.items : right.items
    candidate = nil
    for i in (0..larger.size - 1) do
      if larger[i][1] > difference / 2
        candidate = i
        break
      end
    end
    return [left, right] if candidate.nil?

    # Either the candidate item or the item preceding it should be the best item to swap
    if candidate > 0 && ((larger[candidate][1] - difference) > (larger[candidate - 1][1] - difference))
      candidate -= 1
    end

    # Let's only do the swap if it actually improves the balance between buckets
    if larger[candidate][1] < difference
      puts "Balancing lists: moving item of size #{larger[candidate][1]} to balance a gap of size #{difference}" if ENV['VERBOSE']
      if left.total > right.total
        right.add left.remove(candidate)
      else
        left.add right.remove(candidate)
      end
    else
      puts "Best candidate was #{larger[candidate][1]} to balance a gap of size #{difference}, skipping" if ENV['VERBOSE']
    end
    [left, right]
  end
end

############################
# Bucket - a utility class that contains a bucket of items and calculates their total size.
#  Initialize it with a target size, and then it can calculate the
#    difference between its actual total and the optimal target result size.
#    Let's call this difference the `variance`.
#  It implements two list operations, `add` and `remove`.
#  It has a flag called `full` that allocation algorithms might use
#    to keep track of buckets that have exceeded their optimal capacity.
############################
class Bucket
  attr_accessor :total
  attr_accessor :full
  attr_accessor :target
  attr_accessor :items

  def initialize(target=0)
    @total = 0
    @items = []
    @target = target
    @full = false
  end

  def add(item)
    @total += item[1]
    @items << item
  end

  def remove(item_index)
    raise "can't remove invalid item with index #{item_index}, total items #{@items.size}" if @items[item_index].nil?
    item = @items.delete_at(item_index)
    @total -= item[1]
    item
  end

  def status
    "#{variance}% variance, #{@items.size} items, #{@total} total, #{Integer(@total - @target)} over target"
  end

  # how far off we are from the target bucket size, in percent
  def variance
    (((@total - @target).to_f / @target) * 100).round(4)
  end
end



############################
# Imperative script code:
############################

# little wrapper function to kick off each set of experiments
def run_experiments(label, rounds:, buckets:, tests:)
  puts "==================="
  puts "#{label}: #{buckets} buckets, #{tests} tests"
  Experiment.new :allocate_from_ends, rounds: rounds, buckets: buckets, tests: tests
  Experiment.new :allocate_until_full, rounds: rounds, buckets: buckets, tests: tests
  Experiment.new :recursive_allocation, rounds: rounds, buckets: buckets, tests: tests
  Experiment.new :optimized_recursive_allocation, rounds: rounds, buckets: buckets, tests: tests
end

run_experiments 'Two buckets', rounds: 150, buckets: 2, tests: 40
run_experiments 'Small scale', rounds: 150, buckets: 4, tests: 40
run_experiments 'Medium scale', rounds: 150, buckets: 16, tests: 250
run_experiments 'Large scale', rounds: 30, buckets: 32, tests: 20000
# The large scale one is run for fewer rounds because it gets slow

Message in a bottle (in the comments)

2025-04-24T20:12:00+00:00

I had one of those trivial chores today — adding a new queue to our Sidekiq worker configuration.

It seems like one of those unthinking, mechanical tasks. Just find the list that defines the queues, add a new item, save, commit, send for code review.

I had already committed the code change, but not yet sent it for review, when I noticed a code comment.

Among other things, I was changing some helm kustomization files that pertain to Sidekiq setup. The file looked something like this (not a real code sample, just to give you the idea):

sidekiq:
  deployment:
    keda:
      enable: true
  deployments:
    default:
      config:
        queues:
          # This list must be kept in sync with the autoscaling config below
          - fast
          - slow
          - medium
      spec:
        maxReplicaCount: 3
        triggers:
          - type: prometheus
            metadata:
              serverAddress: http://<some-prometheus-cluster>:9090
              threshold: "150"
              query: "sidekiq_queue_size{queue=~"(fast|slow|medium)"}

The comment says: This list must be kept in sync with the autoscaling config below.

The idea is, if you change which queues are handled by a pool of Sidekiq workers, you want to edit the autoscaling configuration at the same time. Otherwise, you are going to see weird system behavior when your new queue gets a long backlog and autoscaling doesn’t notice…

Nothing forces you to keep the two parts of the configuration in sync, except your own reading skills.

And I almost missed it. Which would have been embarrassing.

But here’s the funny thing: I wrote that comment.

It was sometime last year, when I last worked on this area of our system.

I’m sure I must have thought to myself: It would be so easy to edit the queue configuration without updating the Keda configuration. I’ll just leave a comment to point out this footgun.

Nine months later — the comment worked perfectly, and saved me from myself.

You can’t rely on anyone reading the comments (not even yourself, apparently). But still.

Isolation vs coupling, or the problem with silos

2025-04-13T10:18:00+00:00

In large technical systems, there’s a pervasive tension between isolation and coupling. What I mean by this is:

There are architectural advantages to having strong walls between things.
However, there are also architectural advantages to having tight coupling between things.

These two things often push in opposite directions, making system design awkward, or at least delicate.

Let’s consider a few obvious examples.

Tenant silos, or horizontal boundaries

Suppose we are designing a multitenanted system. We might need to go in one of two contrary directions:

Put every tenant in its own technical silo, keep everything highly separate. Use separate databases, separate workers, separate infrastructure for each tenant.
or: Build multi-tenanted systems that cross logical boundaries and work with data across silos. Share resources (which might reduce infrastructure costs).

Isolating things into silos can be good for security, for privacy, and possibly for operational flexibility. Maybe we want to allocate more resources for certain tenants. Maybe we want to define totally separate user accounts for every tenant.

But isolation can also be inefficient and operationally frustrating — suppose we want to let all the tenants share certain resources, but now this is impossible.

Or suppose we want to know “across all tenants, how many records of type X do we have?” It sucks when the answer is “sorry, we don’t even have tooling to find that out right now.”

Or consider the costs of releasing updates - do you want deployment to be O(1) or O(N) for N tenants? Do you want to run database migrations 500 times or just once?

Or consider the problem of users with access to multiple tenants — do you want them to have to authenticate one time or N times to get access to N tenants?

Of course, you can always try to hack your way through this problem space, and get some of the benefits of both approaches. You can build approach 1 (premised on isolation) and then add some hacks — ahem, I mean extensions — that get you some of the benefits of approach 2 (premised on coupling). Or vice versa.

Web stack components, or vertical boundaries

Here’s a second example. Suppose you have a web stack that is organized into different layers like this:

External load balancer
NGINX
Ruby application server
Rack middleware stack
Ruby on Rails application

Requests normally start at the first layer and, assuming everything is OK, then get passed down to the last layer. Your application layer then handles a response and sends a response back through all the other layers to the client.

This scenario also has a design space with a tradeoff between isolation and coupling, though not exactly the same kind as the horizontal tenant sharding case.

Essentially: Everything is cleaner and simpler when different layers of this stack are indifferent to each other. Everything is swappable. There are no hard dependencies. You want to change load balancers? Go ahead, it doesn’t affect anything in the application server.

But. When the architectural isolation between layers is too absolute, certain use cases become impossible.

A use case we’ve looked at recently involves performance logging. Our default observability stack runs at the application level. It only starts when the other layers are complete.

This means it has blind spots.

If a request never makes it to the application, it is invisible.
If the application hits a request timeout, the observability tools crash with it, and you don’t get logs of what happened.
If you want to know how long a request took, you can only measure starting when the observability tools kick in. What if you want to measure the time starting at level 1 all the way down to level 5?

To fix these problems, you have to build observability tooling that crosses levels of the web stack.

It’s great - now you have more information.

But you also have more dependencies, more tight coupling, more complex (often ad hoc) contracts, and thus more brittleness.

Suppose we started setting a custom request header at the load balancer, to measure total request time.

Now if you switch to a new load balancer, you have to remember to migrate your custom header, or the custom observability tooling will break. You have a new constraint. You’ve moved away from total isolation towards great coupling between components. At your peril.

Every way you go, there are perils.

Discussion

Needless to say, architectural purism is rarely optimal. You go in one direction and then tack back in the other. You want to satisfy competing constraints to the extent you can.

But the fact that we often end up in some middle zone of this design space doesn’t mean that there aren’t competing pressures in opposite directions. In this sense, large technical systems rarely reach a stable equilibrium in their own design space. Rather, they exhibit a moving set of tradeoffs that can shift suddenly under your feet.

This problem space is closely related to the more well-known problems of “standardization vs nonstandardization (autonomy),” which also provide a major tradeoff space. But if you think closely about it, they are orthogonal.

                 autonomy
                    ^
                    |
isolation  <==           ==> coupling
                    |
                    v
              standardization

All your teams can align on a model of tenant silos - that’s standardizing on isolation. Or some of your teams can build global systems and some can build tenant systems - that’s more autonomous, but it’s orthogonal to whether systems are more or less connected to each other.

Coda: Leaky abstractions

I keep thinking this is related to the problem of leaky abstractions, though I haven’t quite found the words to talk about this. Every time we design for “isolation,” we are always also designing interfaces for coupling across our silos, our layers. These interfaces somehow are never good enough, and then we are tempted to realign the architecture, or at least the interfaces, to plug the gaps in our system.

In any case, I constantly see people building isolated “tenanted silos” and then, a few years later, going back towards tightly coupled “global services,” or vice versa. Like we’re all caught in some giant pendulum of unstable requirements.

The pitfalls of things that seem easy

2025-03-14T13:32:00+00:00

Sometimes you want to do something easy and it does not go well.

It looked like an easy hike: An allegory

Once upon a time, I was on vacation and I wanted to hike over to these hills in the middle distance.

I wasn’t familiar with the desert landscape, but I’ve hiked in a lot of other places. This hike didn’t look that hard.

You can see that there are some canyons that block the way, but I thought they looked small. I thought you could just go around them. I thought they had gentle slopes that wouldn’t be difficult to climb.

So I set out.

And I found out that once you were in the canyons, they became a maze that wasn’t easy to cross.

And I found out that the vegetation was huge and thorny.

And I found out that the “gentle slopes” were an ordeal with many obstacles.

I turned around pretty quickly because I realized I was outmatched by the landscape.

Software is full of this

I think every software developer has had this kind of experience (metaphorically speaking).

There’s somewhere you’d like to get; you can easily see the destination; it doesn’t look that far away.

And you weren’t familiar with the landscape in detail.

And you underestimated the pitfalls that were waiting for you.

And… even if you kept going, it was vastly harder than you thought, and you had to solve puzzles that didn’t seem related to the original goal. Getting untangled from cactus spines, for example. Incidental complexities, or at least seemingly incidental complexities.

One can think here of the Dunning-Kruger effect, which can always get you, even for overall pretty competent people. It’s easy to overestimate one’s capabilities.

But I prefer to think of it as being less about competence than about familiarity with an environment.

It’s just hard to estimate what’s feasible in an unfamiliar environment.

Even in a familiar environment, things that ought to be easy can just be very hard.

This gets me to the original thought that inspired this post:

It’s funny how simple, easy tasks can get blocked by other tasks that are dramatically harder.

You want to clean up some simple taxonomy of environment values, like I did the other day, but first you have to clean up a small mountain of other things you didn’t really want to touch.

There’s an art to knowing when to bother.

On actually reading error messages

2025-03-14T13:01:00+00:00

Not all error messages are beautiful and concise.

Frequently we see error messages in the form of a long stacktrace.

./ruby/3.3.0/net/http.rb:1603:in `initialize': Failed to open TCP connection
to nonexistenthostname.local:80 (getaddrinfo: nodename nor servname provided,
or not known) (Socket::ResolutionError)
    from ./ruby/3.3.0/net/http.rb:1603:in `open'
    from ./ruby/3.3.0/net/http.rb:1603:in `block in connect'
    from ./ruby/3.3.0/timeout.rb:186:in `block in timeout'
    from ./ruby/3.3.0/timeout.rb:193:in `timeout'
    from ./ruby/3.3.0/net/http.rb:1601:in `connect'
    from ./ruby/3.3.0/net/http.rb:1580:in `do_start'
    from ./ruby/3.3.0/net/http.rb:1569:in `start'
    from ./ruby/3.3.0/open-uri.rb:334:in `open_http'
    from ./ruby/3.3.0/open-uri.rb:770:in `buffer_open'
    from ./ruby/3.3.0/open-uri.rb:220:in `block in open_loop'
    from ./ruby/3.3.0/open-uri.rb:218:in `catch'
    from ./ruby/3.3.0/open-uri.rb:218:in `open_loop'
    from ./ruby/3.3.0/open-uri.rb:158:in `open_uri'
    from ./ruby/3.3.0/open-uri.rb:750:in `open'
    from ./ruby/3.3.0/open-uri.rb:29:in `open'
    from (irb):7:in `<main>'
    ... 4 levels...
./ruby/3.3.0/net/http.rb:1603:in `initialize': getaddrinfo: nodename nor
servname provided, or not known (Socket::ResolutionError)
    from ./ruby/3.3.0/net/http.rb:1603:in `open'
    from ./ruby/3.3.0/net/http.rb:1603:in `block in connect'
    from ./ruby/3.3.0/timeout.rb:186:in `block in timeout'
    from ./ruby/3.3.0/timeout.rb:193:in `timeout'
    from ./ruby/3.3.0/net/http.rb:1601:in `connect'
    from ./ruby/3.3.0/net/http.rb:1580:in `do_start'
    from ./ruby/3.3.0/net/http.rb:1569:in `start'
    from ./ruby/3.3.0/open-uri.rb:334:in `open_http'
    from ./ruby/3.3.0/open-uri.rb:770:in `buffer_open'
    from ./ruby/3.3.0/open-uri.rb:220:in `block in open_loop'
    from ./ruby/3.3.0/open-uri.rb:218:in `catch'
    from ./ruby/3.3.0/open-uri.rb:218:in `open_loop'
    from ./ruby/3.3.0/open-uri.rb:158:in `open_uri'
    from ./ruby/3.3.0/open-uri.rb:750:in `open'
    from ./ruby/3.3.0/open-uri.rb:29:in `open'

To be clear, this is just a DNS lookup error.

I sent a test request to a nonexistent hostname and got this stacktrace.

But here’s the thing:

A surprising number of professional technology people will look at this, see that there’s a long stacktrace, and just abandon all hope.

They don’t recognize this long blob of text.

It’s unpleasant looking.

So they decide in advance they probably can’t understand it.

There are, indeed, errors that you won’t understand, even when you read every line of the stacktrace. No one understands everything. Technical systems are complicated. Understanding them is an investment that you can’t always make.

But if you don’t actually read the errors — you might give up prematurely.

No, to be precise, you will give up prematurely.

To be clear, I’m not against anyone asking for help. Groups are often smarter than individuals, or at least much more knowledgeable. It’s not bad to check if someone else already knows the answer.

But it would be nice if people read the errors.

Maybe even before asking.

The invisible layers of the stack, or why whitespace broke authentication

2025-03-13T20:47:00+00:00

Someone messaged me the other day asking for assistance.

Their question was intriguing: “Can you help figure out why integration requests to our API have started failing, with an authentication error, when extra whitespace is added to the incoming JSON payload?”

The mysterious case of whitespace that… breaks API authentication

The breaking requests looked something like this:

{
  "data" : {
    "type" : "object",
    "name" : "roses",
    "color" : "red"
  }
}

Our systems responded with HTTP 403 when given this payload, even though they were properly authenticated.

Meanwhile, the very same requests would succeed if formatted like this:

{
  "data": {
    "type": "object",
    "name": "roses",
    "color": "red"
  }
}

The only difference is the presence of extra whitespace in between the JSON keys and the subsequent colon.

I’m used to seeing JSON object keys formatted with no white space in between "key" and the subsequent colon :, like "key": "value". I double checked the JSON spec in RFC 7159 section 2 and found that both ways are perfectly valid JSON:

Insignificant whitespace is allowed before or after any of the six structural characters.

(The “structural characters” include name-separator = ws %x3A ws ; : colon.)

Where does the problem occur?

I checked to see if we had changed our application’s JSON parsing library. I checked if anything had changed on the sending side. In both cases, nothing had recently changed.

In our configuration, our public API endpoints are normally handled by AWS API Gateway.

API Gateway has a bit of internal complexity but roughly we use it like this:

sequenceDiagram
    participant c as Client
    participant gw as AWS API Gateway
    participant app as Application Server

    c->>gw: Incoming request
    gw->>gw: Checks authentication
    Note right of gw: Could have auth/parsing errors?
    gw->>app: Sends allowed, authenticated requests
    Note right of app: Could have auth/parsing errors?
    app-->>gw: Returns response
    gw-->>c: Returns response

We don’t expect API Gateway to do much more than handle authentication and a bit of URL path routing for us. We expect it to pass all authenticated inbound payloads to our back end servers for processing.

However, strangely, the whitespace problem was only reproducible when requests were sent through AWS API Gateway. If we sent test requests straight to our application servers, bypassing the API Gateway, then the whitespace issue vanished.

This helps to localize the problem.

What does API Gateway really do?

I checked whether API Gateway does any kind of processing on the request payloads. I was curious if it attempted to validate JSON syntax and if it somehow considered the extra whitespace a syntax error.

No, and no.

And even if it did validate request payloads, one would not expect it to emit an auth error (HTTP 403 Forbidden, in particular).

So I started looking for the API Gateway logs to see what it thought was happening.

It’s the invisible layers of the stack

The API Gateway access logs promptly revealed the problem: the broken requests were all being blocked by AWS WAF. That’s the web application firewall, a service that inspects incoming requests and tries to block malicious traffic.

The API Gateway logs themselves didn’t reveal why these requests were blocked by the WAF, but this was already enough to work with. We chatted with the infrastructure person who maintains WAF configuration, and found that some new filtering rules had recently been deployed. These rules had erroneously blocked some legit traffic since deployment. The QA team initially discovered the problem in our staging environment, and escalated to engineering.

Around this point, I realized my high-level mental model of the system was incomplete.

An expanded model would look like this:

sequenceDiagram
    participant c as Client
    participant gw as AWS API Gateway
    participant waf as AWS WAF
    participant app as Application Server

    c->>gw: Incoming request
    gw->>gw: Checks authentication
    Note right of gw: Auth/parsing errors?
    gw->>waf: Checks requests against WAF rule set
    Note right of waf: What happens here?
    waf-->>gw: Responds blocked or allowed
    gw->>app: Sends allowed, authenticated requests
    Note right of app: Auth/parsing errors?
    app-->>gw: Returns response
    gw-->>c: Returns response

In other words, WAF constitutes another major failure point in the request flow, one that I hadn’t given much thought to. It’s always the invisible layers in the stack that get you. There’s just so much happening in complex technical systems, and it’s easy to neglect the layers that usually work silently, without problems.

But why would WAF block JSON because of extra whitespace

I don’t have the whole story here as I don’t have access to WAF rule definitions. But I learned that the request was being blocked by a rule set that tried to block known Windows shell attacks. I presume something about "type" : triggered some regex.

Our back end servers don’t run Windows, but these rules were enabled anyway, causing the issue.

As AWS docs explain, WAF will respond with HTTP 403 when a request is blocked. Unfortunately this status is also sometimes emitted by our downstream application servers, so it doesn’t help us localize the issue. But at least now I know to look for this case if we see 403s.

We’ll probably deploy better reporting to try to prevent this scenario in the future. We could have caught it faster if we had alerts on the right things in our logs.

It’s still very unintuitive that anyone could ever get an auth error from adding extra whitespace in a JSON payload. But things make more sense when you remember that every request is screened against a mostly-opaque set of security rulesets that probably don’t even consider the inbound request format.

The thing people like about AWS WAF is that it is supposed to mitigate security risks without much work on the application developer’s part. The problem is, this requires that it should work invisibly like a black box without ever failing. Then if it doesn’t work properly - you still end up having to partly understand it anyway.

At the very least — now we remember it’s there, and we know that this failure mode is possible.

A rat’s nest of configuration management

2025-02-20T11:49:00+00:00

I decided to clean up our environment type taxonomy a little bit at work. This involved a little journey into configuration management.

Configuration strategies

Obviously, there isn’t just one way to configure your software. A few patterns I’ve noticed lately:

Hardcoded constants in the codebase
Conditional logic in the codebase that dynamically generates configuration values
YAML configuration files with global scope
YAML configuration files scoped by environment type
Direct access to environment variables from application code
Dynamic configuration loading from external configuration services.

At a high level, we can distinguish between static configuration and dynamic configuration. (This might be more of a spectrum than a binary distinction; for example, dynamic configuration at initialization time is logically distinct from dynamic configuration that can change at runtime.)

I think it’s pretty reasonable to use static configuration methods like code constants for values you are certain will never change dynamically. But beware: if you write a code constant that says TIMEOUT_SECONDS = 3 and one day the production system urgently needs TIMEOUT_SECONDS = 10, you will wish you had provided a dynamic configuration hook for this value. At the same time, too much configurability is also bad: it inevitably leads to inconsistency across runtime contexts that you don’t want.

Well, anyway, here I’m just interested in configuration that’s based on environment type, because what I wanted to do was clean up our environment types.

Rails configuration

Our applications tend to have complicated settings files organized by RAILS_ENV, a minimalist environment taxonomy provided by Rails which implicitly provides a set of allowed environment types (development, test, staging, production, and anything else you want).

According to the default “Rails way,” you can configure your application by setting environment-specific settings in an environment-specific configuration file located at config/environments/<ENVIRONMENT_NAME>.rb.

In our case, we have a large number of business logic flags and settings that don’t fit neatly into that approach. We handle this case using Rails’ config_for, which provides support for nicely loading a YAML file organized (by default) by RAILS_ENV. The YAML file in question can also evaluate ERB templates, meaning that you can put arbitrary Ruby logic into it.

In practice, mainly we use ERB to evaluate environment variables. Naturally, it ends up with a tangled mess, something like this:

# settings.yml
shared:
  allow_user_registration: true
  send_mail_from_domain: example.com
  cleanup_old_records: <%= ENV.fetch("CLEANUP_OLD_RECORDS", "false") == "true") %>
development:
  allow_user_registration: false
  cleanup_old_records: <%= ENV.fetch("CLEANUP_OLD_RECORDS", "false") == "true") %>
  user_rate_limit: 25
qa:
  <<: *staging
  allow_user_registration: <%= ENV.fetch("ALLOW_USER_REGISTRATION", "true") == "true") %>
  cleanup_old_records: <%= ENV["CLEANUP_POLICY"] == "strict"
    && ENV.fetch("CLEANUP_OLD_RECORDS", "false") == "true") %>
  user_rate_limit: <%= ENV["USER_RATE_LIMIT"] || 5 %>
staging: &staging
  allow_user_registration: <%= ENV.fetch("ALLOW_USER_REGISTRATION", "true") == "true") %>
  cleanup_old_records: <%= ENV.fetch("CLEANUP_OLD_RECORDS", "true") == "true") %>
  user_rate_limit: <%= ENV["SECONDARY_USER_RATE_LIMIT"] || 10 %>

Now suppose you have multiple Rails services built at different times using different configuration structures. And you would like to clean them up and make them consistent. Imagine that one of your services uses RAILS_ENV="qa" in the QA environment, but all the others use RAILS_ENV="staging".

This can be cleaned up parsimoniously by removing RAILS_ENV=qa from the one service that is not like the others.

This, however, means that you have to read all the configuration currently under qa and move it into the staging configuration context. A project which, though I’m no Hercules, reminds me vaguely of cleaning the Augean stables.

Fixing inconsistencies one by one

Let’s consider the cases here one by one.

allow_user_registration

Both qa and staging are set to the same environment variable ALLOW_USER_REGISTRATION with the same default. In this case, it should be fine to switch to staging without doing anything.

But it still took a minute to read both lines of code, compare them, and make sure that they’re identical. “Manual toil,” we call this at work. (This specific case could be handled by an LLM, I think, but you would still have to verify the results, which would take as long as doing it manually, because errors are impermissible.)

cleanup_old_records

The qa config checks two different conditions based on two different env vars (CLEANUP_POLICY, CLEANUP_OLD_RECORDS). Meanwhile, staging just checks one env var, CLEANUP_OLD_RECORDS, with a different default from qa.

What you have to do in this case to consolidate them is to discover what the use case is here, check what the current env vars are actually set to in the qa environment, and then, if possible, consolidate on only using CLEANUP_OLD_RECORDS, so that you can fall back to the staging config safely. You might have to alter the env vars themselves to make this work (we usually set env vars in code, using Helm charts, so this is fine).

This is becoming a research project, because you have to understand the use case and check env vars from a different system. I’m not convinced an LLM can do this yet, because it requires digging around elsewhere in a large technical system.

user_rate_limit

Both qa and staging are set to the same environment variable ALLOW_USER_REGISTRATION, but with different defaults. In this case you have to ensure that the ALLOW_USER_REGISTRATION env var is populated in qa and has the correct value (you can set it to 10 if it is not already set).

This isn’t really complicated but, again, manual toil.

Obviously these aren’t real examples, but imagine doing this project again and again across a large set of configuration points, and you can imagine the experience.

Virtual patch panels

It occurred to me that what we were building in settings.yml was the virtual equivalent of a patch panel.

The function of a patch panel is to allow you to route arbitrary outputs to arbitrary inputs. It is a sort of abstraction layer that allows for flexibility and configurability, while (hopefully) keeping the mess contained to a single zone.

In this case, our virtual patch panel takes inputs from the env vars or from a set of predefined values and then routes them into a settings table exposed to application developers. And that can be more or less of a huge mess. It takes discipline to keep it from becoming a huge mess.

We’d like to keep this fundamentally messy and arbitrary zone as clean and understandable as possible.

Best practices

I started thinking about things to not do in a configuration layer.

Don’t duplicate configuration code (see allow_user_registration).
Prefer flat configuration systems; try not to build inheritance trees of configuration when possible.
Prefer consistency across contexts. If you need a similar configuration system in two separate services, make it as identical as you can. Standardization reduces cognitive load on maintainers and, so, reduces the risk of configuration errors.
Don’t allow direct environment variable access from the codebase (as that is skipping what the “patch panel” is for). Keep all env var access centralized in one site, so it is instantly apparent to maintainers what env vars are required by the program. If this is impossible, at least minimize the number of places that access env vars.
Don’t write to the env vars as if they were a convenient set of mutable global variables. (Exceptions may be allowable in test code, maybe, sometimes. Ugh.)

Coda

Recently we’ve started switching to a feature flag SaaS solution which costs a lot and nobody really loves. It has heavy process overhead from non-engineering staff, a confusing UX, limited licenses because of having high costs per seat, and generally all the modern luxuries we’ve come to expect.

It is fine for its specific usecase (runtime evaluation of boolean flags) and it’s good for managing release-specific flags, but it’s not a complete replacement for settings.yml. So now, of course, we heavily use both systems.

#winning?

Coordinating Unicorn worker processes with flock

2025-01-31T16:31:00+00:00

Some of our current software runs on Unicorn, which if you aren’t the target audience for this post, is a process-based Ruby webserver that has:

a very old-school website,
a GitHub mirror,
and an architecture based on a master process that forks a number of child worker processes to handle requests.

We got interested lately in having exactly one of a set of Unicorn workers spawn a background thread that would report some periodic healthcheck data. The idea was that the healthcheck results would be identical for all workers, so we only needed to report the data once per Unicorn master process. But we didn’t want to run a reporting thread on a master process, as it isn’t encouraged to fork a multithreaded process. (See for example Thorsten Ball’s Why Threads Can’t Fork, rachelbythebay’s Don’t mix threads and forks, or more recently byroot’s Why does everyone hate fork?).

As the fork(2) manpage explains, when you fork, all threads except the active thread will die and not get resumed in the child process:

- The child process is created with a single thread—the one that
called fork().  The entire virtual address space of the parent
is replicated in the child, including the states of mutexes,
condition variables, and other pthreads objects; the use of
pthread_atfork(3) may be helpful for dealing with problems
that this can cause.

- After a fork() in a multithreaded program, the child can
safely call only async-signal-safe functions (see
signal-safety(7)) until such time as it calls execve(2).

Stopping all background threads when forking might be what you want, in some cases, but it can leave resources dangling, connections unreleased, and so on, depending what was happening in the other threads at the time.

Anyway - if we don’t want to report healthcheck data from the master process, and we want to report it from only one of n worker processes, then this raises an interesting interprocess coordination problem.

How can you guarantee that out of a a pool of n workers, exactly one will run a given observability task at any given time? And how can you guarantee that if one worker dies, another will automatically start running the observability task?

It kind of reminds me of Zookeeper - a cluster coordination problem - except that in this case, we aren’t trying to coordinate processes across a whole cluster; we are only trying to coordinate processes within a particular container.

Naive approach

The first thing that occurred to me was this:

At boot time, each child process will check for the existence of a file at a standard path (let’s say /tmp/coordination.pid).
If /tmp/coordination.pid is not found, then create it, and write the current pid to it. Whichever process does this first is volunteering to run the healthcheck task.
If /tmp/coordination.pid is already present, then check if a process with that pid is running.
- If so, then sleep for a while and then check again.
- If not, then proceed from step 2 as if the file were not found.

Problems with this approach:

There is some chance of a race condition in between steps 1 and 2, wherein two processes simultaneously find that /tmp/coordination.pid is absent and then each try to write their pids to the same path. The chances of this could perhaps be mitigated by waiting for a random interval before attempting step 1.
For the numerous workers that are sleeping, it’s inefficient that they have to wake up every so often to recheck step 3. This imposes a pointless polling cost.

My colleague Dmytro suggested that we use flock instead, which essentially delegates the whole coordination problem to the operating system and solves both of these problems.

I had never heard of it before.

Flock(2)

I found flock hard to learn about. There are manpages (flock(2)) and Hacker News discussions, but they don’t cover the set of use cases for file locking very clearly. I think the core use case is “several processes want to write to the same shared file and need to cooperate with each other.”

In any case, it is a system call that comes with some caveats. The first two I found:

The relationship between file descriptors and file locks is slightly confusing to me in the context of forks.
Per this discussion, flock is handled poorly over NFS, although I don’t think that edge case is very relevant to our kubernetes cluster. Fortunately, the edge case pertains to how flock is handled differently for processes running on the NFS server itself than for NFS clients, so even if our ops team started to run NFS for our web workers without telling me, the edge case would not affect us.

In any event, flock can nicely be used to coordinate only-once semantics among a set of worker processes. The way it works for our use case is this:

Open a file at a standard path.
Attempt to acquire an exclusive lock (LOCK_EX) on the file. Use the blocking form of flock that just blocks the caller until the lock can be acquired.
If you acquire the lock, then you can go ahead and run your instrumentation task, or whatever only-once activity you want to conduct.
When the process that is currently holding the lock eventually exits or is killed, the operating system will automatically wake up the next process in line and give them the lock. You never have to poll, in this approach.

Ruby implementation

Ruby provides a standard (though platform-dependent) interface to flock, available at File#lock.

One can write an implementation roughly like this in a Unicorn configuration file:

TMP_FILE_PATH = "/tmp/coordination.pid"

after_fork do |server, worker|
  Thread.new do
    File.open(TMP_FILE_PATH, File::RDWR | File::CREAT, 0644) do |f|
      f.flock(File::LOCK_EX) # will block indefinitely if the lock is not acquired

      # now run whatever background task you want here, such as reporting system health.
    end
  end
end

So far, this has worked quite well for us, and it seems likely to be much more robust than any DIY solution I could have come up with.

How to instrument DNS lookups in Ruby

2024-12-19T06:29:00+00:00

One of the things we did at work, after our strange performance issue in September, was to add some instrumentation to DNS lookups in Ruby.

As I mentioned last time, it isn’t easy to get performance data directly from getaddrinfo in Linux, which is the libc function that does DNS lookups. You would have to have a local DNS proxy service running and do the logging from there, which isn’t the recommended configuration for our Kubernetes-based system, as far as I understand.

Indirectly, this also means that you can’t easily instrument DNS lookups from C extensions in Ruby, if they call getaddrinfo directly. It probably is not worthwhile to fork every library you use and add your own custom instrumentation code.

However, some libraries with C extensions still use Ruby for the DNS lookups, as Ruby 3.1 provided helpful concurrency handling for the address lookup using Fibers. Fortunately, this happens to include our database adapter, pg).

This means that we can add helpful instrumentation directly in Ruby. It’s easy to measure all DNS lookups that use Addrinfo.getaddrinfo.

Sampling methodology

Methodologically speaking, if you are trying to instrument your DNS lookup calls, it doesn’t necessarily matter whether you capture every request. Even if you can manage to do that, you might not want to, as instrumentation has a runtime (and resource) cost. Particularly if you have a high volume of DNS lookups, you might want to use a random sampling approach, and only instrument X percent of DNS requests.

The approach I came up with was this:

Instrument a small percentage of all DNS lookups (like 2%).
Also, instrument all DNS lookups that are slower than a certain threshold (say 100ms, if you are expecting fast local DNS resolution).

If you record those two data streams separately, then you can get a good sense of aggregate performance, plus good coverage of the worst case scenario.

Implementation

Monkey patching the Ruby standard library is something to approach cautiously. It has some risk of breaking shared functionality and of adding extra request latency (if the instrumentation layer takes any time).

require "socket"

module DNSInstrumentation
  SAMPLING_RATE = 0.01
  SLOW_QUERY_REPORTING_THRESHOLD = 0.15

  def getaddrinfo(...)
    start_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)

    result = super

    duration = Process.clock_gettime(Process::CLOCK_MONOTONIC) - start_at

    if duration > SLOW_QUERY_REPORTING_THRESHOLD || Kernel.rand < SAMPLING_RATE
      instrument(:dns_query, duration)
    end

    result
  end
end

# Apply the patch

# At boot time:
Addrinfo.singleton_class.prepend(DNSInstrumentation) if dns_instrumentation_enabled?

(Side note: It drives me slightly crazy that we have to resolve our database server’s IP address for every request, since for tenant partitioning reasons we have to establish a new database connection on every request and pg will look up the database server’s IP from its hostname every single time… There are ways around this, like pinning tenants to specific web workers or adding application-level DNS caching, but we aren’t currently looking into them.)

Suppose your instrument method just prints to stdout:

def instrument(type, duration)
  puts "DNS query took: #{duration}"
end

Then you can see results like this:

irb(main):008> Addrinfo.getaddrinfo("www.decasia.org", 80)
DNS query took: 0.048221999779343605
=>
[#<Addrinfo: [2606:4700:3031::6815:51e0]:80 UDP (www.decasia.org)>,
 #<Addrinfo: [2606:4700:3031::6815:51e0]:80 TCP (www.decasia.org)>,
 #<Addrinfo: [2606:4700:3037::ac43:a564]:80 UDP (www.decasia.org)>,
 #<Addrinfo: [2606:4700:3037::ac43:a564]:80 TCP (www.decasia.org)>,
 #<Addrinfo: 172.67.165.100:80 UDP (www.decasia.org)>,
 #<Addrinfo: 172.67.165.100:80 TCP (www.decasia.org)>,
 #<Addrinfo: 104.21.81.224:80 UDP (www.decasia.org)>,
 #<Addrinfo: 104.21.81.224:80 TCP (www.decasia.org)>]

(48ms is slow, but the second time I look up the same hostname, it takes only 1.7ms, presumably because MacOS — unlike Linux — does run a local DNS cache by default.)

Commentary

There’s nothing very technically remarkable about this instrumentation strategy, except that I didn’t realize we could do this until we did it.

Like a lot of things in software - you aren’t sure in advance if something is possible, and then when you need to know it, you just have to figure it out at the time.

On-demand knowledge, one could call it.

True BASIC/4d/Star Pattern/01

2024-12-04T20:39:00+00:00

I went looking for the source code from my first ever C program (a screen saver). I couldn’t find it, but instead I found the homework exercises from my first ever computer programming class in high school. They are all written in a BASIC flavor called True BASIC.

This class was taught by an aging math teacher who liked playing bridge and Go with his students after school - he was kind of a classic late-20th-century US nerd figure, with glasses and maybe, like, a pocket protector. I guess he’s probably not still around anymore, which is sad to realize. He always liked me and encouraged me to keep learning about computers. I didn’t really understand that I was lucky to get that kind of encouragement.

I remember this class being way too easy for me at the time. The most complicated thing in it was a quicksort implementation.

A more typical exercise was this:

REM Program: True BASIC/4d/Star Pattern/01         5/5
REM Purpose: To print a pattern of stars:
REM                *
REM                **
REM                ***
REM                ****
REM              for 40 rows and then go in reverse:
REM                40 *s
REM                39 *s
REM Author: Eli Thorkelson
REM Date: 14 Mar 1997
REM These are the opening statements
PRINT "                        StarPattern 1.0"
PRINT "                       By Eli Thorkelson"
PRINT ""
PRINT ""
PRINT "                    Press any key when ready"
GET KEY useless                   !This variable is useless
REM Beginning of the first FOR-NEXT loop
REM           This loop prints the first half of the *s
FOR step1 = 1 to 40
    PRINT ""                      ! This moves to the next line
    FOR step2 = 1 to step1
        PRINT "*";
    NEXT step2
NEXT step1
REM Beginning of the second FOR-NEXT loop
REM           This loop prints the second half of the *s
FOR step1 = 40 to 1 step -1
    PRINT ""                      ! Moves to the next line
    FOR step2 = 1 to step1
        PRINT "*";
    NEXT step2
NEXT step1

I haven’t looked at this language in such a long time. I like the nested loop implementations. It’s both clumsy and oddly easy to read.

My favorite line, clearly, is GET KEY useless !This variable is useless, although it should really have read !This comment is useless since the variable name is already self-documenting. Perhaps unused would have been a better name than useless, though.

I tried to figure out how to execute this program. I managed to get some of it to run on qbjs.org. I had to shorten it to ten lines instead of 40, and remove the confirmation step.

                        StarPattern 1.0
                       By Eli Thorkelson


*
**
***
****
*****
******
*******
********
*********
**********
**********
*********
********
*******
******
*****
****
***
**
*

I honestly have no idea why this was even a homework exercise. I guess it assesses whether we could write nested loops. Or maybe our teacher just liked seeing stars.

DIY unit tests

2024-11-20T19:34:00+00:00

Earlier this year, I got rid of my virtual private server and migrated the content to a static hosting system. In the process, I broke an old web project that had a submission form.

The submission form used to be handled by a small Go application that accepted form input, validated it, persisted it to a SQLite database, and then wrote new JSON data for the public website. Obviously, that broke when I got rid of the server where the application was hosted.

I decided to rewrite the form handler as an AWS Lambda function that ingested JSON and updated some S3 content. I hadn’t ever written a Lambda function before, and I decided I should try to deploy it with Terraform, which I also haven’t touched, so there was a slightly painful learning curve. It took an evening to get it all working again.

You can write Lambda functions in Ruby, Golang, Java, and Python, among others, but I decided to write mine in Node, since it seems like the quintessential “serverless” language.

I decided I had to have unit tests for my lambda function. But as far as I know, Node doesn’t come with a built-in test framework.* People generally use third party test frameworks for Node apps.

And the problem is, I didn’t really feel like installing a new dependency just to write some unit tests for one single lambda function. Dependencies are debts. Better to avoid them if you don’t really need them.

So I wrote my own tiny test framework.

A DIY test framework

It is structured as follows.

It does this at the top of the file:

import {testables} from './index.mjs'; // A map of functions to be tested

const testSuite = []; // A list of tests - a "suite" if you will

const test = (functionUnderTest, implementation) => {
  testSuite.push(
    {
      name: functionUnderTest, // the name of the function you want to test
      f: testables[functionUnderTest],
      implementation: implementation // test implementation
    }
  );
};

It then provides you two simple assertions.


const assertEq = (value, expect, eq = (a, b) => { return a == b }) => {
  if (!eq(value, expect)) {
    console.error(`  ❌ ${inspect(value)} != ${inspect(expect)}`);
    process.exit(1);
  } else {
    console.info(`  ✅ ${value} == ${expect}`);
  }
}

const assertJsonEq = (value, expect) => {
  return assertEq(
    value,
    expect,
    (a, b) => { return JSON.stringify(a) == JSON.stringify(b) }
  );
}

These will exit(1) and print an error whenever the test fails. I didn’t need anything more than this.

Finally, at the bottom of the file, there’s a test runner.

const main = async () => {
  for (let t of testSuite) {
    console.info(`Testing ${t.name}...`);
    await t.implementation(t.f);
  }

  console.info('\n🦄 Test suite successful 🦄');
};

main();

If you invoke this with node test.mjs, it runs all the tests in order, exiting with an error as soon as any test fails.

An example test declaration looks like this:

test('beep', (f) => {
  assertEq(f(1), "beep");
  assertEq(f(2), "boop");
});

This just asserts that f(1) equals "beep", and f(2) equals "boop".

The test implementation accepts one argument f, which will be set to the function called beep from the original module. We retrieve the beep() function from the testables module export that we imported at the top. (This all presumes you are using ES6 modules or whatever they’re called now.)

Discussion

I was pretty happy with this amount of test tooling for the tiny personal project I was using. For my purpose, it did an excellent job - in particular, it got the lambda function ready for deployment with relatively high confidence that it would work. And it did deploy with minimal issues (it had a few IAM issues, and one function signature issue that the unit tests missed).

I’m not saying this test framework is good enough for a large project. It always fails fast, which isn’t necessarily what you want. It doesn’t provide multiple output formats, flexible test naming, complex matchers, test randomization, timing data, mocks and stubs, memoized test data setup, or much else.

However, by sacrificing a lot of what you usually get from a test framework, it also gains a few things:

It has a very clean, readable DSL (in my opinion).
It has zero dependencies.
It is very quick to run.

But most importantly, it reminds me that you don’t always need a framework. The concepts here are so straightforward that for a simple use case, you can write your own assertions and your own runner, and it basically just works.

Note 1: Node does come with an assertion framework in the standard library, even if not a full test framework. I would probably use their assertions next time, instead of the DIY assertions.

Note 2: In terms of dependencies, I did end up adding one AWS mocking library, aws-sdk-client-mock, to be able to test functions that used the AWS SDK. But this isn’t essential to the basic DIY test framework here.

Reading programming languages you don't understand

2024-11-10T07:45:00+00:00

Sometimes you want to read code you can’t understand. Sometimes you want to read languages you’ve never learned.

That seems like it should be impossible. No one can speak every language. There are limits to understanding.

The analogy with human languages isn’t encouraging. If you try to understand a human language you’ve never learned, it’s often utterly unintelligible. And acquiring a new human language can be a very slow process, often taking years.

I’m probably not going to spend years learning a new programming language just to read a random codebase. Nevertheless, there are lots of technical systems I want to understand at work, and even when I don’t have much other context, I can usually find their git repositories. So I open up their source code to see what I find, even if it’s not in a programming language I know.

I’ve tried this out recently with Elixir, Kotlin, and Scala. And what surprises me is that, contrary to my intuitions that this should not work, I often learn something from reading.

Example: Where does this configuration value come from?

Here’s a real life example.

We’d like to update a configuration value in some other system. I’d like know how the configuration value is set in the first place, so we can update what needs updating.

I would probably start by trying to find the calling site where the configuration is used. A controller class in a web service, perhaps. From there, I can look for the origins of whatever value I’m trying to trace. Then I can follow that object (variable, parameter, function, etc) back through the codebase to see if I can arrive at a point of origin.

Last time I tried this exercise, I did find the right answer, even though the codebase was in Scala. (Then I checked my answer with an expert to confirm.)

In a case like this, we know up front that there are only so many possible sources of initial state for a program: configuration files, external systems, constants, user input, and so on. Even if we don’t know the language, we have a sense of the problem space.

Concretely, how do you read unknown languages?

This isn’t a very satisfactory answer, but I think you just kind of… do your best.

Scan source code
Attempt to parse, liberally using your knowledge of other languages, ignoring unfamiliar symbols or syntax
Trace execution flow via function names and variable name flow.

This is a problematic reading strategy, and we know that up front. But understanding is a gradient, after all. You can get farther along, or you can get stuck quickly. And depending on the case, there can be greater or lesser degrees of interpretive uncertainty. You understand, but with an asterisk; you acknowledge where you’re guessing. “I understand this bit *unless the | operator does something totally unexpected.” That kind of thing.

A lot of programming languages have similar syntax and semantics. And the program structure can be apparent, from naming and file organizations, even if you don’t understand exactly how the language works.

(I suppose at this point in history, it would probably work well to ask an LLM to explain a codebase to me. I suppose I will try it. I doubt it will be honest with me about the parts it isn’t sure about, though.)

Why not just ask for help?

There’s nothing wrong with asking someone for help, needless to say. Often asking is necessary; often it’s much faster than any alternative.

However, there are reasons not to always ask first.

Asking can be slow. Implicitly, it can cost you something, in some annoying social-capital sense. It feels uncharitable to interrupt someone else to get answers you could trivially find yourself.

So I ask questions all the time, but I don’t usually ask before trying to self-solve. If the answer becomes clear from my own research, that’s great. If not, I can ask for help with a little more context than I had at first.

This gets us to the higher order reasons to try to do your own research first. Reading other teams’ codebases can help to build relationships with them. If I understand something about their systems, I might be able to have better conversations, starting from a point of greater understanding. “I looked at ABC, and I think your system handles X by doing Y; is that right?” — I think people usually respond better questions like this, as they give a clearer point of departure.

And down the road, it usually comes in handy to know something about other parts of the technical ecosystem I work in.

The illusion of not understanding

Sometimes understanding is portrayed as more binary than it really is. “Either you understand something or you don’t.” But the truth is, partial understanding is often all we have.

This is exacerbated in the case of language use, I think. There’s so much shaming around our understanding of languages. It’s risky to say you understand something, and then miss something that an expert would catch, and then get put down for it.

It’s exacerbated in software, too, where the economy likes to put us in boxes. You have to list your programming languages on your CV and it’s presumed you can’t use anything else.

Meanwhile, everything I’ve learned from reading unknown programming languages points away from these dogmas.

Understanding is a gradient, and there are ways of making uncertainty work for us, rather than undermining us.

Golang, flow state, and Copilot

2024-10-29T19:49:00+00:00

Lately I started to get that excellent sense of flow state while working on my Golang project.

The flow state of programming is such a nice feeling, when it happens. I don’t get to experience it as much as I used to, because so much of my work currently consists of meetings, plans, Slack messages, and things that don’t involve long periods of focus.

And when I first started writing Go for this project, I found it a little tedious. Go has a deliberately limited vocabulary and a lot of boilerplate error handling.

But after a few solid days, I started to really enjoy writing Go. It’s not as elegant as Swift (the other strongly typed language I’ve played with lately). But it doesn’t have the horrible type inference slowdowns either. It’s more like writing C, but with a much better standard library, fewer semicolons, and a better DX.

What’s good about writing C, and also about writing Go, is this: It’s all functions and structs. The building blocks are very uniform. If you can write a good function, with a sensible name, and compose it nicely with other functions, using sensible data structures, then the program almost starts to write itself. It has a kind of minimalist harmony.

The flow is mostly this: a flow of writing functions, specifying data structures, extracting shared functionality, arranging a project into sensible packages, and testing.

And quite often, if my Go program can compile, then the behavior is fundamentally correct out of the box. I do write tests too, and I test interactively. But it really improves flow state and productivity to know that, if my code compiles, then there’s an excellent chance it’s already in working order.

I started to ignore the boilerplate, too. I used to be annoyed by all the if err != nil blocks, but lately I just … don’t pay them much attention.

Notes on using Copilot

It’s the first time I’ve ever used Copilot with a new programming language.

I’m more excited currently about the code autocompletion than about the Copilot chat. The chat has lackluster performance and a bad UX for applying changes. The autocompletion is quite good and saves me a bunch of typing. It tends to be fast at noticing common patterns and wanting to apply them. It’s sometimes incorrect, of course, but I always re-read and edit the output as soon as I hit tab.

Sometimes I use the Copilot chat to basically substitute for things I would Google. “How do you unmashal a blob of YAML data into a slice of structs?” The result from Copilot is similar in quality to the result from StackOverflow. (Copilot is clearly less reliable than the excellent gobyexample.com, however.)

Sometimes it improves flow state to use Copilot, because you don’t have to leave your text editor.

Sometimes Copilot breaks flow state, though, by giving slow or unreliable answers. I’m torn about that part.

Things that break flow state

I did find a few strange things:

Failed type inferences

The Go tooling in VSCode is mostly great, but there’s an edge case where the Go language server can fail to infer return types correctly. In the case I stumbled into, my code would still compile and run correctly, but in VSCode, some files were wrongly marked as broken, as having invalid types in their function signatures.

Circular dependencies

You can’t have circular package dependencies, which makes sense. But what if package A depends on package B and C, and then package C wants to use a struct that’s defined in package B?

I understand it’s idiomatic to use interfaces in such circumstances, and then the interface definition can just be imported from some base package that has no dependencies. This avoids any circularities.

I was, however, surprised that Go’s interfaces don’t cover fields; they only cover methods. That feels a bit redundant if all you need is to define a few accessors.

This brings me to a related issue with interfaces.

Inconsistent calling semantics for interfaces

Suppose you have an interface for a Point and you have a struct that implements it:

type Point struct {
      X int
      Y int
}

func (p Point) s() string {
      return fmt.Sprintf("%d/%d", p.X, p.Y)
}

type PointInterface interface {
      s() string
}

This works:

func printPoint(p PointInterface) {
      fmt.Println(p.s())
}

func main() {
      p := Point{5, 2}
      printPoint(p)
}

This does not work:

func printList(list []PointInterface) {
      for i, _ := range list {
            fmt.Println(list[i].s())
      }
}

func main() {
      plist := []Point{{10, 20}, {20, 30}}

      printList(plist)
}

The error reads: cannot use plist (variable of type []Point) as []PointInterface value in argument to printList.

Basically, you pass a Point into a function that expects a PointInterface. But you can’t pass []Point to a function that expects a []PointInterface. Instead, you have to do a conversion step to manually transform a slice of structs into a slice of interfaces.

Per Stack Overflow, the explanation seems to be this: It’s an O(N) operation to convert a slice of structs into a slice of interfaces that the struct implements. And Go doesn’t want to do that automatically. But converting just one at a time is O(1) so the compiler will handle it silently.

I found this unintuitive.

Not my favorite ever package manager

There’s nothing seriously wrong with go mod, but I do not love it, either. I was confused by the way Go handles multiple modules inside a single project. I don’t really need multiple modules, as dividing my project into packages is plenty of structure for my use case; but I found it less than luminous.

A lot better than C: I’ll say that much.

In sum

It’s such a nice experience to seriously sit down with a new language at work. I wouldn’t use Go for everything. And the boilerplate is slightly exhausting. But it clearly has a sweet spot, and it seems very maintainable once you write it.

These are good things.

Mental models of the command line

2024-10-07T08:45:00+00:00

When I first encountered the Linux command line, I was pretty bad at it.

I thought of it as a purely imperative interface.

You run xyz <enter> and you see some output. Then you can run xyz again if you want, or a different command if you prefer.

It’s just the “push a button, get a result” model of execution. It’s not wrong — every process does return an exit code indicating success or failure — but it’s a bit limiting.

The problem with the imperative model is: you can only push buttons. If you want to do something for which there’s no button, you’re stuck. It’s not a very extensible approach.

I’ve learned a few things since then, but I still feel like I’m bad at the command line.

Map and filter

Eventually I learned python and discovered map and filter operators.

Sometime afterwards, it started to sink in that the command line is also a playground for functional transformations of an input. (Conventionally, the inputs are line delimited plaintext or maybe CSV/TSV, but with tools like jq it’s easier to transform other structured formats too. I’m constantly reaching for curl | jq.)

I find it helpful to think of something like grep as a tool that implements filter(), and something like cut as something that implements map(). That way, you can see the general patterns beneath the arcane Unix tool names. For example, you can grep "WARN" log.txt | cut -d ':' -f2 to effectively filter and then map the bits of a log line that you want to read.

I’ll mention xargs (or find -exec) as essential functional programming tools too. Often you want to do something with the results of your functional pipelines.

Process stacks and trees

The pipe | that we use for data transformations above is obviously a way of chaining processes together in linear execution order. It’s a bit like .then() in Javascript promises. But you can also think of the shell as a stack of processes that you can navigate up and down.

I remember learning really early on about & to run something in the background. I can’t say that I use & much anymore. Remember when it was common to have an account on some big Unix server? The kind where you could leave some processes running in the background, log off, and come back later? Those days are gone. Now it’s all ephemeral containers and replaceable VMs.

Anyway, the process stack got more interactive when I found out about using ^Z to background an interactive process, and then fg to restore it. For example:

$ bin/rails console
>     # do some rails console commands
^Z    # leaves the rails console running in the background

$     # back at the regular shell to run something there...
$ fg
>     # back to the rails console again...

Meanwhile, it’s also sometimes helpful to picture the larger process space as an irregular tree, one whose branches are weakly linked together by dictionaries of environment variables that propagate from parent to child processes.

Shell vocabulary

Those are a couple of conceptual approaches to the shell environment. There are functional pipelines; there are process stacks; there are process trees.

But it’s not just about concepts either, right? It’s about learning the “vocabulary” or, dare I say, the “culture” of the shell environment. You can’t get too far at the command line without finding out about shell configuration and aliases, or why it’s annoying to use sh for your shell, or how $PATH works, or how Unix groups and file modes work. It’s hugely useful to know how redirection operators < and > work. And certain tools like vim or curl are in constant use – it doesn’t always matter if there is an inner logic to them; you just have to get used to them to find your way around. (I actually like vim.)

The shell’s culture is kind of vast, and I always feel like I’m still learning more about it.

Things I’m still bad at

Never got good at AWK.
Never really learned tmux. (For most use cases, split panes in a terminal emulator come close enough, or maybe splits in vim.)
I can kind of get by with Bash scripting. I have to constantly check the user’s guide: How do you use a hash in Bash, again? How do you remember all the arcane test operators?
Never learned perl or emacs (but I don’t really miss them).

It’s funny to be writing a command line tool and still feel like a beginner, in some ways.

After a while, you’re not really a beginner anymore. But you’re experienced enough to start understanding your own limits, and so you still feel … modest.

Command line tool design

2024-10-01T08:01:00+00:00

I promised another team at work I would rewrite their shell script as a command line dev tool in Golang.

The old UX of the script was suboptimal. It ran a slow interactive shell script that made you go through an interactive menu tree every time, even when you already knew what you wanted to do. It had to be run inside Docker to handle a python dependency, making the initial startup even more tedious.

It got me thinking: What makes a good command line tool?

To be sure, we already have CLI tool frameworks (like Cobra) that tell us some common best practices in this area:

Standard Linux-style argument parsing
Short and long option support
Subcommand support (like git blame, kubectl exec).

(I’ll use Cobra for extensibility, even though the Go standard lib already has a basic flag package.)

A few more things I thought of:

Multiple usage modes: It can let you specify the exact command you want, or it can show your options if you don’t know. You can type cook rice --color=brown or if you just type cook, it will ask if you want to make rice, barley, orzo, or couscous.
Explain and/or dry run mode: It should be able to tell you what it will do without doing it. It should be able to explain its own activity in case of doubts.
Verbose output: It should give you the choice between seeing the implementation details or just seeing the TDLR of what happened.
Pretty output, using colors and/or emojis nicely to convey meaning. (It’s 2024 - our terminals support Unicode too.)
Diagnostics. It should be able to check its own configuration and self-describe.
Solid help pages. tool --help should be comprehensive. tool subcommand --help should work too.
Credential integrations. If you have to enter creds for any operations, it should make this as easy as possible. (For instance, using password manager integrations when possible.)
Progress bars. When we are doing slow operations, display progress nicely.
Plays well with other tools. It should provide either colorized or noncolorized output; it should provide human and machine readable output; it should fit organically into the shell environment.
Minimal dependencies.

A few things I don’t like in command line tools:

Inadequate --help output.
Non-GNU option parsing style (see: JVM tools like keytool; see also: openssl).
Pedantic, when that adds no special value.
Inflexible or difficult to configure.
Poorly organized codebase (for tools where you can consult the codebase).
Difficult to install.
Difficult to upgrade.

I can’t promise anyone that I can accomplish 100% of this, but I think it’s a good start at a wishlist.

Fun Fact

Ironically, the `go` compiler tool does something I find really unhelpful:


$ go mod --help      [7:12:27]
go mod --help: unknown command
Run 'go help mod' for usage.

Come on, couldn’t you just translate god mod --help into go help mod for me?

Pointers in Ruby

2024-09-26T12:11:00+00:00

No one: It’s so sad that there are no pointers in Ruby.

Me: Did you know that you can trivially implement pointers, though?

class Obj < Struct.new(:name, :shortcode, keyword_init: true)
  def ~
    Pointer.new(object_id: object_id)
  end
end

class Pointer < Struct.new(:object_id, keyword_init: true)
  def +@
    ObjectSpace._id2ref(object_id)
  end
end

You can then reference and dereference using a relatively simple semantics, using ~x to create a Pointer from an Obj, and +y to dereference a Pointer.

Here, ~x (tilde) is similar to C’s &x, “get the address of x”.

Meanwhile +y is similar to C’s *y, “dereference the pointer stored at y”.

Usage

irb(main):012> o=Obj.new(name: "A")
=> #<struct Obj name="A", shortcode=nil>
irb(main):013> p = ~o
=> #<struct Pointer object_id=29020>
irb(main):014> +p
=> #<struct Obj name="A", shortcode=nil>

What… is this

Well, they aren’t pointers in the C sense, exactly. But you can happily use the Ruby VM object IDs for the same concept.

You just have to use .object_id to fetch the Object ID, and ObjectSpace._id2ref to dereference. And overload some unary operators to provide … some kind of semantics.

I wish I could implement this with * and &, for visual similarity to the C operators, but Ruby doesn’t support unary * or &.

Why???

People say you can’t do pass by reference in Ruby; you have to do “pass by object reference” or “pass reference by value”.

Discussion:

Well, now you can do pass by reference in your Ruby method calls too, if you really want to :)

def welcome(obj_pointer)
  puts "Hello, #{(+obj_pointer).name}"
end

o=Obj.new(name: "Ambassador Spock")

welcome(~o)
=> "Hello, Ambassador Spock"

Disclaimer

Please don’t try C-style pointer arithmetic with these pointers.

Not ready for production usage yet.

Always document flows

2024-09-25T20:29:00+00:00

When you are adding something new to a large technical system, always document flows.

What is a flow?

By a “flow,” I mean a flow of information, events, user actions or business logic from point A to point B.

You start somewhere. You end somewhere. You might stop at a lot of places along the way.

Suppose you are implementing flows that cross lots of different systems or components.

Maybe each piece of the puzzle is straightforward, or even self-documenting. It’s great when you write a system component that’s easily understandable to future maintainers.

But if you have five or fifteen components, and they all have to talk to each other in a certain way, you need to understand the overall flow too.

If you don’t know what the flow is, it takes a lot of archaeology work to figure it out again.

That’s why you should document your flows — before you start to forget how they work.

Multi system flows

Here’s an imaginary flow that involves asking a document store for a document.

Behind the scenes, the document store is asking an auth service if this request is duly authenticated and authorized.

In turn, the auth service is asking a directory service for information about the requester. (Let’s suppose that in this imaginary system, our authorization policies are based on group memberships in a directory service.)

sequenceDiagram
    Requester->>Document Store: Requests a document
    Document Store->>Auth Service: Checks authentication
    Auth Service->>Directory Service: Retrieves the requester's group memberships
    Auth Service->>Document Store: Indicates if the request is valid
    Document Store->>Requester: Returns a document or else 401 Unauthorized

If I gave you the source code for all these services, even if it was the most readable source code in the world, it might take you some work to figure out the order of operations of this flow.

But if I gave you the source code plus this picture, you would instantly see what the overall story was, and it would be much easier to put the pieces together.

But I’m working on a system that has no flow diagrams

It sounds like I’m talking about some utopian world where people usually document things well.

In reality, we’re often working on existing systems that don’t have great documentation. They were built in a hurry or heavily modified since the initial design.

Now you have to add something new, or fix some bug, and you don’t know what the larger flow is.

What you should do is this: Go ahead and draw a flow diagram. Even if you are documenting something that has existed for years and has never been properly documented before. The best way to understand something that already exists is to write the docs that should exist for it.

By writing docs for existing things, I promise you will end up understanding them better than you thought you could.

(I’m not the first person to say this. I saw a team at work where every new engineer has to write docs for something as part of onboarding. I’m in awe of that.)

But what should my diagram look like?

I suspect this really isn’t too important, as long as there is something that captures the system in one view.

I personally am a very cartographic person and I like pictures, graphics, and technical diagrams. I love digital whiteboards like Excalidraw. The diagrams in this post are from Mermaid.

If it matters, I tend to prefer flow charts over sequence diagrams. Here’s a flow chart that covers the same things as the sequence diagram above:

flowchart TD
    Requester-->|Requests a document|DS
    DS(Document Store)
    AS(Auth Service)
    Dir(Directory Service)
    DS-->|Checks auth|AS
    AS-->|Checks requester groups|Dir
    DS-->|Returns doc or 401|Requester

But if you aren’t into the pictures, you can document a flow in words too, or with ascii art, or whatever. It would be OK to document a flow with a quick paragraph of “TLDR: Here’s how it all works.”

Here’s the sequence diagram without Mermaid:

Requester -> Document Store: Requests a document
Document Store -> Auth Service: Checks authentication
Auth Service -> Directory Service: Retrieves the requester's group memberships
Auth Service -> Document Store: Indicates if the request is valid
Document Store -> Requester: Returns a document or else 401 Unauthorized

Still pretty readable, right?

I really think it’s better not to bikeshed over documentation too much. That indirectly discourages people from doing documentation, in the long run, because often people aren’t in the mood for bikeshedding.

Don’t document the wrong things.

I’m not going to comment too much on technical documentation in general, but let me just say that it’s pointless to document things that are already obvious to a competent reader.

The codebase can be self documenting up to a point.

Let it do that. If your codebase is the source of truth for your input validations, your database schemas, your runtime configuration, or your class hierarchy, then let it express that naturally. Don’t write separate documentation for things that are sufficiently self-documenting.

Don’t add new flows when you don’t have to

Flows are often cognitively expensive.

This has a practical implication. If you have the choice between using an existing flow or adding a brand new one to a system, everything else being equal, you should prefer the former. Prefer fewer flows to more flows, given the choice.

This can significantly reduce cognitive load for future maintainers.

In conclusion

Document your flows.

Your future self, or your future colleagues, will appreciate it.

A confusing issue to debug, or the mysterious case of 5000ms server delays

2024-09-05T21:17:00+00:00

We had an interesting case for performance analysis this week at work. Our staging server started to display a strange pattern of intermittent slow responses.

It was an unusual response pattern because it wasn’t localized to certain endpoints, nor was it a general system slowdown. Overall system resources like CPU usage and memory were all fine across all the different components of our system. However, slowness was scattered randomly across almost all our service endpoints.

The problem manifested as a very particular 5-second delay, intermittently added to the server response time. So instead of having the usual response time graph — where most of the endpoints are fast and then there is a long, slightly random tail of slow responses — in this case we were looking at our regular response times, mixed together with bursts of requests that took about 5 seconds.

The response time logs looked about like this:

GET / 201ms
GET / 130ms
GET / 86ms
GET / 160ms
GET / 5102ms
GET / 5073ms
GET / 95ms
GET / 188ms

I’m just making up these exact numbers, but you can get the general pattern.

The affected endpoints included everything except the service health check endpoint. Upon investigation, the health check endpoint turns out to use a different base class from all the other requests. That was a useful finding, because it allowed us to narrow down the problem roughly to “functionality included in our standard controller base class.”

Early investigation steps

We checked out everything we could think of and couldn’t find a root cause. Not even a good clue. Lots of dead ends.

We found a flood of erroneous healthcheck requests from a load balancer going to the wrong endpoint. Devops team fixed it.
We read the codebase for our controller base class and tried to find anything that connected to external services (database IO, redis IO, etc). Found nothing useful.
We checked the transaction traces from our standard observability tooling, which automatically runs profiles against slow requests and sends them for analysis. In every case, our observability tooling did not pinpoint the problem. Clearly, our observability tooling had some gaps.
We scanned through the application logs and the error handling reports. Found nothing.
I grepped the codebase and infrastructure config for anything with a 5 second timeout. Found a few possibilities, but none of them panned out.
I checked the logs and found the date when the problem started. It was the same date as a new code deployment, so I checked what changes had been deployed in that release cycle. Unfortunately, there were ~1k commits in the deployment, so it wasn’t easy to read through it all. I checked for changes to the controller base class and to the overall service configuration. Found more dead ends.
Our DBA checked for suspicious or slow database queries. Found some more dead ends. Reset the postgres stats collector to get more data. Still found nothing.

So.

What do you do when you’ve checked all the usual observability tools and found nothing useful?

Automated reproduction of the bug

After a day of getting nowhere, I wrote a quick concurrent load test that attempted to reproduce the problem.

It looked roughly like this:

#!/usr/bin/env ruby

require "net/http"

# Invoke with ruby test.rb "https://path.to.server/endpoint" 20 10
# where 20 is the number of threads and 10 is the number of requests per thread

url = URI(ARGV[0])
num_threads = ARGV[1].to_i
requests_per_thread = ARGV[2].to_i

results = []
mutex = Mutex.new

def send_request(url)
  start = Time.now
  print "."
  code = Net::HTTP.get_response(url).code

  { start_time: start, status: code, duration: Time.now - start }
end

threads = num_threads.times.map do |n|
  Thread.new do
    requests_per_thread.times.each do
      result = send_request(url)
      mutex.synchronize { results << result }
    end
  end
end

threads.each(&:join)

puts "\nSent #{num_threads * requests_per_thread} requests to #{url}"
puts "  #{num_threads} threads x #{requests_per_thread} requests per thread"

results.sort_by { _1[:start_time] }.each do |r|
  puts "%s: %.2f ms - HTTP %d" % [
    r[:start_time].strftime("%H:%M:%S.%L"),
    r[:duration] * 1000,
    r[:status]
  ]
end

(Disclaimer - not meant to be production ready code, just a quick script to find a problem.)

The script instantly reproduced the problem, and gave us some quantitative metrics about its extent, which was super useful.

I ran it a bunch of times against different endpoints, with varying thread counts and requests per thead. I think the smallest useful test set was 50-75 requests (5 threads), and the largest batch was 4000 requests (200 threads).

The two initial findings from the script were:

The more concurrent requests we got, the more often the 5 second slowdown started happening.
The health check endpoint never had the problem.

Later on, as we tried different things, I also used the load test script to check “Did this change make things better or worse? Is the problem still happening?” We tried a few infrastructure changes, like increasing the number of web workers and changing some karpenter node scaling configuration. They didn’t help, or at least not much.

Profiling your code

Unfortunately, being able to reproduce the problem still wasn’t enough to pinpoint the root cause. And the staging system’s users were eager for answers.

I tried to reproduce the problem in a few other dev/test environments. I could not reproduce it on any environment besides staging.

Eventually, I started to want a more detailed profile of what gets executed in our controller base class (ApplicationController in Rails terminology). I set up some profilers locally (not on staging) and ran it against our server (in particular, I ran ruby-prof and stackprof). The local profiler data helped me get a sense of what normal operation looks like, but, of course, it didn’t find the root cause of the staging issue.

It started to seem like the only thing that would help was to run the profiler on the staging system. We consider our staging system to be production-like, so we don’t normally have those kinds of tools available. We don’t usually add debugging output to our staging deployments, either.

I could have gone ahead with that route. I would probably have added StackProf::Middleware to the Rack stack, redeployed, re-run the load test, and dug through the output.

But instead I found something better.

Profiling your code in prod

I remembered that Julia Evans had worked on a Ruby profiler a few years ago. This led me to the fantastic tool she built, rbspy.

What’s great about rbspy is that you don’t have to change anything about your application itself. You just run rbspy, tell it the PID of your chosen process, and let it profile your service for a while. You re-run your load test to reproduce the problem while the profiler is running. Then you inspect the results.

The first time I tried to run rbspy, it crashed the web worker container by running out of memory.

But we increased system resources and I tried again.

All bugs are shallow with the right data

The first rbspy profiler run spotted the problem instantly. (It outputs results into a huge SVG flamechart that you can click around.)

The random slowness came from getaddrinfo, the libc function that does DNS lookups. The calling site was inside our postgres database connection routine, which is in a library that isn’t covered by our standard instrumentation tooling.

Some of the time, looking up the database hostname was taking ~5 seconds. Our application does a lot of per-request database connection switching (because reasons), so the problem could have affected almost any endpoint. Except the healthcheck.

I told this to my colleague in devops and he remembered that they recently changed the hosting setup for our local dns service (coredns). They rolled back the change and the 5 second delays went away instantly.

Lessons learned

All observability tooling has gaps. It’s good to fix them, when possible.
We don’t have good observability on how long DNS lookups are taking. Unfortunately, it seems to be nontrivial to instrument them on the DNS client side, because the DNS requests get sent out from getaddrinfo (I think) and there are no standard logs of getaddrinfo request performance. (Devops checked the coredns server metrics and they never showed any problems; but client-side metrics would have revealed the issue instantly.)
It’s good to know the difference between “the whole system is under heavy load,” “a certain endpoint is slow,” and “a rare resource contention issue is affecting performance intermittently.” In this case - it became clearer and clearer that it had to be the last of these possibilities.
For obscure issues, an automated reproduction script is great, both to help pinpoint the problem and to quickly check if your latest fix is working.
Sometimes there’s no substitute for profiling in prod. (By prod, here, I mean staging. But the point is the same — you can’t always analyze problems in the development environment; you might have to analyze them where they occur.)
Being able to profile a running server with rbspy, without any code changes, is amazing.
We still don’t know exactly why dns lookup from coredns was intermittently slow in the first place. We’ll figure it out, though.

Identity management in big places

2024-08-18T12:09:00+00:00

A thing I’ve noticed about working in a large organization is that you often have to make access requests.

The access requests are something like: “I have a business reason to be able to use system X. Please grant me access.”

Usually the end result of this request is being added to a group. Sometimes it involves being given a role in some third party system, but most often it just gets you added to a new LDAP/Active Directory group. (Sometimes the LDAP/AD group is set up to imply that you should get a certain role in some third party system.)

I started thinking: Shouldn’t the organization know which tools I need, given the role I am in?

A lot of the time, they don’t.

Groups vs roles

Let’s distinguish groups from roles.

A group is something that you happen to belong to.
A role describes your local identity (as my dictionary puts it, a role is “the function assumed or part played in a particular situation.”)

Groups are always multiple, and you belong to them contingently. At a given time, you can belong to group A, B and C, and not to D or E, without any special contradiction. (Think of groups in the Unix sense: no systemic logic governs their assignment; you can be in n different groups at once.)

Meanwhile, roles are often mutually exclusive. You have Role A or Role B, but not both simultaneously. You are an administrator or an end user. You are playing the role of Romeo or of Juliet, but not both simultaneously.

(Admittedly, there can be roles that are not exclusive, but I would argue that mutual exclusivity often does go along with the notion of a role. You are an individual contributor or you are an engineering manager; if you try to be both, it doesn’t usually go well.)

It’s usually easy, in a big organization, to know what someone’s role is. At least their main role that goes on their human resources records.

Why is this not the primary key for one’s system access needs?

Seeing like a state

In the abstract, it makes sense to allocate permissions by role. We know what role you are in, so we should be able to give you access accordingly. (By role in this sense, I mean “your primary job role.”)

Somehow, in a big enough organization, this doesn’t work well at all.

I’ve never actually read James C. Scott’s Seeing Like a State. But from what I understand, it’s all about how top down visibility never quite works out. Big organizations systematically can’t quite see everything that’s happening on the ground across all the different territories they govern. When top-down visibility is imposed, this erases local diversity, making things awkward or disastrous, and leaves things out.

From the perspective of centralized IT or HR systems, I don’t think anyone knows the exact tools that hundreds or thousands of different employees need to use. It’s too difficult to infer from someone’s role what they will need to access.

Instead, we adjudicate this question using the access request.

Access requests, or decentralized access control

An access request: “I need access to system X because Y.” The Y provides individual context, which is lacking in the top-down role-based model.

Access requests are evaluated case by case. I’m a developer who needs access to production logs to analyze an error. I’m a manager who needs access to usage statistics to see how our product rollout is going. I’m an on-call engineer who needs access to some infrastructure settings to fix a problem. I want to get to review the source code of some other team whose source code I can’t see.

All these kinds of things end up in an access request. The access request is the court of appeals for all the exceptions to the rules, for all the unforeseen needs. It changes the unit of analysis from roles to group memberships.

In a way, it’s great that it works like this. It allows individuals to figure out what they need, and the organization to accommodate. It makes big organizations more flexible, more decentralized.

But it’s also a symptom of the opacity of big organizations. The people who run centralized IT systems don’t know what you need. Why do two developers in the same team need different access permissions? There isn’t always a good philosophical answer to that. Everything is context.

What then?

As you spend time in this kind of system, your access requests pile up. You can access more things. You belong to more LDAP groups.

There usually isn’t any clear lifecycle for an access request. Once you have it, you have it. You usually aren’t asked, “Do you still need this?” You might end up retaining access to things you don’t still need. When you change roles, you might still keep some of your old permissions, because permissions weren’t tied to your role in the first place.

From a security policy perspective, it’s bad to have group memberships that don’t necessarily expire. That adds unneeded risk.

All the access requests also take time, which you don’t notice, because it’s spread out over months or years.

But you start to notice the inefficiency of this when someone new shows up and wants to use the tools you have. “Oh, you have the same role as me? But if you want access to the tools I use, you have to make 17 separate access requests, and we don’t even remember how to submit all the access requests…”

What’s the moral of this?

Access control is an area where we have to compromise between top-down visibility and local autonomy. (It rarely works to create top-down access control systems that are too rigid, because if people can’t do their jobs because they can’t get access to the right systems, they tend to look for workarounds.)

When I started writing this, I imagined that permissions ideally should be top-down, and purely role based.

Now that I’m at the end, I feel like I understand more clearly why we need autonomy and case-by-case evaluations.

The perils of IPv6, or, why I got rid of my virtual private server

2024-01-21T09:20:00+00:00

I mentioned last year that I “downsized” my web projects from DigitalOcean to super cheap EC2. Naturally, a few months later AWS announced that they would start charging a few dollars per month for a dedicated IPv4 address.

I asked around on Hacker News and decided that the simplest solution was to put the site on Cloudflare’s free proxy plan, and then update the web server to only use IPv6, thus saving on the IPv4 costs. The way it works is this:

You import your DNS records into Cloudflare.
Cloudflare inspects your A and/or AAAA records to figure out how to find your back end service.
Cloudflare then broadcasts its own A and AAAA proxy addresses when asked to resolve your host.

The key point is that you can have an upstream service that is only accessible via IPv6, and Cloudflare helpfully still proxies traffic to it through its set of IPv4 addresses, maintaining compatibility for legacy IPv4 clients.

That all sounded pretty straightforward, but I still learned a few things from my upgrade experience:

I’m bad at network configuration.
Cloudflare is kind of opaque.
AWS makes it really hard to support IPv4 to IPv6 transitions for running EC2 instances.
In the end, it’s not really worth it.

How to assign an IPv6 to your EC2 instance

There is a field in the EC2 instance properties that indicates your IPv6. Mine was blank.

I hadn’t really understood that if you create an EC2 instance, it comes with an entire virtual private networking setup. There’s something so easy about old-school IPv4 networking. Your server gets a public IP address, it can reach the whole public internet with zero complexity, you advertise your IP address in a DNS A record, and you’re done.

IPv6 isn’t like that, because huge numbers of clients still just don’t support IPv6. And the AWS upgrade path from IPv4 to IPv6 is thorny.

The upgrade path to enabling IPv6 is:

Assign a /64 prefix to your VPC (virtual private cloud). I guess an IPv6 address range isn’t allocated by default.
Assign a /64 prefix to your subnet.
Then you can enable IPv6 address assignment from your EC2 instance’s network interface settings.

In general, I got the impression from AWS’s own blog posts that they only support “dual stack” networking as a sort of awkward transitional path.

https://aws.amazon.com/blogs/networking-and-content-delivery/dual-stack-ipv6-architectures-for-aws-and-hybrid-networks/

https://superuser.com/questions/1801957/how-to-turn-on-ipv6-on-amazon-ec2-instance

After you finish getting an IPv6 address

You have to reconfigure all your services that listen on a network port to listen on the IPv6 interface. By default, NGINX and sshd will only listen on IPv4.

If you try removing the IPv4 address from your virtual server, and you didn’t figure out that sshd needs reconfiguring, then you can’t ssh to your instance anymore. (It’s not difficult to log in with the remote web console and fix it. It’s just annoying. It’s death by a thousand cuts, and by death I mean nuisance.)

Anyway, I eventually got my webserver reconfigured to use only IPv6. If I were doing it again — I would just make a new instance that was IPv6 by default, and delete the old one.

But that wasn’t the bad part.

The bad part was: No matter what I did, I could not figure out how to delete all the IPv4 virtual private networking components from AWS. For something like 4 months, I kept paying the useless IPv4 fee, because I just could not figure out how to avoid it.

I probably should have just asked my work colleagues in devops what to do. But it really seemed like it should not be that hard.

In the end

I started writing this post last January and only got around to publishing it in August. Oops.

There’s a point of diminishing returns for personal projects. Eventually you can run out of spare energy for them.

I deleted my EC2 instance and moved all the websites to static S3 buckets.

There was only one dynamic web application left on my server. It was only for a niche digital art project, so I just left it broken. It didn’t seem worth migrating it to use a serverless architecture (AWS Lambda).

My AWS bills went down to like 25 cents per month. So in the end - it’s like a 20x cost savings over the $6 Digital Ocean box I used to have.

The downside is that I don’t have a virtual Linux box anymore. But at the same time, it’s a relief not to have to do maintenance or system updates anymore.

Software and softness

2023-09-16T07:45:00+00:00

It’s funny that we build “software” and yet, so much of the time, our technical communities do not particularly value softness.

How do we explain this?

What makes software soft?

What makes software “soft”? It’s hard to find an adequate account of this because language evolves gradually, following obscure histories. But the history of the word is somewhat illuminating.

The term software is defined in opposition to hardware. “Hardware” is an old word that historically has not had too much philosophical baggage, as far as I can tell: in the sense of “small metal items,” the word entered the English language in the early modern period, when it meant “ware (such as fittings, cutlery, tools, utensils, or parts of machines) made of metal” (per Merriam-Webster). The term dates as far back as 1419, while the related term “hardware store” entered circulation by 1789.

The term software, by contrast, is a decidedly 20th century invention. Some say it first appeared in print in 1958 in an article by statistician John W. Tukey. Tukey wrote:

Today the “software” comprising the carefully planned interpretive routines, compilers, and other aspects of automative programming are at least as important to the modern electronic calculator as its “hardware” of tubes, transistors, wires, tapes and the like. (src)

You can see the scare quotes suggesting that at the time, this terminology was a neologism, not standard usage.

It’s unlikely that Tukey had coined the word. The engineer Paul Niquette, in an odd online autobiography (archive), recounts having discovered the term “software” as early as 1953, while working on the early computer SWAC. He says he had the following “epiphany”:

I was thinking to myself that I wanted nothing to do with the SWAC “hardware” – that the machine was the mindless means for executing my programs – a necessary evil, mostly evil. It was about at that moment, I seized upon the consummate reality of what I was doing – that what I was doing was sharply different from what [the hardware maintainer] Dr. Whitcomb was doing – that what I was doing was writing on a coding sheet, not plugging jacks into sockets, not clipping leads onto terminal posts, not soldering wires, not bending relay contacts, not replacing vacuum tubes. What I was doing was writing on a coding sheet! The exclamation point was right there in my thought back then and in my memory now.

It was October 1953 and I was experiencing an epiphany. Before my eyes, I saw my own markings carefully scrawled inside printed blocks on the coding sheet. They comprised numerical “words” – the only vocabulary the computer could understand. My coded words were not anything like those other things – those machine things, those “hardware” things. I could write down numerical words – right or wrong – and after they were were punched into cards and fed into the reader, the SWAC would be commanded to perform my mandated operations in exactly the sequence I had written them – right or wrong.

The written codes – my written codes – had absolute power over Dr. Whitcomb’s “hardware.” Then too, I could erase what I had written down and write down something different, then punch a new card and insert it into the deck. The SWAC, slavishly obedient in its hardware ways, would then be commanded to do my work differently – to do different work entirely, in fact. The writing on the coding sheet was changeable; it was decidedly not hardware. It was – well, it was “soft-ware.”

This terminology was new in the 1950s, but it emerged from a longstanding current in intellectual history. The software/hardware divide is arguably just a new permutation of a long-standing dualism in European philosophy, according to which there is a radical difference between mind and matter, the ideal and the physical.

You don’t have to read too much Plato to see how strongly-rooted this kind of dualist view can be. And it usually comes with a preference for one side over the other: the ideal gets valorized, the merely physical gets put down. That’s what Niquette was doing when he declared that software had “absolute power” over the hardware, which was “slavishly obedient” to the software’s “commands”. Mind over matter.

At the same time, in an interesting wrinkle, software was imagined as metaphorically soft because it was more “changeable” than the hardware. It’s easy to think new thoughts and write new code, while it’s hard to wire up new circuits: that’s the argument.

Does it make any sense, though?

Two kinds of “hardness”

Hardness means several quite different (orthogonal) things, two of which tend to get mashed together in talk about software vs hardware.

Hard as in metal.
Hard as in difficult.

But these are not the same. At all.

Physical hardness is something you test in an engineering lab, for example by applying a known amount of force to a surface and measuring the resulting indentation. Metals tend to be hard in this purely physical sense (though they aren’t always).

Difficulty meanwhile has nothing to do with physical properties alone, and has everything to do with the relationship between an acting subject and a practical task. What’s easy to one person can be difficult to someone else; difficulty is relative to your capacities to solve a given problem in the world.

Dumb example: swimming can be very hard, experientially, even though nothing about the water is physically hard.

This suggests to me that it doesn’t make much sense to assume that “software,” by virtue of being a set of computer procedures and not a bag of bolts and wires, has any intrinsic softness in the sense of malleability, ease of change.

Software isn’t easy to change, actually

I would go farther: it’s just false that software is easier to change than hardware. Niquette argued that “The writing on the coding sheet was changeable; it was decidedly not hardware.” But all this shows is that changing the code was easy for him. Who’s to say what was easier for the hardware specialist Dr. Whitcomb across the room?

In truth, even for software professionals, “software” often isn’t very easy to change. On the contrary. Software problems can be absolutely intractable. Programs are complex systems that can seem to resist your efforts to alter them. All working software developers have had the experience of finding that a simple change is impossible to implement in the time available.

Software can be hard.

The people who talk about how easy it is to change the software… tend to be people who just happen to be good at software and not especially good at hardware. I’m not great at hardware - I’ve never built a computer from scratch (though that would be fun). I can splice a wire and build a thing or two with an Arduino and that’s about it. Hardware is hard for me.

But this is saying more about my own particular expertise than anything else.

Meanwhile, I’ve met electricians who can wire all kinds of amazing hardware from scratch, and would never be able to write a computer program. Software would be super hard for them. Nothing “easy to modify” about it.

The kind of hardness that software people like

As it turns out, software people seem to really like hard problems. There’s even a prestige associated with working on clever algorithms, fancy architecture, large-scale systems and loads. There’s a corresponding disdain for “easy” problems: CRUD apps, small-scale projects, following existing patterns.

Certain kinds of difficulty, of course, are preferred over others. Logical and computational difficulty are valued; organizational, political or emotional difficulty, often much less so. In this, we’re still a field that’s captive to the nerdy technocratic values of 1960s engineering culture.

I just started reading Emily Chang’s Brotopia. It explains in detail how this nerdy culture was partly invented by psychologists, to the great detriment of women in tech. I’ll have to write more about that sometime.

In any case, there’s something more than a bit gendered about what’s valued in software culture and what’s not. For example, we don’t give as much respect to “soft skills” as we could, and these have a historical association with femininity. Skills such as reading the room, relationship-building, empathy, caretaking, and sociability are not highly valued by the nerdy side of programming culture.

It’s sometimes assumed that these interpersonal skills are also “soft” in the sense of being easy and nontechnical. They can be cast as something that programmers can disdain. They probably shouldn’t be called “soft” in the first place; we can distinguish math skills from social skills without alleging that one is more technical, harder and more prestigious than the other. Skills are just skills; there is no clear hierarchy of them. And it seems to me that all the “soft skills” are good for everybody, of whatever gender, and are not necessarily easy to acquire either.

There may be other kinds of softness to think about as well: I’m not sure how to think about this systematically. Maybe we should not be so quick to use softness and hardness as metaphors for things in the world in the first place.

But I would also say that there is nothing shameful about softness. To the extent that it is used to describe valuable things in the world, I wish it were more highly valued in the software field.

How to write your own Jira client and suffer slightly less

2023-08-22T07:59:00+00:00

So, Jira.

I do not love it.

Preliminary concessions

Admittedly, Jira does some things well.

It’s a good system of record. We don’t delete anything from it, so you can check project histories from years ago. It tends to keep records of things we didn’t get around to doing, so if we forgot to do X, Jira might remind us later. It’s OK at managing distributed organizational processes like multi team approvals, or requests to other teams.

Above all, it provides a kind of visibility for management into the engineering process. We use it for reporting high level project status to our chain of command, and for release management. It’s good at tracking “In what release will XYZ get released to customers?”

Why I don’t love Jira

Above all, Jira has a really bad user interface, compared to something like Asana. I remember what good UIs feel like. They feel enjoyable, quick to navigate, discoverable, intuitive. Jira is configurable and extensible and highly integrated with other systems, but in my opinion, it has absolutely awful UX and UI. This means that a big chunk of my workday involves bad UX and bad UI.

And above all, it is horrendously slow. It takes forever (in our environment) to create a new Jira (I should time it; sometimes it seems like it’s 20+ seconds). It’s long enough to get bored and want to change focus to something else. Making a new ticket once is bad; making 20 Jira tickets at once is an exceptionally tedious activity, of a kind that software developers should not have to suffer through.

Finally, Jira is bad at some of the things you would expect it to be really good at. For example, Jira is really bad at managing tasks and todos. It’s bad at managing workload. It’s even bad at managing projects.

Jira is bad at tasks

It’s too slow and tedious to put every single thing you need to do into Jira. As a result, most people don’t do that. Some people use Microsoft To Do instead, which is single-user task tracking software that actually works fairly well. It has a clean interface, and it’s a desktop app with desktop app-quality performance.

A lot of people use todos in Google Docs or in Confluence. It’s a lot more lightweight – you can tag a @username and a checkbox will just auto-appear.

Honestly, sometimes I just make todos using MacOS’s Stickies or a plain text doc. I also love Slack’s reminders for lightweight todos.

It’s not just individual workflow tasks that don’t fit nicely in Jira: it can be team workflow stuff too.

Jira is bad at workload planning

We’re constantly trying to plan our workload from one sprint to the next. (Everything is divided into sprints, even though we don’t really use agile processes in most other ways.)

You can assign your Jira tickets to a sprint and give them a workload estimate. I’ve found that Jira sprint plans are a mixed blessing. They’re too granular, for one thing. And yet there’s a lot that we never capture on a Jira - the time cost of going to meetings, the time cost of updating Jira. We end up getting pulled into new questions and discussions from one day to the next, in nonlinear ways that defy the plans again and again.

But no one can realistically plan their sprints more than 1 or 2 at a time. So if you want to know “Who will be available to start a new project in 8 weeks,” Jira is useless.

Sometimes we just use a Google sheet for workload planning.

Jira is bad at project management

I’ve noticed that when my organization needs actual large-scale project management software, we just use Smartsheet. It’s just a much better tool that models the project management problem better. I see why skilled project managers prefer it.

“Voting with your feet”

In short, for a lot of problems, Jira is a bad tool, even when it theoretically provides the right feature set for the task. So people vote with their feet: they just skip Jira where it’s not effective and use something else.

This is a good strategy. I endorse it (when practicable).

But sometimes Jira can’t be ignored.

What are our options then?

Let’s write our own Jira client

So here’s the thing about Jira. It has reasonable JSON-based REST APIs and an excellent SQL-like query language, JQL.

And this means you don’t have to suffer through all the bad parts of Jira.

To be clear, you can’t fix the Jira workflow for yourself. If that’s what you hate, I can’t help you.

But it’s absolutely possible to fix the awful UX and dodge the disastrous slowness of the UI.

We are software developers, are we not? If someone asks us to use a horrible interface all day, and there is an easy workaround, shouldn’t we just … use it?

I ended up writing my own Jira client. It’s a native MacOS app that connects to Jira, fetches useful data about tickets I care about, and displays precisely the fields that I want to see. It uses SwiftUI, so it has native app performance and uses native app widgets. I added some custom functionality that I want (e.g. it can track my current priorities if I have a lot on my plate, and it can store notes on tickets that are private to me.)

It’s read only (for now). I don’t want to rewrite the whole Jira UI. I just want to improve the UX for things I do a lot.

Here are some things I often want to do quickly:

Finding tickets currently assigned to me.
Finding tickets that I created but am not assigned to.
Finding all the tickets I’ve commented on.
Finding tickets that I have closed in the past.

It’s a lot more common for me to look something up in Jira than to change something in Jira.

Consider: - “What was that ticket I just finished?” - “Where’s the devops ticket I recently opened?” - “What ticket number should I reference in my next Git commit?”

I ask myself these questions all day.

It’s more rare that I actually want to close a ticket or comment on something. I set up my viewer app so that you can click on an issue to open in your browser. Then you can use the Jira UI for editing.

It’s not about rewriting the whole application. It’s just about making it suck less for my own workflow.

I can’t release the source code for it at this point, alas. But I’ll just say it’s absurdly simple to write a Swift app that fetches some JSON and draws some tables in a MacOS window.

I don’t usually work with Swift, but this is a trivial project to set up. It took me a spare afternoon or two, including reading a bunch of SwiftUI docs and tutorials since that’s not what I usually use at work.

(To me, the details of the implementation aren’t interesting, since all you are doing is writing a few UI components based on Apple’s stock interface elements. Every so often, I ran into “how do you update a value without triggering re-rendering loops?” But I’ve done some Javascript component work in the past, so it was quick to resolve those issues.)

I can’t emphasize this thought enough: if you hate the Jira UI, write your own client. It turns a horrible experience into a fun toy project.

I still hate Jira

I still hate Jira. But now, every time I open up my own little Jira viewer, I feel content.

Most problems in the world can’t be solved with code. Which makes it even nicer to solve the ones that can.

Two years in enterprise software

2023-07-21T23:13:00+00:00

A few years ago I decided I wanted to work in a big tech company. I thought it would be an interesting experience, and I wanted to work in a place with more technically advanced colleagues.

So now I work in a big tech company that makes software for other big organizations. What is this like?

I’ll limit myself to discussing some broad organizational patterns. Obviously it’s very delicate to write about one’s work life, and I’m not going to say anything that’s specific to the org, the tech stack, or the business. I’m just going to discuss some organizational dynamics that, from what I can tell, are similar at many large tech companies.

Life in teams

It’s the first place I’ve worked where everything is organized into “teams.” If there are 10 or 3 software engineers in your whole organization, you don’t need to divide things up into teams; you can just be the “engineering department.” Come to think of it, I used to work in places where I did “programming” instead of “software engineering,” and the phrase “software engineering” itself used to sound weird to me. Somehow, I got used to it.

So: you have a team; you belong to it; you keep it going; you hope to improve it… But you might also change teams, or get reorganized into some new structure at any moment. A team is both necessary and ephemeral. The idea of a team hints at sports, although I guess it’s also short for “scrum teams.” As Atlassian puts it, “A scrum team is a small and nimble team dedicated to delivering committed product increments.”

Some weeks feel more nimble than others.

Life outside teams

We’re all in teams, but we’re constantly working with other teams.

Because it is a big organization, the costs of coordination across teams are relatively high. Information can travel slowly from one place to another. Sometimes you meet other teams and find out that they have very different assumptions about how the world works. Here it’s handy to be trained as an anthropologist — it makes it easier to expect difference instead of expecting cognitive similarity across contexts.

You can sometimes start to feel slightly isolated in a big organization divided into so many silos. Some people get news faster than others, and you aren’t necessarily first to hear if you are an individual contributor. I often think of Zane Bitter’s excellent essay, Senior Engineers are Living in the Future.

In this environment, I’ve tried to get good at listening to the faint incoming signals from other planets in our universe, since they often bring essential information.

In a big and complex organization, human relationships are surprisingly important.

When I first got there, everything was confusing, because there was so much local culture and history to assimilate. What made things easier was gradually learning who to ask.

Now if I have a question,I usually know someone who can help me. I know people in customer support, billing, IT, sales, implementation consulting, security, infrastructure, product management, user research, user design, technical writing, architecture, and so on.

It gets dramatically easier to get things done if you know who to talk to. In that sense, relationships have a certain usefulness. You could even call them an “asset,” although I find that term a bit dehumanizing.

In any case, it’s a much friendlier place to work when you have more people to talk to.

Technical specialization

I’m much more specialized than I used to be. I don’t write front end code anymore. I don’t have server admin accounts or write deployment scripts. I don’t usually get the alerts if the systems are broken. I never get email from our customers. (Well, maybe once, ever.) I work strictly on back end software development, focusing on one particular technical area of a particular enterprise product.

The pervasiveness of specialization enables a certain kind of focus, which is the purpose of it, of course. Do one thing and do it well. And yet specialization has some funny side effects. There’s a constant risk of tunnel vision. People get invested in the minutia of code style, in a way that I never saw in small shops. We have debates about automatic code formatting rules, and then we have meta debates about how to refactor the automatic code formatting rules. People sweat over trying to standardize code interfaces and software design patterns. I see a lot of enterprise style code, by which I mean, code that is carved up into many tiny pieces and wrapped in many layers of abstraction. I always see that as an aesthetic preference as much as anything else.

I used to laugh a lot at FizzBuzz Enterprise Edition. If you never saw it, it takes a very simple assignment and overcomplicates it with sententious ceremonies, too many design patterns, and too many abstractions.

Now it’s not as funny… because it hits closer to home.

While our teams are specialized by topic, our work can also involve a lot of role ambiguity. Some software engineers end up doing project management. Some of us end up knowing a lot about the infrastructure, even though we don’t technically work on infrastructure, because we end up needing to solve problems that don’t follow the org chart. Paradoxically, we’re very specialized, and yet we’re often working outside our specialties.

Architecture

The organization needs “architecture” (and the people who specialize in it, “architects”) as the seemingly natural corollary of having so much specialization. We have such large systems that it’s hard to keep track of all of them and how we fit together. Enter the “architect,” a paradoxical role for a specialist in generalization. Or rather: a specialist in thinking about systems holistically.

The career path for software engineers largely points towards becoming architects. It’s a career direction for successful software people who don’t want to become managers.

In practice, our local architects are fun to talk to and very thoughtful. But sometimes it surprises me that systems thinking isn’t considered a core competency for all software engineers instead of a prestigious specialization.

Scale and surprises

We have more users and larger-scale architecture than I’m used to. I certainly don’t work on anything that’s “internet scale,” but our systems do have lots of active users. These users produce a long stream of interesting feedback, new feature requests, and above all, a stream of new edge cases. You have to solve more edge cases when you have a larger and more demanding user base.

Our attention becomes a scarce resource compared to the scale of the system.

To put things in perspective: At my first full time gig, we had a small number of users, and I used to get an email every single time our code raised an exception in production. Days went by without getting those emails.

Now we have a system that manages production exception reports in large volumes. They have options like “Don’t bother me again about this until it happens another 100 times.”

As always, it’s very hard to write code without making lots of assumptions about the world in which it will be executed. It’s impossible for software engineers to anticipate everything, no matter how we try. Thus, while we have fairly rigorous testing processes, we still get surprised by what users do with our systems.

Sometimes it’s unclear if it’s a bug or a new use case.

Jira

We use Jira a lot. A lot.

I do not love it.

I’ll save my thoughts about it for another time.

The tech community

I used to feel slightly more in touch with the rest of the web technical community. If you work in small places, you are more often allowed to try new things without huge barriers. You can test new libraries, new architectures or new styles without having to convince a large organization to approve it. You can probably contribute to open source projects, if that’s relevant.

In a larger organization, there is a much larger internal technical community, which substitutes to some extent for interaction with the larger ecosystem. It’s like living in a microclimate: it has its own weather patterns; it’s less tightly coupled to the surrounding ecosystem.

I’m not saying we are totally decoupled, of course. We keep a close eye on our dependency chain. We integrate lots of new things. Lots of my colleagues read Hacker News to keep an eye on the zeitgeist.

But there’s a certain turn inward just because the internal environment is, comparatively, so large, and so decisive for people’s careers within the organization.

Here’s a good barometer of that.

I used to go to public tech conferences sometimes, not to present, just to listen and learn some new things. Now I have an internal tech conference to attend instead.

Working environment

It’s a pretty good working environment. We rarely have “drop everything for this emergency” problems. That’s much more common in agency work.

My current management is fairly hands off, which I love. You get handed projects, plan timelines to deliver them, provide (lots of) project documentation, and implement. You explain your sprint plan to your teammates, and announce if your plans get blocked by something, so there is some visibility into your work plans, but it isn’t otherwise micromanaged.

That being said, I do notice two things:

The longer you are there, the more meetings you seem to end up in.
The longer you’re there, the more you get pinged with unexpected questions and requests. (Not that you must address them all, but you could.)

There’s a lot more you could say about working in a big public company, but I’m trying to keep it broad strokes.

How to downsize a tiny web server and the services on it

2023-05-09T10:24:00+00:00

Until yesterday, I hosted this website on DigitalOcean. Now it’s on EC2 instead. These are a few notes about how and why.

The old setup

For nine years, I’ve hosted this site on the same virtual machine on DigitalOcean. It was originally Ubuntu 14, and later upgraded to Ubuntu 18, which is now EOL.

It’s always been a tiny system running NGINX. Behind NGINX, I’ve hosted a bunch of different web services. There were several WordPress sites, a short-lived Drupal site, and even some raw PHP scripts. There were a couple of toy Ruby on Rails projects. They all had hosting configuration, SSL certs, their own Linux users, their own databases (MySQL, SQLite), their own server processes (Unicorn, PHP-FPM). Most of them have been decommissioned, and along the way, the system got full of the debris of old projects.

(It’s great to have a sandbox. Highly recommended.)

The server itself was originally the classic $5 DigitalOcean droplet. It has 1gb of RAM and enough disk space for my projects. I also paid an extra 20% for automatic backups, just in case.

But recently, DigitalOcean raised all the prices by 20%, so I started to wonder: Is $7.20/month really the best I can do for basic Linux web hosting?

Tiny web servers on AWS EC2

I like having a Linux web server to play with. I don’t want to go serverless. I don’t want to move all my static sites to S3 and my remaining back end services to AWS Lambda. And while DigitalOcean isn’t the absolute cheapest option for a cheap linux VPS, most of the competition is not a lot cheaper. (And DigitalOcean has been flawless for my use case, with a nice user interface and excellent service, so that’s worth something too.)

But then I noticed that you can get a t4g.nano EC2 instance for $22/year, which is 69% cheaper than DigitalOcean. The big tradeoff is that it’s 500mb of RAM (and an expectation of very low average load, which is probably fine for me 🤞🏼).

So I thought it over and decided that the cost savings was worthwhile. And maybe I could clean up the cruft from my old server while I was at it.

Notes on Amazon Linux 2023

I spun up a new EC2 instance with Amazon Linux and moved all my static sites there. Here are a few notes on that experience.

It does not have Cron. I was kind of amazed by this, since Cron seems like a bedrock part of Unix-like systems. The Amazon Linux devs feel that you can just use systemd timers instead. I only need this functionality for renewing letsencrypt certs, so I followed Steven Westmoreland’s handy instructions to set that up.
The package manager is annoying compared to Ubuntu. There’s churn from one Amazon Linux version to the next, so some of the docs are useless, and some basic packages are unavailable. In particular, Certbot seems like it probably should be present in every Linux package manager by this point. I had to install it with pip, which is a barely-supported official installation approach. I guess Amazon just doesn’t want to encourage using Letsencrypt.

Other than those minor details, it was pretty straightforward to get a new system set up and running. It isn’t exactly the same as Ubuntu, but the differences aren’t consequential.

From Ruby to Go

It was easy to migrate the static HTML sites from one server to another. You just create the right web root directories, copy the NGINX vhost configuration from the old server (with a few updates), and point the deploy scripts to a new place.

My static sites are all built by Middleman, so the deploy scripts are very simple, like this:

#!/bin/bash

bundle exec middleman build
rsync -rzvu build/ webserver:/path/to/webroot

But I do have one project left over that isn’t a static site. It’s a tiny Rails app that shows a little art project about leaving academia. It did some server side rendering for a few templates, accepted user input, and updated records in a SQLite database. Using Rails made it very quick to put together (it was the classic “optimize for dev time” approach).

The problem is, even a tiny Rails app takes a couple of hundred megabytes of RAM to run. It’s not a good choice for a very tiny, low resource web server.

So I decided to rewrite my Rails app like this:

All the front end code (HTML/CSS/Javascript) was moved into a new React app. I’ve never used React before, but it was easy to get going and spin up some basic components. I didn’t try to write a single page app; I just used React to render components on top of static HTML files served by NGINX. The HTML/CSS mostly stayed the same; the Javascript had to get rewritten, but it was fun to do that.
All the back end code was rewritten in Go, which is a language I have never touched before. I was looking for a language that could offer good performance and low resource use, but something with a better developer experience than writing a web service in C. I looked a little bit at Rust and Go; Go was the obvious winner for my use case. It turned out to take less than 500 lines of Go to spin up a basic web service that could accept form submissions, update a SQLite database, and write a JSON file that provided data to the React user interface.

I felt pleased with myself for figuring out how to write a Systemd service that would deploy my app properly, complete with logging and configuration in env vars. I even wrote a handy deployment script that builds for ARM64 (because my EC2 instance runs on ARM), copies the compiled application to the server, stops the service, copies the compiled application to the right place on the filesystem, and restarts the service.

The new Go back end service runs in 13 MB of memory instead of 97MB, and is easier to manage (because it has fewer moving parts and a simpler deployment process).

Migrating DNS

One of the worst parts about leaving DigitalOcean was leaving behind their excellent interface for updating DNS records.

I changed nameservers for my domain, and I moved all my DNS records over to the DNS system provided by my domain registrar (currently NameCheap). But I was pretty frustrated with how much less nice their interface was compared to DO. It required some downtime for the site as well, basically because of the bad UI choices (no way to provide structured data, or to provide configuration before changing the nameserver setting).

This is just my private website, so it was OK. But in a medium-sized org, I’d think you would probably not want to ever switch nameservers if you could possibly help it, or at least you would use DNS services from someone who provided a much better admin experience.

Was this worth it?

If I’m being honest, probably the cost savings ($50/year) was not worth the several evenings I spent moving everything around and writing a new Go application. If I really counted up the hours to move everything over and write two new projects (golang/React), even though it was a relatively quick and straightforward project, I would probably bill someone at least a few thousand dollars at professional software rates. Probably a lot more if it were a full fledged consulting project.

But money aside, it feels excellent to learn new things. It’s great to explore new hosting options, new infrastructure, new programming languages. So … yes it was worth it; but the value was more intellectual than economic.

Just where do env vars come from?

2023-03-06T20:16:00+00:00

Famously, Linux processes accept an array of arguments at start time. In C, this looks like int main(int argc, char *argv[]).

But as we all learn sometime after writing hello world for the first time, these arguments aren’t the only arguments passed to your program at startup. There’s also a second set of arguments, termed the environment. These are the things we know colloquially as “env vars.”

# Passing into argv:
$ my_program --name emma

# Passing into an env var:
$ NAME="emma" my_program

(You can access them in C with getenv but you can also declare them as an argument to main, like int main(int argc, char *argv[], char *envp[]. This makes them available as a local variable.)

Environment variables are a complex system of their own. In an organization like mine, managing environment variables is a huge endeavor.

I started to get curious: What is an environment, technically speaking? And where does the environment come from?

Data structure

The arguments are an array (an ordered list), whereas the environment sometimes acts like a dictionary.

Until I wrote this post, I imagined that in Ruby, the environment was literally a hash (ENV). It turns out that no, Ruby just wraps the OS’s environment implementation in a hash-like interface.

Generally speaking, in unix-like systems, the environment variables are not implemented as a hash table. They are just an unordered array of null-terminated strings, where each string is a key-value pair combined with the character =. (Therefore, you can’t use the = character in an env var name, though it is perfectly valid as part of the value.) The final item in the list of env vars is a null pointer.

You can then use some common accessor functions provided by glibc. The most important ones are setenv, putenv, getenv. Setenv can only add or update env vars, while putenv can also remove them, and getenv is obvious.

Every time you get an env variable from glibc, it does a linear search through the current list of env vars to find a match. (For a slight performance boost, the current implementation filters by the first two characters before doing a full string comparison.)

Are environment variables part of the operating system?

I started to wonder: Are environment variables a fundamental feature of the Linux kernel? Are they part of the definition of a process? Are env variables handled by the system task manager?

Answer: Not really. If you look at what’s stored in the Linux kernel for each process, it doesn’t contain anything like a list of environment variables. (At least that’s how it looks to me from taking a glance at task_struct, the kernel data structure that represents a process.)

However, it turns out that the environment is part of the calling conventions of program execution in Unix systems. For example, it’s common to use a Linux system interface called execve to execute new programs. (execve is what the Bash shell uses to execute a command.) And when you call execve, you must pass the environment variables as an argument: int execve(const char *pathname, char *const argv[], char *const envp[]).

Thus, Linux absolutely does expect that every new process will be invoked with environment variables (even if the environment variables are an empty array). The environment variables aren’t used for process management by the kernel; they are just provided to your program as part of the program data (stored on the stack). You can then use that data for anything you want.

Where does the environment come from?

One of the things you learn as a working software developer is that usually the env vars are inherited from the parent process by default. Of course, the environment can be modified when you invoke the child process, but it’s often the case that, for instance, the PATH and other crucial env vars are propagated down through the process tree, unchanged unless you explicitly change them. There is kind of an implicit tree of env vars, starting at a parent process and propagating across all the child processes.

This being said, there are plenty of special cases where the child environment is reset to blank. Most often that would be for security reasons of one kind or another. As a result, environment variables aren’t really a tree structure as a result; they are a sort of broken tree, logically speaking.

This being said, we can still try to follow the tree up as far as we can. The question becomes: where does our environment get its initial state?

1. The shell

We often run Linux programs through a shell. Thus when you invoke a process, it’s common to get the initial set of env vars from the shell. You might customize these env vars in your shell configuration, typically with export FOO="bar".

A shell like Bash has its own variable handling system (bash:variables.c) that’s separate from the glibc environment handling system. But this variable handling system is itself initialized from the parent environment in #initialize_shell_variables.

So where does your shell session get its initial env vars from?

The shell gets its env vars from its parent process. If you log in from a console, your shell will be spawed by a process called login (the process that checks your credentials and then invokes your designated shell process). If you log in with SSH, your shell will be spawned by the sshd process.

OpenSSH provides a function called do_setup_env that initializes the basic environment variables before loading your shell. These would include HOME, USER, SHELL, TERM, and PATH (see openssh-portable:session.c). The analogous function in login would be init_environ, which does similar operations (see util-linux:login-utils/login.c).

But if you read the code, you’ll see that the sshd process also propagates its own env vars into the child shell processes. Where do those env vars come from?

3. Init

All processes in Linux descend from an init process, which has PID 1, and is the parent of all other processes. On systems I use, the init process is generally systemd.

It looks to me like systemd builds the initial env for a child process from several sources in systemd:src/core/execute.c.

accum_env = strv_env_merge(params->environment,
   our_env,
   joined_exec_search_path,
   pass_env,
   context->environment,
   files_env);

The systemd man page has more details on what those different sources are. When running services like sshd, systemd usually prefers to spawn new processes with a blank environment (except for env vars configured for that specific service). But when running interactive user programs, systemd will generally pass through its own environment vars by default. (See systemd:src/core/manager.c#manager_default_environment.)

If even systemd has environment variables, just like every other process, then where do those come from?

4. The Linux kernel

In the end, they have to come from the kernel. There’s nowhere else at this point, right?

The init process is invoked via the very simple function run_init_process. It executes the init process with execve, using a provided set of argv and envp values:

kernel_execve(init_filename, argv_init, envp_init);

What is the value of envp_init here?

In linux:init/main.c, we finally find the most basic default values for envp_init. They are the following:

HOME="/"
TERM="linux"

There you are: the default env vars set for a linux system. They’re pretty useless, honestly.

(These values have long since been overwritten by the time you log in with SSH. In practice, sshd is the top level source of env vars for your interactive sessions with remote systems.)

5. Arguments to the kernel

But there’s one last funny detail. It turns out that if you pass env var-like arguments into the kernel as arguments at boot time (docs), they will magically become env vars appended to the default envp_init values, and then they will be passed down into the init process (see unknown_bootoption).

So in the end, the very distinction between env vars and arguments breaks down. argc can magically become envp.

It’s unintuitive, but if you think about it, there’s no hard categorical distinction between args and env vars in the first place. You can pass values into your program either way, with only minor adjustments to your code. The distinction between the two is largely a matter of convention and semantics.

An “environment” is a fundamentally complex thing. It makes sense to me that there’s something arbitrary about how we represent it.

My first day using Docker

2022-11-29T21:53:00+00:00

My current org uses Docker containers heavily in our development environment. For the most part, back end engineers rarely configure the containerized environment. We have other groups that do that for us. There’s a development infrastructure group, which overlaps somewhat with the larger infrastructure group.

I get what containers are good for — they get us standardized, repeatable, isolable environments. They make it much easier to keep our development environment in sync with our production infrastructure. And they are a step up, in many ways, from the way I used to do this. The old way was just “Install the development environment and all its dependencies on my workstation,” which gets old fast, and scales poorly.

Anyway, this week I wanted to set up a brand new demo environment, so I decided to learn Docker from scratch.

It took about 6 hours start to finish, including learning how to write a FastCGI process in Ruby. Basically I built a demo project with one NGINX web server container and two back-end application server containers (one running Puma, one running a FastCGI process). Then I used it for some performance testing I wanted to do.

So these are just some notes on getting started with Docker and Docker Compose.

How do you learn your way around Docker?

For what it’s worth, this was pretty much my approach:

Google “how to create a Docker container.”
Figure out which of the existing docs were actually worth reading (reliable, comprehensive, readable, current).
Set out to create the most basic possible Docker environment: an NGINX container that displayed the default homepage.
Create a project folder on a Linux dev box that already had Docker tooling installed.
Make a basic docker-compose.yml file with one service defined.
Browse around in our existing work repos to find a suitable base image for the container.
Try a command like docker-compose up in my project folder.
Watch it build.
Log into the container using docker-compose run or docker exec -it [container] sh.
Install bash inside the container, because sh was mediocre.
Install vim to be able to edit the NGINX configuration interactively.
Figure out how to generate a custom Docker image, by writing a Dockerfile, which added custom packages and configuration to a given base image. (Learned that docker-compose is for orchestration containers at runtime, while a Dockerfile governs image building.)
Fiddle around with NGINX configuration inside the container to ensure that it listened nicely on http/port 80.
Learn that you can use nginx -s reload to live-reload the running NGINX settings without restarting the container.
Read the Docker docs to figure out how to expose a container (on a certain port) to the host. Use port mapping.
Restart the containerized environment and check that you see the NGINX default homepage at http://localhost:8088 (let’s say 8088 was the port on the host that pointed to port 80 in the container).
Put my custom NGINX configuration in a file on the parent host. Use the Dockerfile to copy it into the container.
Rebuild the container a few times to make sure it works.
Make a second app service in docker-compose.yml, using a Ruby 2.7.6 image we had lying around.
Stumble over the question of how to do containerized development in a more exploratory way. (Containers need to have a process running at start time, but when you’re doing new development, you might not know how to start your process just yet. I put tail -f /dev/null as the initial container process, after a handy stackoverflow tip.)
Set up a Ruby project inside the app container with a Gemfile.
Realize that most Ruby web server libraries will need C development tools to build. Install them from inside the container (gcc, build-essential, etc).
Pick through some verbose make output to detect other missing dependencies. Install them too.
After bundle install worked manually inside the container, I moved all the dependency setup and the actual bundle install command into the Dockerfile for my app service.
Set up the Dockerfile to copy my Ruby project onto the container during the build process (COPY ...).
Google how to set up file synchronization between a container and the host file system. I used bind mounts, which is discouraged, since you’re supposed to use virtual volumes now, but bind mount worked just fine for my case. It’s configured in docker-compose.yml, as it’s a container “runtime” feature, rather than a container “build” feature.
Spend some time poking around at how Docker does virtualized networking, to try to figure out how to communicate from one container to the next (since NGINX needs to be able to reach the upstream service).
Try using container IP addresses to communicate (172.16.x.x), but they changed sometimes when I restarted the docker-compose environment. I couldn’t readily provision them at container build time, and it seemed hacky to pass them down to NGINX at container runtime, if that is even possible.
Look in /etc/hosts on the container. Didn’t help me.
Google some questions about Docker networking.
Realize that I’m doing it the suboptimal (basic) way, with bridge networking mode instead of something fancier. No worries there, doesn’t matter in this case.
Read something on Stackoverflow and learned that you can just use the other container’s name as a hostname. It Just Works™ because of some custom DNS setup in Docker.
Update the NGINX config, rebuild the environment.
OK then why does NGINX still not connect to the upstream?
Oh right, the upstream web process needs to be listening on a public network interface instead of on localhost.
Add the correct incantation to the Dockerfile for my app container, rebuild the environment.
It works! Now it’s time to add a second app container for FastCGI (the first one used Puma) and point NGINX at both of them…

In the end I had a containerized environment with an NGINX container plus two upstream containers (Puma and FastCGI).

Then I was able to finish my little demo project, doing some basic performance testing for different Ruby web server processes. (In particular, I was curious about comparative memory usage for Puma, WEBrick, Unicorn and FastCGI-based back end servers. TLDR: FastCGI uses much less runtime memory than any of the alternatives.)

How I didn’t learn my way around Docker

Note that I didn’t do any of these other possible strategies:

Run man docker.
Read a technical book about Docker. (I’m sure there are good ones.)
Watch videos about Docker. (I’m kind of a text-based person.)
Ask a colleague for assistance. (I have lots of highly experienced colleagues in this area, but they’re all busy and it’s fun to teach myself.)
Use an existing containerized environment as a point of departure, and then customize it. (I built from scratch instead).
Have a completely clear plan about how the environment needed to work (e.g. networking, volume mounting). (I was OK with not knowing exactly what I was going to do, as long as it worked in general.)
Use best practices for production-ready containers. (Some of the best practices are too heavy-duty for a basic use case.)

To be clear, any of these approaches would have been valid! I just didn’t use them.

I was happy with my very hands-on, iterative, solo, approach.

Reflections on Docker

I dislike the way that Docker can become a black box in my organization, maintained by specialists even though we all use it all day. I courteously dislike that approach, because what Docker does is really just the basics of Linux-based systems administration, organized in a particular way around a particular core abstraction. I think developers should know their way around those things, even if we don’t know every detail of a complex dev environment.

Anyway, once I dug into it, it wasn’t that hard to understand Docker because I already knew some basic Linux systems administration things, e.g. about networking, file systems, package management, and OS virtualization. So I just applied what I already knew to the Docker environment, trying to figure out “How do I do that here?” Once I thought of it that way, it was all relatively easy.

(It helps that documentation was so easy to find, since Docker is common, well-documented technology.)

I didn’t love some of the inconsistencies between Docker and Docker Compose. I guess they are technically two separate tools, but I wanted them to feel more like an integrated system, instead of having one DSL for one of them and another for the other.

But I did appreciate how Docker pushes you into an ephemeral, fully declarative environment setup.* With a long running virtual Linux system, even if you use something like Ansible for initial setup, it can be tempting to make custom tweaks to a running environment, ignoring your own configuration management. It’s very hard to do this with a containerized environment; you find yourself rebuilding the containers pretty frequently. This causes you to put all the setup in the relevant Dockerfile, with no cheating.

(Fortunately, it’s still possible to log into a container and interactively configure it. If you look at my notes above, I frequently started out with “How do I do XYZ from a shell inside the container,” and only subsequently moved the incantation into the Dockerfile. This speeds up the dev feedback loop.)

It was a fun afternoon of digging into this stuff, honestly. It’s not every day I learn new things.

* To be precise, a Dockerfile is an imperative build script, but docker-compose wraps it in a declarative configuration system.

Were you root?

2022-10-18T16:23:00+00:00

Back when I was a teenager, and all I had was an old Macintosh to hack on, I used to think it sounded amazing to be a Unix system administrator.

In particular, I was super excited about being root. There was such a mystique around that user. I knew it was somehow powerful. Dangerous. I found it mysterious.

I was so excited about it that I tried typing su once on my ISP’s Unix system. (This was back in the days when ordinary dialup accounts came with Unix shell accounts.) I obviously had no idea what the password was (and I didn’t even try to do anything remotely like “hacking”), but my failed attempt ended up in their logs.

The ISP was not pleased. They disabled our account.

My dad had to call up the company owner (I think they had been high school classmates; it was all very small-town) and explain that I was not going to do that again.

They re-enabled our account.

(And I never did that again.)

A few years went by. I got a job doing some web software development (mostly Python). I had more legit reasons to play with servers. I learned enough to be dangerous.

One day, I was at some kind of web training and I wanted to be helpful to the event organizers. So they gave me admin access to their little Debian server and told me to make myself useful.

I was trying to learn my way around their environment, but I wasn’t really too familiar with how it was laid out. I was exploring and I typed cat /dev/something.

(I didn’t know yet that /dev/ is the directory tree that Linux uses to expose system devices as if they were files. Nothing in there is actually a normal file.)

The whole system froze.

Whatever I had done was impossible to exit or cancel.

The shell stopped responding. So did httpd.

Sheepishly, I had to approach the event organizers and explain I had just crashed their webserver.

“What did you do?” they asked.

“I ran cat on some file in /dev/.”

“Were you root?”

“Yeah.”

They chuckled and hit the restart button on the web server. Problem solved.

On that day, root lost its mystique for me.

It’s not exciting to break something that people depend on. It’s embarrassing.

For professional tech people, admin rights on servers are often as much a burden as a privilege.

If you have root and the system breaks, it’s possibly your fault and you probably have to fix it.

I’m not saying that all hierarchies of access are fair or great. We should ask questions about who has power on computer systems.

But sometimes… the access controls are guardrails. They can keep the users from breaking things.

I suppose we all probably have to learn that the hard way.

Around then, I also remember being in a programming class in high school. The teacher had installed some security software on the lab computers. You couldn’t modify most of the files on the system.

I showed my teacher that the security software was inadequate. It only protected against deleting files through the ordinary Macintosh Finder. I demonstrated that you could delete any arbitrary file, including the security system configuration, just by writing a few lines of code.

The teacher was way ahead of me. He laughed. He wasn’t alarmed.

He said, “Look, Eli, the security software isn’t really there for you. I just installed it to stop random kids from renaming the system files to have dirty names.”

I smiled.

And I see his point even better now than I did then.

Thoughts on URL path routing

2022-10-07T00:30:00+00:00

URL path routing is one of those things that gets more interesting the longer you think about it.

(This post is geared toward mid-level web developers. It won’t teach anything to the NGINX core developers.)

By “URL path routing,” I mean the part of a web server process that parses incoming HTTP requests, looks at the request path (and the HTTP verb), and ensures that your request is handled by the right handler function for that path.

The first line of an HTTP request looks like this: GET /path/to/something HTTP/1.1

So the question is, how does path/to/something get handled?

(The HTTP 1.1 spec calls this part of the request the request-target. I’ll call it the “request path” here, which is a term widely used in practice.)

The file system is an implicit router

If you look at the most basic, old-school, static web server, path routing is largely invisible.

Every URL path is mapped to a file on a server. You put files into your document root folder, and the web server serves them right back to the users. If your document root is /var/www, then you can put story.html into that folder, and it automatically shows up at website.com/story.html.

In this case, request path routing is almost a one to one map onto a directory tree. Almost one to one, but not completely. There are questions to answer. Options to configure.

The first special case here is this: What do we return if someone requests a directory instead of a file? Here we find our old friend index.html, a convention that you see in a basic NGINX configuration like this:

location / {
  index index.html;
  try_files $uri $uri/ =404;
}

Which tells the NGINX index module that if you request a path ending with a slash, the index.html file beneath that path will be returned. It also says that if you request /folder with no trailing slash, then the server will check if /folder/ (the corresponding directory) is available instead.

I must say that this approach to URL routing is surprisingly effective and scalable, up to a point. You can have folders and subfolders many levels deep. Maybe you can even use symlinks. I’ve seen huge archives (tens of thousands of files?) served up from this sort of configuration.

Back in PHP: a fractal of bad design, Eevee described filesystem routing as amounting to “No Routing,” period. But I think I would say, instead, that the file system itself is also a sort of routing system, which maps file system paths to different physical or virtual storage handlers. In a sense, a static web server just delegates path handling to another routing system.

But file-based routing breaks down as soon as you want to respond to a request with something other than the contents of a file.

Why do we route, anyway?

What if you want to send back to a user the result of an arbitrary function?

(And, let’s say, you don’t want to do anything as silly as wrap it in a Linux kernel module.)

Technically, a path routing layer is not even required for a web server. You wouldn’t need it for an application that handles every request the exact same way. The most basic Rack application just looks like this:

# config.ru - Version 1
run -> (env) { [200, {}, ["The meaning of life is 42\n"]] }

That’s just a single function* that always returns the same response, no matter what the input parameters. You can request any path and this application will return the same output. (*OK OK, technically speaking this is a Ruby stabby lambda, since Ruby doesn’t quite have first-class functions.)

So now what if you want to handle two paths differently?

# config.ru - Version 2
run -> (env) {
  case env["REQUEST_PATH"]
  when "/secret"
    [401, {}, ["The real meaning of life is a secret\n"]]
  else
    [200, {}, ["Officially, the meaning of life is 42\n"]]
  end
}

Now we have a handler function that inspects the input and responds differently depending what path you send it.

OK, but what if you decide that this is an ugly, un-extensible bit of code? What if you refactor this?

I’ve written test code that looks like this:

# config.ru - Version 3
run -> (env) {
  case env["REQUEST_PATH"]
  when "/config"
    config_response()
  when "/info"
    info_response()
  when "/normal"
    normal_response()
  else
    error_response()
  end
}

Now you have a function that just maps between request paths and handler functions.

Congrats, we just reinvented the use case for a routing layer! It’s just a standard way of mapping possible inputs onto handler functions. Instead of handling every request with the same function, a path router gives us a layer of indirection so we can send different requests to different places.

Routing, fundamentally, is a software design pattern, a permutation of what Kwindla Hultman Kramer calls the dispatcher pattern. It addresses the general problem, How do you map a complex input set to an arbitrarily large set of possible handler functions, when a case statement is inadequate? The specific implementations we’ll see here are all just possible solutions to this general question.

You can deduce the need for a routing layer by considering two problems with Version 3.

It’s full of hardcoded string constants and hardcoded handler names. What if we wanted to make this function configurable, so you don’t have to edit the source code every time you change the handler configuration?
How do you make this function as performant as possible?

Routing algorithms

Considering Version 3, the first question is: what is the performance of a big case statement?

Looks like it runs in O(n) where n is the number of possible path handlers. Not the best, especially as n gets larger. And we really want URL routing performance to be optimal, since this is a component of our web server that gets called on every single request, and it’s just overhead, taking time before we can even start generating the right response.

There are various approaches that people use at this point. Here are two common ones:

A routing search tree
A routing hash table

Let’s take a look at the search tree approach first.

Dynamic routing 1: An NGINX location tree

I’ve used NGINX as a general purpose webserver for a while. It’s highly configurable; the application is organized around the concept of different modules and multiple phases of request handling.

Let’s take a quick look at the way that static (non-regex) location routing is handled by the NGINX core http module.

The route handlers you see in something like NGINX are usually only accessible indirectly, in normal use cases. You don’t write a route mapping function like the one we wrote above; its functionality has been abstracted into an operation on a configuration tree. You don’t, directly, write a handler function for specific routes either; you invoke an NGINX module that handles some specific path that might, then, pass along a request to your actual application code. All you need to do as an ordinary developer is to generate the needed configuration data, which tells NGINX which modules should handle which paths, and with which configuration options.

Fun Fact

Technically, it is also possible to write ad hoc handler functions in NGINX configuration, if you really want to do this. You can abuse if and return from the rewrite module to build arbitrary logic:

if ($the_world_is_round) {
  return 200 "Welcome to a heliocentric universe, ${user_name}!";
}
return 500 "Hello ${user_name}, did you still believe in geocentrism?";

There are also modules that implement full-fledged scripting languages inside NGINX, such as njs. You could develop arbitrarily complex code this way, though it’s probably a terrible idea, and I’ve never seen that strategy used in practice. We usually do configuration in NGINX, and then put all our business logic in some other framework upstream of it.

For path routing purposes, each separate configuration node in NGINX is called a Location. A location can be defined as a prefix string, an exact string match, or a regular expression. The general strategy is to find the best possible prefix/exact match first and then test all the regular expressions. (The regular expressions are all tested in the order they appear in the configuration folder; I don’t know how to optimize a search through a list of arbitrary regular expressions.)

NGINX parses its static (non-regex) locations into a ternary search tree with this kind of structure:

         [root element]    <== represented by `/`
                |
                |
               egg
   box -----/   |    \------ house
  /   \         |           /     \
 bat   cat      |        glass    plate
                |
              plant

This would be a root with seven locations nested beneath it (/bat, /box, /cat, /egg, /house, /glass, and /plate), and one child location, /egg/plant.

It’s sorted lexicographically with lower values to the left, higher ones to the right, and child trees (termed inclusive) in the middle. The parent of each subtree is set to the middle element of the relevant subset of location blocks. It all descends from the root location.

Looks like the search time here is is O(log n) for n sibling location blocks, so we can see why they use this structure. The parent-child relations are just a linear search as far as I know; it will collapse into O(n) in the pathological case where you had only one gigantic parent-child structure (say a single folder tree 50 levels deep, with no sibling folders).

Once NGINX finds the correct location block, the location block (and other configuration) will invoke the relevant NGINX modules to build and return a response, handle headers, check authentication, and so on.

(You can learn a lot more about how this works by compiling NGINX with --with-debug and setting the debug log level; it then will report precisely how it searches through the static location tree.)

Dynamic routing 2: A Drupal URL Alias table

Meanwhile, consider an alternative approach that I’ve also seen in the wild: the routing table.

The routing table is most viable if you are using uniform request handling functions with no dynamic path inspection or parent-child directory hierarchy. Suppose your routing table looks like this:

| path       | handler_function | id |
|------------|------------------|----|
| /about     | page             | 1  |
| /company   | page             | 2  |
| /faq       | page             | 3  |
| /contact   | form             | 1  |
| /buy       | form             | 2  |
| /buy/now   | form             | 3  |

You only have two handler functions, page and form, so all your router has to do is build this hash table, use path for the keys, and then call the named type handler function with the given id parameter. Lookup is theoretically O(1), it’s perfect! You could use such an architecture to map URL paths to files, to database rows, to anything with functions and arguments!

What’s the big use case for a routing table?

In short: User contributed path structures. You don’t want end users to have to write a Ruby method. You don’t want to let them anywhere near an NGINX configuration file. You just want to show them a text field: “What path should this page have?”

In that context, a routing table is cheap; it’s safe; and it’s flexible without needing code or infrastructure changes.

Drupal 7 does something like this with a url_alias table (see system schema). This is just a relational database table with an alias column and then some other columns telling the system what to route to. Users get to specify the alias value. If they want it to look like a directory tree, they can just put slashes into the path value. It’s very brittle because end users usually don’t do a good job of maintaining a tree-like structure. (Predictably, there is an additional module you can use to auto-populate these values, helpfully named pathauto.)

This routing table can’t be a hash table in memory, because unlike NGINX, Drupal is a PHP application that doesn’t persist configuration data between requests. So it’s still a O(1) indexed database query, but it’s a database query per request to figure out the right path. I’m pretty sure it’s cached, at least, so it is probably quick when the cache is warm.

The actual schema looks like this:

| pid | alias     | source       | lang |
|-----|-----------|--------------|------|
| 100 | about     | node/1       | en   |
| 101 | home      | node/2       | en   |
| 102 | faq       | node/3       | en   |
| 103 | contact   | form/1       | en   |
| 104 | buy       | form/2       | und  |

With indexes:

index alias_language_pid on ('alias', 'language', 'pid')
index source_language_pid on ('source', 'language', 'pid')

Note that it includes a locale parameter lang (so you can route the same paths differently in different locales). It has bidirectional indexes (you can look up both by alias and by source). And the source field points, curiously, not to a function but to an internal path.

It’s all a bit funky. It turns out that Drupal’s url alias system is bolted on top of a whole separate routing system, the strangely termed “menu” system which handles Drupal’s internal paths. So the url alias table just tells Drupal what system path should handle a given request, and then the menu system figures out how to actually call the right handler (node or form in this case).

You have to register internal URL handlers by providing “configuration” in PHP code like this one for the node module:

  $items['node/%node'] = array(
    'title callback' => 'node_page_title',
    'title arguments' => array(1),
    'page callback' => 'node_page_view',
    'page arguments' => array(1),
    'access callback' => 'node_access',
    'access arguments' => array('view', 1),
  );

Believe it or not, this PHP configuration then gets persisted in yet another database table, menu_router. The full implementation uses 25 columns of settings and arguments in addition to the path spec itself, and it’s not very clearly designed, since “menus” in Drupal are an awkward mashup of a path routing system with UI navigation menu configuration.

I’m not going to go any farther into the weeds here. Drupal 7 is generally agreed to have really ugly technical architecture. I will say that this whole awkward system actually does work in production, and has powered innumerable large websites. I used to help maintain the website for a billion-dollar-a-year home building company built mainly on Drupal 7. It was a huge mess of spaghetti code too, but that wasn’t even the fault of the framework itself. And it lets non-technical users easily generate their own URL path structures. That’s arguably a big core feature for a CMS.

The Daily WTF of routers

Here’s another fun thing I found at a previous place I worked:

A homegrown content management system built on Ruby on Rails where URL routing was really, unusably slow.

It turned out to be implemented with a recursive path lookup function that generated n x m database queries to produce the routing table, where n was the number of pages and m was the depth of the routing tree.

To make matters worse, this was for a multisited system that didn’t assume that the site root started with “/”.

The algorithm was something like this:


class Router
  def route(request_path)
    # Note: This cache worked OK in prod, but was disabled in development:
    routes = Rails.cache.fetch("page_routes") do
      Page.all.map {|page| [page.id, page.path] }
    end

    routes.find { |route| request_path == route }
  end
end

class Page < ActiveRecord::Base
  belongs_to :parent, class_name: "Page", optional: true

  def path(site_id)
    if parent
      # This does a recursive lookup of parent paths,
      # and loading each parent is a separate db call:
      [self.filename, parent.path(site_id)].join("/")
    else
      [self.root_path(site_id), self.filename].join("/")
    end
  end

  # this is also a database call:
  def self.root_path(site_id)
    self.find_by(site_id: site_id, is_site_root: true)
  end
end

In development, the whole routing table was reloaded on every page load. It ended up causing huge delays during development on large sites.

I improved the performance of this by roughly 50% with some basic improvements, like not looking up the root_path again for each route. I then suggested shifting the design to use a Drupal-like routing table in the database. However, I believe they may have since abandoned the whole product, which would have rendered any further architectural improvements irrelevant.

Sometimes the products cease to exist long before they can be improved.

Well, we all know the problems with premature optimization.

Conclusions

Back in the day, I used to understand web servers as being fundamentally document-based. Static Apache website style. Request a path; it matches a document in a directory tree on a disk; and the document gets sent back to you.

But you can learn a lot from decoupling your understanding of a web server that speaks HTTP from the concept of a document. In a more abstract way, you can think of an HTTP request as just being a function call with some input parameters. And one of the parameters just happens to be something we call “a path.” It’s just hard to think of it this way when you start out by staring at the configuration layer of something like NGINX. The “functional” part of it is deeply buried by that point.

What’s more interesting is the huge set of design tradeoffs you can make here. What’s the ratio between convention, configuration, and pure flexibility? How much technical expertise will be needed to create a new route? Will your user want to write a request handler from scratch? Will they want to use a DSL to configure routing without needing to execute absolutely arbitrary code? Or will you make routing that’s so simple that even a nontechnical user can use it, as in a CMS?

You can see here how each framework draws the lines differently, and then it’s just up to the users to work with the constraints — or struggle against them.

The hacker spirit

2022-10-03T15:47:00+00:00

I have a PhD in cultural anthropology and I’m a software engineer. How did that happen?

I guess I’ve had a bit of the hacker spirit for a long time. It was part of the culture I was raised with. Building things. Stumbling into new places. Asking questions. Being skeptical, but not just in a negative way: skepticism can be a very hopeful gesture.

Exploring. That’s what it’s about for me: exploring.

It seems like I have had an unusual path through tech, compared to everyone who was a CS major in college and worked in tech their whole career. Here are some notes on how that happened.

I was a kid in the late 20th century. When I was little we had a Macintosh SE (when my dad did graphic design) and an old PC with DOS. I wrote some little BASIC programs and (believe it or not) some HyperCard stacks. I think the first machine I owned was a Macintosh Performa. I spent a while trying to learn native GUI programming at that point, using a thick reference manual for Apple interface building. I think the only thing I ever finished was a screen saver demo animation.

Computers were only one of the technical systems I used to like. For a few years, I was in love with video production. At that point the professional gear was still largely analog. I spent some time in TV studios — they were little, but run by professionals. I crewed some educational broadcasts that went out on satellite; I was an intern at a local cable company for a year; I went on some multicamera shoots, with a van on location. I never tried to do it for a living, not even close. I just loved being around the technology, the visual design part too, framing shots just the right way, cutting clips at just the right moment. I played with editing gear and I made a trippy video of my own about the alienating landscape of my high school.

Then I got deep into lighting for theatres. I worked in summer theatre as a stage electrician; I ran a followspot one season; I climbed a lot of ladders and catwalks, lugged around a lot of gear, and worked late nights for free or really bad pay. I had a lighting design teacher from the local university, which had a MFA drama school. He brought me as an assistant to one of his professional gigs, doing lighting design for an opera. I learned how to design lights for a show, how to run the lighting console, how to plan the logistics.

There’s a lot of hacker spirit in theatres. You’re building things that you just dreamed up. You’re running at the very limits of your capacities. It’s a wild place.

Meanwhile, I was taking some computer science classes — mainly C style languages with object oriented features thrown in, as used to be the rage in the 1990s. I must have done a year of C, a semester of C++, and a semester of Java. We did quicksort and I wrote a really basic web crawler implementation. Meh.

I just didn’t love CS classes. Whatever the hacker spirit is, they didn’t have enough of it. They were taught in a pretty rote, “memorize this” way. They were dull. The intro ones were all a little too easy. And they didn’t help me (when I was 18 or 19) figure out the answers to the big existential questions that I desperately wanted to figure out.

So I ended up studying a humanities field, cultural anthropology, that was a lot better at big philosophical questions than anything I found in STEM.

Around the same time, I found out I could get paid to build software without finishing a CS degree.

That was the beginning of a long, meandering period where I learned tech on the job.

I started to work for a language laboratory in college, Cornell’s Language Resource Center. I learned some Python, which powered our then-fancy Zope platform. I wrote online quiz software for language learners (we weren’t using commercial learning management systems in those days). Before long before we needed non-document-based data storage, so I set up a MySQL instance, made it talk to Python, and learned something about normalized database schemas.

Then I went straight to grad school in cultural anthropology and didn’t write much code for a few years.

After I finished the field research part of grad school, I got back into web programming at the University of Chicago, in the IT group for the humanities graduate school. At first I worked on their public-facing websites (mainly Drupal); I remember building a custom event planning module for a big annual event. Next door, there was a web applications programmer who was building internal administrative software in Ruby on Rails, which sounded more exciting. He gave me a crash course in Ruby, in the MVC pattern, and in test-driven development. Soon he left for a startup, and I got hired into his position.

I found myself going every day to the office and sitting at a big software development setup with a bunch of monitors.

What I loved about my first full-time software development job was that I had so much to explore.

The culture of technology was wild around then (~2012). In academia, things move really slowly, but in tech, there was constant flux, shifting trends, unstable new projects. The “new Javascript framework every month” thing was setting in taking hold. A lot of history was happening, somehow.

My boss told me once that I had a particular skill: it wasn’t just that I could write code, it was also that I could take a preliminary set of requirements — usually not very clear ones — and then build working systems from that starting place, pretty much all by myself, without needing micromanaging. I have to say, I enjoyed the autonomy I had.

I was building administrative applications in Ruby on Rails and Javascript. So we had clients, but they were only internal clients. I got a salary and we were free from commercial pressures.

I built our testing infrastructure up from almost nothing. I built a realtime dashboard app to monitor activity on our products. I got a lot of practice triaging production exceptions (which ones are urgent? which ones can wait a little bit? do we need extra logging or debugging?). I worked on performance, on authentication systems, on database design. I built lots of things.

As long as there were tests and there was a good project plan, I was trusted to be an autonomous professional and to do solid work.

I didn’t realize until later that in bigger commercial environments, you usually don’t run your own projects as a software engineer. But in that case, I was the only web application developer in our group, so I spent quite a bit of time talking with our clients, mainly admin staff who needed software to simplify their work. I knew almost every one of our users by name. They could email me and I would help them if they needed support.

That human connection was possible because we were writing software for only a few hundred users at most.

These days, I have to say I miss that sense of personal connection with the users.

Looking back, it seems so improbable that I could go from playing with HyperCard in the 1990s to being a professional software developer a few decades later. I suppose life is full of those surprises.

I just still look out for the moments of joy and exploration in what I do.

And I automate the boring stuff.

How this site is built

2022-10-02T20:33:00+00:00

Update: For a more recent account of the site setup, see How to downsize a tiny web server and the services on it.

OK, here we are, on my website. How is it generated? How is it hosted?

First, let’s talk about the context. The technical constraints always come from the context.

This is a very low traffic site, with only static content. It contains a basic website, plus some downloadable PDFs of things I wrote.

That’s already a very different problem from the things I work on at work!

Priorities

First of all, since this is my project, I get to choose the priorities.

I care a lot about the content. There isn’t a lot of new content here, but what’s here should be solid.
I care about site performance. Fortunately, static HTML on modern hosting is already fast, so I don’t have to do much to improve it.
I care about continuity. If I publish a piece of writing here, I want the link to keep working indefinitely.
I care about basic security. Note 1: Don’t run web applications unless you can commit to security patches… Note 2: I do have a valid TLS cert now. Though I certainly didn’t have a TLS cert in the early days - nobody did, not for a site like this. I remember when I first wanted one, it was still a huge pain to get a certificate — you had to buy it through some awful DNS registrar package deal, and manually download new cert files every time it expired. Now I just use letsencrypt.
I care about minimizing useless maintenance overhead. I care about avoiding unnecessary runtime dependencies (which can in turn create security issues).
I use this site to stay in touch with basic Linux administration and old school web technology. I like tinkering with my own HTML. I like poking around at nginx configuration. I like remembering how much you can get done with something super, super basic. No containerization here, not so far.
I care about minimizing hosting costs (up to a point).

In sum, I’m here to learn a few new things, play with servers a little, and keep a stable web presence.

Non-priorities

There are also some things I don’t really care about.

I don’t really care about analytics or server monitoring (as long as the server is not on fire).
I don’t care about tracking inbound links.
I don’t care about supporting discussion or dialogue on the site itself. I love talking to people, I just don’t have to run a forum right here.
I’m not optimizing SEO. At all. But my name is already unusual so that’s doing some SEO all by itself.

Hosting

I’ve had a lot of web hosting arrangements over the years. I think the history was something like this:

1999: My very first, very silly website was static HTML hosted by my ISP. They had a unix system you could log into, back when small-town local ISPs were more common, the kind that could serve a website for every user at server.com/~user. I think you had to upload the files with FTP. It wasn’t encrypted, but using a dialup connection to their systems, maybe it wasn’t all that insecure?
2000-2002: Every dorm room at my college had wired Ethernet, a public IP address, and a stable hostname. So I self-hosted my website from an ancient version of OSX (which had a built-in web server back in the day).
2003-4: I had a personal website in some sandbox folder of a shared campus web server.
2005–6: I was starting grad school and didn’t have a web presence for a year or two.
2007–2014: I registered this website and set it up on “shared hosting.” It was cheap but irritating; I disliked cPanel.
2014–present: Switched over to hosting on a cheap virtualized linux box. Originally it ran Apache, later switched to nginx. It’s boring. It works great. CPU load is usually near 0%.

Site generation

Historically, this site has always been basically static HTML, with hand-rolled CSS. I used to write a new stylesheet every so often, just because I could.

I’ve always supplemented the static files with some extra programmatic tools, when I needed them. For example:

In 2002, the site was static, but it hosted some images on a Python-based web server, which provided analytics.
From 2007 to 2022, I had a WordPress blog hosted in a subdirectory of the main site. I liked the (old-school) WordPress post editor, and I liked blog comments back in the days when people actually used to use them.
In about 2013, I wrote some dynamic code to programmatically display my progress in writing my dissertation. It used data from a git repository history and from Asana (a task tracking application) to display the progress. The data on the server had to be updated periodically, with a script I invoked manually - I never needed to automate it.
In 2016 or so, I wrote a Ruby script to generate the navigation menu for the static files.
In 2022, I ported the static site over to Middleman to make it easier to maintain. I also like Markdown.

Right now, a static site generator is the sweet spot for me between “100% hand edited HTML files in a directory” and “100% dynamically generated content.”

I do wish I had a lightweight solution for contact forms. I used to use PHP for that once in a while, but only because I used to need it for WordPress. Now… 🤷‍♀️.

Conclusions

This site looks pretty basic, but it actually takes a lot of work over the years to keep it going. The requirements of the web are always changing. I don’t want it to look too dated. I want it to work on mobile. I want it to keep running for decades at a time.

Minimalism is not actually all that cheap, when you think about it.

Eli Thorkelson

Bike safety rules in cities

Basic rules

Intersections and traffic

Relationships with drivers

Failures of communication

Parked cars

Road shapes

Relationships with pedestrians

Miscellaneous risks

Gear

Accidents

Meta

Caveats:

Against the Five Whys

Why the Five Whys is inadequate

Alternative approaches

Ruby 'require' can deadlock

Reproducing

Why does this happen

Fix

Raise isn't a Ruby reserved word

The risks of looking too closely at things

Anomalies in a human body

Anomalies in a building

Anomalies in software systems

Reading graphs and diagrams

Training the eye

Diagrams tell stories

All diagrams lie

Nonlinear behavior is common

Your understanding of diagrams reflects your understanding of the system

On incident fatigue

What we did: Improvisation

What would I do, next time?

Coda: What we did afterwards

Naive optimizations of CI job allocations

What counts as optimal?

Some naive allocation algorithms

Measuring the results

Takeaways

Coda

Message in a bottle (in the comments)

Isolation vs coupling, or the problem with silos

Tenant silos, or horizontal boundaries

Web stack components, or vertical boundaries

Discussion

Coda: Leaky abstractions

The pitfalls of things that seem easy

It looked like an easy hike: An allegory

Software is full of this

On actually reading error messages

The invisible layers of the stack, or why whitespace broke authentication

The mysterious case of whitespace that… breaks API authentication

Where does the problem occur?

What does API Gateway really do?

It’s the invisible layers of the stack

But why would WAF block JSON because of extra whitespace

Fixing blind spots

A rat’s nest of configuration management

Configuration strategies

Rails configuration

Fixing inconsistencies one by one

Virtual patch panels

Best practices

Coda

Coordinating Unicorn worker processes with flock

Naive approach

Flock(2)

Ruby implementation

Further reading

How to instrument DNS lookups in Ruby

Sampling methodology

Implementation

Commentary

True BASIC/4d/Star Pattern/01

DIY unit tests

A DIY test framework

Discussion

Reading programming languages you don't understand