The Distributed Systems You Already Use

Most people think of distributed systems as something that lives in a data centers. But you’ve almost certainly used some of the biggest ones this week: you flew somewhere, rode a train or bus, got something delivered, or paid for something with a card. Aviation, public transit, logistics, and finance are my favorite examples, and I think that’s not a coincidence. They’re the same problems I work on at my desk, just with wings, rails, wheels, and ledgers.

A distributed system is a lot of independent parts that have to cooperate to do one job, with no single part that sees or controls everything. Each of these has that shape, and each one is a good teacher.

Aviation: handoffs, redundancy, and learning from every failure

A flight is a relay. Your plane is handed from one air traffic controller to the next as it crosses sectors, and every handoff follows a strict, rehearsed protocol so that nothing is lost in the gap. Each controller only sees part of the picture, and the system works because the parts agree on how to talk to each other.

  • Redundancy is the default. Aircraft carry multiple independent systems for the things that matter, and procedures assume that any one of them can fail.
  • Failures don’t stay local. A storm over one hub produces ground stops and delays hundreds of miles away. That’s a cascading failure, and operations centers spend their days containing it.
  • The postmortem culture is the best there is. After a 1974 crash in which another airline had seen a similar problem but had no way to share it, the US created NASA’s Aviation Safety Reporting System: voluntary, confidential, and non-punitive, run by NASA as a neutral third party so people can report mistakes without fear. The point is to learn what happened, not who to blame, and the lessons go back into checklists and procedures. Software teams talk about blameless postmortems. Aviation has been doing it for decades.

Public transit: schedules, partial failure, and graceful degradation

A subway system is a distributed system you can stand inside. Trains, signals, track, stations, crews, and riders are all independent actors sharing the same infrastructure, and something is always a little bit broken somewhere.

  • Shared resources create contention. Only so many trains fit on a stretch of track, so one delay ripples backward through everything behind it.
  • Good systems degrade gracefully. When a line goes down, the best networks keep most of the service running, reroute what they can, and tell riders what’s changed. Riders do their own failover, too: they check an app, switch lines, and walk.
  • Open data made it better. Many agencies publish schedules and live vehicle positions in open formats (GTFS and GTFS Realtime), and the MTA offers its subway and rail feeds for free, which is why you can open any number of apps and get a good answer. Open protocols let a whole ecosystem build on top of one system without asking permission.

Logistics: tracking, hubs, and moving things through a system nobody fully controls

A package crossing the country passes through trucks, planes, sorting hubs, and local routes, run by many people who will never meet. Nobody holds the whole route in their head. What holds it together is a stream of scan events, each one a small record that says “this thing was here at this time.”

  • Hub-and-spoke is a design decision. Routing through central hubs trades a bit of distance for enormous efficiency, and it concentrates risk in the same places it concentrates volume.
  • Tracking is observability. The scan history on a package is a trace. When something goes wrong, you find the last place it was seen and work forward from there.
  • Peak season is a load test. Every year demand spikes far beyond a normal day, and the system survives on capacity planning, queues, and knowing which things can wait. That’s backpressure, load shedding, and autoscaling with trucks.

I spent nearly four years at UPS, which gave me a good look at how much work goes into making this look effortless.

Finance: ledgers, settlement, and trust

Money feels like a number in an app, but moving it between two banks is a distributed systems problem, and an old one. There is no single database of who owns what. There are thousands of institutions, each with its own ledger, and they have to agree on the result without any one of them being in charge.

  • Authorization and settlement are different things. When you tap a card, the purchase is approved in seconds, but the money moves later, in batches, through clearing and settlement. For a while, the world is in a pending state that everyone has agreed to treat as good enough. That’s eventual consistency, with a legal framework around it.
  • Messages are not money. A payment message, like the ones that travel over a network such as SWIFT, is an instruction. The actual movement happens in ledger entries at the banks involved, often through accounts they hold with each other (banks call these nostro and vostro accounts, the same account seen from each side), and ultimately at a central bank. Keeping those ledgers in agreement is the daily work of reconciliation.
  • Failure is expected, so there is an undo. Chargebacks, reversals, and return windows exist because things go wrong. It’s the same idea as a compensating transaction: you can’t always roll back, but you can make the books right again. And anything that moves money has to survive retries without paying twice, which is why idempotency is a core idea in payments.
  • The speeds are different on purpose. In the US, Fedwire settles each payment individually, immediately, and finally. ACH collects payments into batches and settles the net position between banks on a schedule, and it can be reversed within the network rules. Securities settlement has been getting faster too: the US moved most trades to next-day (T+1) settlement in May 2024. Each is a trade-off between speed, cost, and risk.

Underneath all of it is trust. Banks extend credit to each other, courts enforce the contracts, regulators set the rules, and central banks stand behind the system. Most of the time that trust holds, which is why you never think about it, and when it breaks, it breaks everywhere at once.

That’s also what makes crypto and Web3 interesting from an engineering point of view. They’re an attempt to replace some of that institutional trust with consensus among machines, and they come with their own trade-offs. I spent time building in that space at Forbes, and I find the question of where trust should live more useful than any argument about whether it’s the future.

How I learn a new industry

When I join a new company, I try to learn how the whole industry works, not just my corner of it. Delivering packages, publishing media, and running a bank are very different businesses, but understanding the physical or financial reality underneath makes me a better engineer. It tells me which failures actually matter, what the regulations and customers expect, and why a system was built the way it was. The best reliability work starts with knowing what the system is for.

Why this matters to me

Software people didn’t invent any of these ideas. Redundancy, handoffs, graceful degradation, observability, and blameless review all show up in the physical world first, and they got there because someone learned the hard way. When I work on reliability, I’m often borrowing from the people who keep planes in the air, the 7 train running, and the money moving.

It also shapes how I think about the people side of the work. The best of these systems succeed because of the people operating them: the controller, the dispatcher, the person running the sort at 4 a.m. Good engineering makes their job easier, which is the same thing I’d say about platform engineering.