The book introduces the CAP theorem and discusses the trade-offs between consistency, availability, and partition tolerance. The idea is that you can build a system with only two of these three attributes.
For instance, you can build a distributed system with strong consistency, which writes to multiple data centers around the world before confirming the transaction, but if any one of those data centers is slow or offline, the system is not fast or easily available. On the other hand, you could build an eCommerce store with eventual consistency that writes the transaction to one data center only and confirms the transaction, but if the user loads the page again and the data is retrieved from a different location, their transaction might not have been replicated yet and it’ll show up as missing.
The author then talks about different types of patterns for working with containers, the first one being single node patterns, and then the sidecar pattern. The sidecar pattern is where one node has a HTTPS proxy, and that routes traffic over HTTP to your web app or container, which reminds me a lot of how Heroku works with the HTTPS proxy, where everything after the proxy is accessed via localhost. The sidecar pattern can also be used to deploy a container alongside your app with a shared file system which would then watch for Git changes or Git merges, and trigger a redeploy of your application. So that’s probably another trick a PaaS like Heroku would use to update your application when the code changes.
#!/bin/bash
while true; do
git pull
sleep 10
doneAbove is a simple example of a tiny sidecar container that would watch for changes on Github and automatically re-deploy the code for your application. This sidecar container could be attached to every application running on a PaaS like Heroku.
Aaron Francis also has a Sidecar package for Laravel that lets you deploy and run code on AWS Lambdas, in various languages besides PHP, and I assume the naming is intentional as each one of these would spin up a tiny container to augment your application code.
The next pattern the author introduces is the Ambassador pattern, which can be used to work with multiple shards. This pattern could be used if you had a cluster of elastic search engines, for example. You could have an ambassador in the middle that the client talks to, and the ambassador would route the search request to the appropriate shard in the elastic cluster. This is something I’ve done before on fly.io and I had multiple ambassadors which the user would be routed to according to their geographic location. Each ambassador would then select the appropriate elastic search shard from the available ones online at the moment and fetch the search result from there. This approach made it easier to take elastic search engine shards online or offline, or do updates on them without interrupting the search engine on the client side.
The next pattern the author talks about is the Adapter pattern, which simply translates interfaces, for example from your internal service to a vendor service… so it seems similar to the adapter design pattern. The main example in this chapter is to use the Adapter pattern to handle logging from multiple application services and translate them into a format that’s consistent.
Moving on to the availability attribute of CAP theorem, three-nines (99.9%) availability means you only get 1.4 minutes of downtime per day. That includes time when you might be updating the server or rolling out updates. If you do continuous updates and deploy once an hour, you only get 3.6 seconds per update… so using a load-balancer with a blue-green deployment is a safe way to maintain uptime. The most difficult part of rapid deployments and keeping the system online is of course cache invalidation.
Moving on to more complex processing, the scatter/gather pattern is one suggestion to split work into smaller chunks and spread it out to shards for computation, then using a root node to combine the results. This is one way to achieve higher parallelism, but I think it might introduce too much complexity to the system. It would be a difficult trade-off to make. I think in the example given for search it makes sense, because if you distribute the search request to multiple shards and one shard fails to respond in time – you can still show the most relevant search results gracefully, rather than sending a failed response.
Another approach to using scattering the author hasn’t mentioned in the book is when you scatter a request to multiple geographically distributed leaf nodes, and take the first response, which ensures you always get the fastest response for each user request, while the slower responses are dropped.
The more interesting concept to think about when designing distributed systems is the messaging system between servers, using RPCs or event driven systems… which I think is the more difficult part of the equation, especially when it comes to having multiple environments or replaying events when debugging.
Work queues are another method of managing batch processing or parallelism, which allows workers to be scaled up or down. The queue acts as the Ambassador and assigns work to other containers, as well as managing the backlog and distributing work. Using queues is fairly typical for working on a Laravel application to speed up the responses you send back to users, and offloading long-running tasks to the queue, such as generating PDFs or sending emails.
Following on the topic of queues, the next natural pattern is workflows which uses queues to manage a series of discrete actions.
The book was a fairly simple overview of topics on managing servers, nothing particularly interesting or novel about managing distributed systems or how different software architecture solutions deal with CAP theorem. There was nothing mentioned on database connectivity, database migrations, approaches for error handling, or the different types of queues and workflows that could be used. It also didn’t cover anything on local-first software which is another interesting consideration of CAP theorem, if users can work with CRDTs offline, and sync when connectivity is available. It provides a good high-level overview but it seems like most of the content was covered by common knowledge.