Scenario: a server, then a whole zone, goes down
15 outages to analyse: single points of failure, health checks, database failover, zones, async replication, cascading failures, retry storms, deploys, certificates, DNS, stuck queues, backups, alerting. The cause, then the fix that makes the service resilient.
- 36 cards
- 3 topics
- Free
The outline, topic by topic
Each topic is weighted the way the exam weighs it.
- Redundancy
- Failure propagation
- Operations
A few cards from the deck
One server dies, the whole site dies
- The site runs on a single application server
- Its motherboard fails at 3 am
- The site is down until the team arrives
What is the most likely cause?- A single point of failure
- A cyberattack
- A traffic overload
- A DNS error
Show the answer
A single point of failure
When one component carries everything, its failure stops everything.
One server dies, the whole site dies
Cause: a single point of failure.
What is the typical solution?- Several servers behind a load balancer that checks their health
- A sturdier server
- A more responsive night on-call
- More frequent backups
Show the answer
Several servers behind a load balancer that checks their health
If a server dies, the load balancer removes it and the others take the load, with no intervention.
Alternative: A standby server ready to take over the main one's address (active-passive).
Pitfall: A single load balancer becomes the point of failure in turn: it must be doubled too.One request in three fails for 2 minutes
- One of the three servers crashed
- The load balancer keeps sending it traffic
- It only removes it after 2 minutes
What is the most likely cause?- The load balancer is overloaded
- Health checks are missing or too far apart
- The database is down
- DNS points to the wrong server
Show the answer
Health checks are missing or too far apart
The load balancer does not know a server is dead until it tests it.