Supervisors and Fault Tolerance

Let it crash, then recover

A supervisor is a process that starts child processes and restarts them when they crash, returning the system to a known good state. Each child has a child spec describing how to start it and its restart type: :permanent (always restart), :transient (restart only after abnormal exits) or :temporary (never). The supervisor's strategy decides what happens on a crash: :one_for_one restarts only the failed child; :one_for_all restarts all children, for tightly coupled processes; :rest_for_one restarts the failed child and those started after it. max_restarts and max_seconds stop endless crash loops by escalating the failure to the parent supervisor. Supervisors nest into a supervision tree, with the Application module at the root, started automatically by Mix. DynamicSupervisor starts children on demand, such as one cart per customer, and Registry gives processes names by key. Task.Supervisor supervises short-lived tasks. This structure, rather than defensive try/catch, is how Elixir systems achieve high availability.

An application supervision tree

A Registry, a DynamicSupervisor for carts and a helper to start carts on demand.

defmodule Shop.Application do
  use Application

  @impl true
  def start(_type, _args) do
    children = [
      {Registry, keys: :unique, name: Shop.CartRegistry},
      {DynamicSupervisor, name: Shop.CartSupervisor, strategy: :one_for_one},
      {Task.Supervisor, name: Shop.TaskSupervisor}
    ]

    Supervisor.start_link(children, strategy: :one_for_one, name: Shop.Supervisor)
  end
end

defmodule Shop.Carts do
  def ensure_started(customer_id) do
    spec = %{
      id: {Shop.Cart, customer_id},
      start: {Shop.Cart, :start_link, [customer_id]},
      restart: :transient                     # restart on crash, not on a normal :stop
    }

    case DynamicSupervisor.start_child(Shop.CartSupervisor, spec) do
      {:ok, pid} -> {:ok, pid}
      {:error, {:already_started, pid}} -> {:ok, pid}
      other -> other
    end
  end
end

# In iex -S mix:
# Shop.Carts.ensure_started("c-1")
# Shop.Cart.add_item("c-1", "pen", 2)
# [{pid, _}] = Registry.lookup(Shop.CartRegistry, "c-1")
# Process.exit(pid, :kill)                  # the supervisor restarts the cart
# Shop.Cart.items("c-1")                     # %{} - fresh state after restart

Restart means fresh state

A restarted process starts from init/1, so in-memory state is lost. Keep important state in a database, ETS table or another process, and use the restart to recover from corrupted or unexpected state.

Quick check: With the :one_for_one strategy, what happens when a child crashes?

• Only the crashed child is restarted
• All children restart
• The whole application stops
• Nothing happens