A minimal observability stack

15 September 2026 · 5 min read

Observability tooling has a gravity problem. Every component you add is easy to justify in isolation and hard to justify in aggregate, and six months later you are maintaining a distributed system whose only job is to tell you whether your distributed system is working.

For anything up to a few dozen hosts, three components cover almost everything:

• Metrics for the shape of things over time.
• Logs for the specific thing that happened.
• One dashboard you actually look at, with numbers you can compare against last week.

What I actually run

node_exporter on every box, Prometheus scraping every fifteen seconds, Loki for logs, Grafana in front. That is four processes and about 300 MB of RSS for a five-node setup, which is less than the JVM heap of some agents I have been asked to install.

scrape_configs:
  - job_name: node
    static_configs:
      - targets:
          - 10.0.0.11:9100
          - 10.0.0.12:9100
          - 10.0.0.13:9100
    relabel_configs:
      - source_labels: [__address__]
        regex: '([^:]+):.*'
        target_label: instance

The four alerts worth having

Alerting is where minimalism pays off fastest. Every alert you cannot act on is a page you will learn to ignore. I keep four:

Alert	Condition	Why
Disk filling	predict_linear(node_filesystem_avail[6h], 86400) < 0	Gives a day of warning instead of an hour
Certificate expiry	< 14 days remaining	Renewal failures are silent until they are not
Target down	up == 0 for 5m	The only alert that catches everything at once
Memory pressure	available < 8% for 10m	OOM killer arrives without warning otherwise

Everything else goes on the dashboard and nobody gets woken up for it. If a condition is worth a page, it is worth writing down what the responder should do. If you cannot write that sentence, it is not an alert — it is a graph.

Retention

Thirty days of metrics, seven days of logs, both on the same disk. People ask for a year and then look at the last six hours. If you need a year, you need it for a specific question, and that question deserves its own pipeline rather than a blanket retention policy that quietly fills the disk.

The dashboard that gets used is the one with four panels, not the one with forty.

What I deliberately skip

No tracing. No service mesh. No anomaly detection. No hosted APM. Each of these is correct for some scale, and none of them is correct for five nodes where the failure modes are already known. The value of the small stack is not that it is cheap — it is that when something breaks, the tooling is not one of the suspects.

If the setup ever outgrows this, the migration is a config change, not a rewrite. That is the point of keeping it small: you get to choose when to grow.