Observability for Backend Developers: Logs, Metrics and Traces

Every developer knows this situation. A client says: "The app was slow this morning and some payments failed." You open the logs. You see thousands of lines. None of them explains what happened.

The problem is not that you have no data. The problem is that you have the wrong data, or the data is impossible to connect.

Observability is the ability to understand what is happening inside your system by looking at what it produces from the outside. A good definition I like: observability lets you answer questions about your system that you didn't know you would need to ask.

In this post I explain the three main pillars (logs, metrics and traces), how they work together, and how I apply them in backend projects, including features with AI.

Monitoring vs observability#

These two words are often used as synonyms, but there is a difference.

• Monitoring answers known questions: "Is the CPU above 80%?", "Is the error rate above 1%?". You decide the questions before the problem happens.
• Observability lets you investigate unknown problems: "Why are payments from one specific client slow only when they use the mobile app?".

You need both. Monitoring tells you that something is wrong. Observability helps you discover why.

Pillar 1: Logs#

A log is a record of an event: "Transfer 123 failed because the balance was not enough."

Use structured logs#

The most important change you can make is to stop writing logs as free text and start writing structured logs (usually JSON).

Bad:

[2026-10-23 10:15:02] ERROR: Transfer failed for user 42, amount 120.00

Good:

{
  "timestamp": "2026-10-23T10:15:02Z",
  "level": "error",
  "message": "transfer_failed",
  "user_id": 42,
  "transfer_id": "tr_123",
  "amount_cents": 12000,
  "reason": "insufficient_balance",
  "trace_id": "4bf92f3577b34da6",
  "service": "wallet-api"
}

With structured logs, you can search and filter: "Show me all transfer_failed with reason = insufficient_balance in the last hour, grouped by user." With free text, you can only use grep and hope.

In PHP, Monolog has a JSON formatter. In Python, structlog is excellent. In Laravel and Symfony, it is a configuration change.

Log levels#

Use levels with a clear meaning:

• ERROR: something failed and needs attention.
• WARNING: something unusual happened, but the system handled it (for example, a retry that worked).
• INFO: important business events (user registered, payment completed).
• DEBUG: details for development. Usually disabled in production.

If everything is an ERROR, nothing is an ERROR.

What not to log#

• Passwords, tokens, API keys. Never.
• Personal data that you don't need: full names, emails, bank account numbers, ID numbers. In fintech and with GDPR, this is critical. Use IDs instead.
• Full request bodies by default. They often contain sensitive data.

Pillar 2: Metrics#

A metric is a number measured over time: requests per second, error rate, response time, queue size.

Metrics are cheap to store and fast to query. They are perfect for dashboards and alerts.

The four golden signals#

Google's SRE book proposes four signals that every service should measure. They are a great starting point:

1. Latency: how long requests take. Use percentiles (p50, p95, p99), not only the average. The average hides the users who wait 10 seconds.
2. Traffic: how many requests per second.
3. Errors: how many requests fail (for example, the percentage of 5xx responses).
4. Saturation: how full your resources are (CPU, memory, database connections, queue length).

Business metrics#

Technical metrics are not enough. Also measure what matters for the business:

• Payments completed per minute.
• Failed logins.
• New sign-ups.

Sometimes the technical metrics look perfect, but payments drop to zero because of a bug in a third-party integration. A business metric shows the problem immediately.

Be careful with cardinality#

A common mistake: adding labels with too many different values to metrics, like user_id or transfer_id. This creates millions of time series and your metrics system becomes slow and expensive. Use labels with few values (status, endpoint, region). Put the IDs in logs and traces.

Pillar 3: Traces#

In a modern system, one user request can go through many places: an API gateway, an API, a database, a queue, a worker, an external payment provider, an LLM.

A distributed trace shows the full path of one request through all these components, with the time of each step.

POST /transfers                              820 ms
├── validate request                           3 ms
├── SELECT wallet (PostgreSQL)                12 ms
├── call fraud-check service                 610 ms   <-- the problem
│   └── call external risk API               590 ms
├── UPDATE wallets                            18 ms
└── publish TransferCompleted (Service Bus)   25 ms

With this trace, you see in one second that the problem is the external risk API. Without traces, you can spend hours guessing.

Each step is called a span. All the spans of a request share the same trace ID.

The trace ID connects everything#

This is the key idea that connects the three pillars:

• Put the trace_id in every log line.
• Return the trace_id in error responses of your API (as I recommend in REST API Design Best Practices).
• Pass the trace context in messages that you send to queues.

Now a client can send you a trace_id from an error message, and you can see the full trace and all the logs of that exact request.

OpenTelemetry: the standard#

Some years ago, each vendor (Datadog, New Relic, Elastic...) had its own libraries. Changing vendor meant changing code everywhere.

Today there is a standard: OpenTelemetry (OTel). It gives you:

• SDKs for many languages, including PHP, Python, Java, Go and Node.
• Automatic instrumentation for common libraries: HTTP clients, databases, frameworks.
• A standard format to send logs, metrics and traces.
• The OpenTelemetry Collector, which receives data and sends it to any backend.

You instrument your code once, and you can send the data to Grafana, Jaeger, Datadog, Honeycomb or any other tool. My advice: start with OpenTelemetry, even for small projects.

A simple local stack for learning: OpenTelemetry Collector + Prometheus (metrics) + Loki (logs) + Tempo or Jaeger (traces) + Grafana (dashboards). Everything runs with Docker Compose.

Alerts: less is more#

A dashboard is useful only if somebody looks at it. Alerts are what wake you up. Some rules I learned the hard way:

• Alert on symptoms, not causes. "Error rate above 2% for 5 minutes" is a good alert. "CPU at 85%" is usually not: high CPU without user impact is not an emergency.
• Every alert must be actionable. If you receive an alert and you do nothing, delete the alert or change it.
• Too many alerts = no alerts. People start to ignore them. This is called alert fatigue.
• Use SLOs (Service Level Objectives) when you are ready: "99.5% of transfers complete in under 1 second". Alert when you are spending your error budget too fast.

Observability for AI features#

AI features need everything above, plus some specific things. For every LLM call, I record:

• Model and prompt version.
• Input and output tokens (that means cost).
• Latency, and the time to first token for streaming.
• Errors, retries and fallbacks.
• The feature name, to know which feature spends the money.

And I create spans for each step: retrieval, LLM call, validation. In a RAG system, a trace shows if the slow part is the vector search or the model.

But remember: an HTTP 200 from the LLM doesn't mean the answer was good. Technical observability doesn't measure quality. For that you need evals, which I explain in Testing AI Features with Evals.

There is also the other direction: AI can help you with observability. This is exactly what I built with Scout. It checks the metrics, finds the failing code and asks a model to explain the problem. Good observability data makes tools like Scout much better: structured logs and traces give the model clear context instead of noise.

How to start in an existing project#

You don't need to do everything in one week. My usual order:

1. Switch to structured JSON logs. Add service, environment and a request ID.
2. Add the four golden signals for your main API.
3. Add OpenTelemetry auto-instrumentation for HTTP and the database.
4. Put the trace_id in logs and in error responses.
5. Create two or three good alerts on error rate and latency.
6. Add business metrics for the most important flows.
7. Instrument AI calls with tokens, cost, latency and prompt version.

Summary#

• Logs tell you what happened. Make them structured and safe.
• Metrics tell you how much and how fast. Start with the four golden signals.
• Traces tell you where the time goes across services.
• The trace ID connects everything.
• Use OpenTelemetry to avoid vendor lock-in.
• Few, actionable alerts are better than many.
• For AI features, also track tokens, cost, model and prompt version.

The goal is simple: when something breaks at 3 a.m., you want answers in minutes, not hours.