# Observability

Emit traces and metrics from noop to stdout to an OTLP collector with one provider.

`observability` gives traces and metrics in every environment. One `Provider` hands out scoped `Tracer` and `Meter` handles with typed `Attr` values (`String`, `Int`, `Int64`, `Float64`, `Bool`). It never wires a global SDK behind your back: `set_globals` stays opt-in. Switching backends is a config change, not a rewrite of `Span` or `Attr` calls.

## Quick start

`stdout` (the default) prints spans and metrics locally. Register the adapter, then trace an operation:

```go title="main.go"
import (
    "context"
    "github.com/zenta-dev/zever/adapters/observability/stdout"
    "github.com/zenta-dev/zever/config"
    "github.com/zenta-dev/zever/container"
    "github.com/zenta-dev/zever/core/observability"
)

stdout.Register() // container wires nothing
c := container.New(config.Default())
p, err := c.Observability()
if err != nil {
    panic(err)
}
defer p.Shutdown(context.Background())

ctx, span := p.Tracer("bookings").Start(context.Background(), "booking.create")
span.SetAttributes(observability.String("space_id", "room-1"))
span.End()
p.Meter("bookings").Counter(ctx, "bookings_created", 1)
```

Expected output: span lines in human-readable form.

## Export to a collector

Use `otlp` to reach a real collector. Replicas all funnel into one collector keyed by `service_name`.

```sh
OBSERVABILITY_ADAPTER=otlp
OBSERVABILITY_SERVICE_NAME=bookings
OBSERVABILITY_ENDPOINT=collector:4317
OBSERVABILITY_SAMPLE_RATIO=0.1
OBSERVABILITY_HEADERS_X-TENANT=acme
```

Env names follow the usual scheme: `OBSERVABILITY_ADAPTER`, `OBSERVABILITY_SERVICENAME`, `OBSERVABILITY_ENDPOINT`, `OBSERVABILITY_SAMPLERATIO`, `OBSERVABILITY_ATTRVALUELIMIT`.

`Validate` requires `service_name` of 1 to 128 characters with no control characters, `endpoint` as `host:port`, `insecure` only for loopback, `cert_file` and `key_file` together, at most 32 headers, and `sample_ratio` in [0,1]. `insecure` plus an `authorization` header is rejected. Other keys: `set_globals`, `verbose`, `ca_file`. Adapters are `noop` for tests, `stdout` (default) for local visibility, and `otlp` for production. See the [observability config reference](https://zenta-dev.github.io/zever/reference/config/#observability).

## Metrics

Zever does not serve a Prometheus `/metrics` endpoint. Metrics (for example the `outbox.*` relay counters) leave the process over OTLP together with the traces, so run a collector and let it expose or forward them: the OpenTelemetry Collector's `prometheus` exporter gives Prometheus a scrape target, and `prometheusremotewrite` pushes instead. See the [outbox operations guide](https://zenta-dev.github.io/zever/guides/digging-deeper/outbox-ops/) for the metric names and alert rules.

## Profiling with pprof

The generated server can serve Go runtime profiles, off by default. Set `--pprof-addr` (or `PPROF_ADDR`) to turn it on:

```sh
PPROF_ADDR=127.0.0.1:6060 ./server        # or: ./server --pprof-addr 127.0.0.1:6060
go tool pprof http://127.0.0.1:6060/debug/pprof/heap
curl -o cpu.pb.gz 'http://127.0.0.1:6060/debug/pprof/profile?seconds=30'
```

Profiles are served on their own listener, never on the public router, and the address must be loopback (`127.0.0.1:PORT` or `localhost:PORT`); a wildcard or public host fails startup. Reach it from outside the host through an SSH tunnel or `kubectl port-forward pod/<name> 6060`. The server logs a warning when it is on. Endpoints: the index, any named runtime profile (`heap`, `goroutine`, `allocs`, `block`, `mutex`, `threadcreate`, with `?debug=1` for text), `profile` (CPU) and `trace` (both take `?seconds=N`, 1 to 300). `cmdline` is not served because argv can hold secrets.

Code that builds its own server can use `container.StartPprof(addr, logger)` directly; the handler uses a private mux and never registers on `http.DefaultServeMux`.

## Bound and redact attributes

Keys cap at 256 bytes (`MaxKeyLen`), string values at 4096 by default (`MaxValueLen`), and spans at 32 attributes (`MaxAttrs`). `attr_value_limit` tunes the value cap (0 means default, otherwise 256 to 16384). Sensitive keys (password, secret, token, api_key, auth, cookie, session, private_key) export as `[redacted]`, but never attach a secret to a span at all. Name spans after the operation (`queue.dispatch`), not the adapter, and keep high-cardinality values such as user IDs out of metric labels.

## Production checklist

- Providers declare `Shutdown(ctx)`, not `Close`. `c.Close(ctx)` probes it so buffered spans flush. Always shut down with a timeout-bearing context.
- Set `observability.adapter: otlp` in production; the default `stdout` keeps traces on the box.
- Never log raw option maps. Use `cfg.RedactedServices()`.

## Troubleshooting

<details>
<summary>`service_name must be 1-128 characters`</summary>

The name is empty or too long. Set a short stable name such as `bookings`; it keys every dashboard.
</details>

<details>
<summary>`insecure requires a loopback endpoint`</summary>

`insecure: true` points at a non-loopback collector. Use TLS, or point at localhost for local collectors only.
</details>

<details>
<summary>Attributes are silently truncated</summary>

A string over `attr_value_limit` (default 4096) or more than 32 attributes per span is cut by design. Send IDs and counts, not blobs.
</details>

## Related

- [Scheduler and jobs](https://zenta-dev.github.io/zever/guides/background/scheduler-jobs/): Spans around job runs.
- [Tenants](https://zenta-dev.github.io/zever/guides/operate/tenants/): Tenant as a span attribute.
- [Cache](https://zenta-dev.github.io/zever/guides/data/cache/): Cache hit counters as metrics.
- [Cross-Service Tracing](https://zenta-dev.github.io/zever/guides/operate/observability/cross-service-tracing/): Inbound extraction, outbound injection, baggage, semconv names.