Observability

Emit traces and metrics from noop to stdout to an OTLP collector with one provider.

observability gives traces and metrics in every environment. One Provider hands out scoped Tracer and Meter handles with typed Attr values (String, Int, Int64, Float64, Bool). It never wires a global SDK behind your back: set_globals stays opt-in. Switching backends is a config change, not a rewrite of Span or Attr calls.

Quick start

stdout (the default) prints spans and metrics locally. Register the adapter, then trace an operation:

import (
    "context"
    "github.com/zenta-dev/zever/adapters/observability/stdout"
    "github.com/zenta-dev/zever/config"
    "github.com/zenta-dev/zever/container"
    "github.com/zenta-dev/zever/core/observability"
)

stdout.Register() // container wires nothing
c := container.New(config.Default())
p, err := c.Observability()
if err != nil {
    panic(err)
}
defer p.Shutdown(context.Background())

ctx, span := p.Tracer("bookings").Start(context.Background(), "booking.create")
span.SetAttributes(observability.String("space_id", "room-1"))
span.End()
p.Meter("bookings").Counter(ctx, "bookings_created", 1)

Expected output: span lines in human-readable form.

Export to a collector

Use otlp to reach a real collector. Replicas all funnel into one collector keyed by service_name.

OBSERVABILITY_ADAPTER=otlp
OBSERVABILITY_SERVICE_NAME=bookings
OBSERVABILITY_ENDPOINT=collector:4317
OBSERVABILITY_SAMPLE_RATIO=0.1
OBSERVABILITY_HEADERS_X-TENANT=acme

Env names follow the usual scheme: OBSERVABILITY_ADAPTER, OBSERVABILITY_SERVICENAME, OBSERVABILITY_ENDPOINT, OBSERVABILITY_SAMPLERATIO, OBSERVABILITY_ATTRVALUELIMIT.

Validate requires service_name of 1 to 128 characters with no control characters, endpoint as host:port, insecure only for loopback, cert_file and key_file together, at most 32 headers, and sample_ratio in [0,1]. insecure plus an authorization header is rejected. Other keys: set_globals, verbose, ca_file. Adapters are noop for tests, stdout (default) for local visibility, and otlp for production. See the observability config reference.

Metrics

Zever does not serve a Prometheus /metrics endpoint. Metrics (for example the outbox.* relay counters) leave the process over OTLP together with the traces, so run a collector and let it expose or forward them: the OpenTelemetry Collector’s prometheus exporter gives Prometheus a scrape target, and prometheusremotewrite pushes instead. See the outbox operations guide for the metric names and alert rules.

Profiling with pprof

The generated server can serve Go runtime profiles, off by default. Set --pprof-addr (or PPROF_ADDR) to turn it on:

PPROF_ADDR=127.0.0.1:6060 ./server        # or: ./server --pprof-addr 127.0.0.1:6060
go tool pprof http://127.0.0.1:6060/debug/pprof/heap
curl -o cpu.pb.gz 'http://127.0.0.1:6060/debug/pprof/profile?seconds=30'

Profiles are served on their own listener, never on the public router, and the address must be loopback (127.0.0.1:PORT or localhost:PORT); a wildcard or public host fails startup. Reach it from outside the host through an SSH tunnel or kubectl port-forward pod/<name> 6060. The server logs a warning when it is on. Endpoints: the index, any named runtime profile (heap, goroutine, allocs, block, mutex, threadcreate, with ?debug=1 for text), profile (CPU) and trace (both take ?seconds=N, 1 to 300). cmdline is not served because argv can hold secrets.

Code that builds its own server can use container.StartPprof(addr, logger) directly; the handler uses a private mux and never registers on http.DefaultServeMux.

Bound and redact attributes

Keys cap at 256 bytes (MaxKeyLen), string values at 4096 by default (MaxValueLen), and spans at 32 attributes (MaxAttrs). attr_value_limit tunes the value cap (0 means default, otherwise 256 to 16384). Sensitive keys (password, secret, token, api_key, auth, cookie, session, private_key) export as [redacted], but never attach a secret to a span at all. Name spans after the operation (queue.dispatch), not the adapter, and keep high-cardinality values such as user IDs out of metric labels.

Production checklist

Providers declare Shutdown(ctx), not Close. c.Close(ctx) probes it so buffered spans flush. Always shut down with a timeout-bearing context.
Set observability.adapter: otlp in production; the default stdout keeps traces on the box.
Never log raw option maps. Use cfg.RedactedServices().

Troubleshooting

service_name must be 1-128 characters

The name is empty or too long. Set a short stable name such as bookings; it keys every dashboard.

insecure requires a loopback endpoint

insecure: true points at a non-loopback collector. Use TLS, or point at localhost for local collectors only.

Attributes are silently truncated

A string over attr_value_limit (default 4096) or more than 32 attributes per span is cut by design. Send IDs and counts, not blobs.

Scheduler and jobsSpans around job runs.

TenantsTenant as a span attribute.

CacheCache hit counters as metrics.

Cross-Service TracingInbound extraction, outbound injection, baggage, semconv names.