OpenTelemetry¶

Configure W3C trace-context and baggage propagation before you enqueue tasks or start workers:

fn main() {
    pgtask::otel::configure_propagation();

    // Install your OpenTelemetry tracer and meter providers before starting pgtask.
}

pgtask uses the global OpenTelemetry providers. It does not select an exporter or telemetry backend. This keeps the engine compatible with OTLP collectors, managed observability services, and application-specific sampling.

Set service.name, service.version, deployment.environment, and a unique service.instance.id on the OpenTelemetry resource before you start a worker. Configure your tracer and meter providers before constructing Worker, and shut them down when the process exits.

Traces¶

The engine creates these spans:

Trace context is stored in the task headers JSON object. Existing application headers are preserved. The default propagators write traceparent, tracestate, and baggage when those values exist.

Metrics¶

Metric attributes include queue name, task name, transition state, execution outcome, and lease-renewal result. Task identifiers, payloads, results, errors, and idempotency keys are excluded to keep cardinality bounded.

Worker-capacity gauges only use pgtask.queue.name. Configure a stable service.instance.id resource attribute for each process. Backends must retain that resource boundary when aggregating several workers for one queue.

pgtask.queue.ready.tasks counts due tasks supported by the process's registered task names and handler versions. Use it for queue autoscaling. pgtask.queue.unroutable.tasks counts due tasks with no live, non-draining worker that advertises the required capability. Alert when it remains nonzero. Take the maximum across worker instances for both gauges; every replica observes the same durable queue.

Configured concurrency is the hard process limit. Effective concurrency is the current in-memory admission limit. Available slots are max(effective - active, 0). Lowering the effective limit stops new claims and never cancels an active handler. Event-loop lag measures delay beyond the one-second runtime sampling deadline. Lease-renewal age is the oldest active lease age at the renewal sampling point, or zero when the worker has no active lease.

Do not sum concurrency, handler, or slot gauges across time. Use the latest value per service.instance.id, then sum instances when you need queue capacity. Use the maximum event-loop lag and lease-renewal age across instances for alerting.

The admission-limit gauge uses bounded pgtask.admission.decision values of proposed or applied. Reasons are database_unavailable, lease_renewal_late, event_loop_lag, recovery, or manual. The default detector is observe-only. Sustained runtime lag, an unsafe lease-renewal age, or a database failure changes WorkerControl.proposed_concurrency but does not change the effective limit. This makes thresholds measurable before enforcement is enabled.

Set WorkerConfig.overload_protection.enforce only after validating thresholds for your workload. sustained_samples filters transient event-loop lag. recovery_samples requires a stable healthy window before recovery starts. Enforcement halves the effective limit down to minimum_concurrency while overload persists. Healthy samples then add one slot until the configured limit is restored. Active handlers are never cancelled, and lease renewal remains independent of claim admission.

PostgreSQL CPU, memory, connections, locks, WAL, cache, and storage are infrastructure metrics. Collect them with the OpenTelemetry Collector Contrib PostgreSQL receiver. Kubernetes node, pod, and workload metrics belong in a Collector Deployment and DaemonSet with the Kubernetes attributes processor. Keep those collectors outside the pgtask worker so worker failure cannot hide database or cluster health.

Worker health¶

Set WorkerConfig.health_address to serve /livez and /readyz from the dedicated Rust supervisor thread. /livez only proves that the supervisor can respond. /readyz requires open claim admission, database connectivity, a healthy notification listener, and safe lease renewal. Dependency failure removes the worker from readiness without asking Kubernetes to restart a process whose supervisor is healthy.

Worker liveness¶

pgtask.workers.live counts the workers the database still considers live for a queue, and every worker reports it on its heartbeat tick.

That is deliberately not the same as counting the processes reporting metrics. A worker whose heartbeat is failing keeps exporting everything else and stops being live, which is exactly the failure worth alerting on. Compare the two and a divergence means workers are running but the database cannot see them.

pgtask.worker.heartbeats counts attempts with a pgtask.heartbeat.outcome attribute of ok, missing when the registration has gone, or error when the call failed.

Heartbeats are not spans. They are periodic and unparented, so a span for each would add one single-span trace per worker every ten seconds and still not answer how many workers are alive.