Observability
Prometheus metrics, health checks, and the info endpoint exposed by the Portwing agent.
Portwing ships a zero-dependency Prometheus metrics endpoint plus explicit liveness and readiness routes in both standard and edge modes. No instrumentation library is embedded — the agent writes the exposition format directly.
Metrics endpoint
Metrics are available at two paths; both serve identical output:
| Path | Purpose |
|---|---|
/_portwing/metrics | Canonical path. Prefixed portwing_ and container_ metrics only. |
/metrics | Alias for compatibility with default Prometheus scrape configs. |
In standard mode both paths require a valid Bearer token. In edge mode the
local operations listener exposes /health, /ready, /_portwing/health,
/metrics, and /_portwing/audit/export; keep that listener on a private
container or pod network because the metrics and audit-export routes have no
inbound credential surface. The audit export can contain client addresses,
request paths, stack names, and container identifiers, so restrict listener
access to the monitoring/exporter workload rather than the whole cluster.
The response content-type is text/plain; version=0.0.4; charset=utf-8
(Prometheus text exposition format 0.0.4).
Prometheus scrape config
scrape_configs:
- job_name: portwing
scheme: https # use http if TLS is not configured
static_configs:
- targets: ["your-host:3000"]
authorization:
type: Bearer
credentials: YOUR_PORTWING_TOKEN
tls_config:
# ca_file: /etc/prometheus/portwing-ca.crt # uncomment if using a custom CA
insecure_skip_verify: falseIf you scrape multiple agents, give each a distinct job_name or add a relabel_configs block to attach a host label so time series from different agents stay identifiable in your dashboards.
Edge-mode scrape config on the same private container or pod network:
scrape_configs:
- job_name: portwing-edge
scheme: http
static_configs:
- targets: ["portwing-edge:3000"]Exported metrics
Agent metrics
These metrics are always present and use the portwing_ prefix.
| Metric | Type | Description |
|---|---|---|
portwing_build_info | gauge | Always 1. Carries a version label with the agent version string. |
portwing_uptime_seconds | gauge | Seconds elapsed since the agent process started. |
portwing_host_cpu_usage_percent | gauge | Host CPU usage percentage (delta between consecutive /proc/stat samples). Returns 0 on the first scrape — a second sample is needed for a meaningful value. |
portwing_host_memory_total_bytes | gauge | Total host memory in bytes (from /proc/meminfo MemTotal). |
portwing_host_memory_used_bytes | gauge | Used host memory in bytes (MemTotal minus available). |
portwing_host_disk_total_bytes | gauge | Total disk capacity in bytes on the Docker data-root filesystem. |
portwing_host_disk_used_bytes | gauge | Used disk space in bytes on the Docker data-root filesystem. |
portwing_host_network_receive_bytes_total | counter | Cumulative bytes received across all non-loopback host interfaces (from /proc/net/dev). |
portwing_host_network_transmit_bytes_total | counter | Cumulative bytes transmitted across all non-loopback host interfaces. |
Disk metrics (portwing_host_disk_total_bytes, portwing_host_disk_used_bytes) reflect the filesystem that contains the Docker data root — typically /var/lib/docker. They are omitted when the agent is started with disk collection disabled.
Per-container metrics
These metrics are emitted once per running container and carry id, name, and image labels. They are absent from the output if no containers are running or if Docker is unreachable at scrape time.
| Metric | Type | Labels | Description |
|---|---|---|---|
container_cpu_usage_seconds_total | counter | id, name, image | Cumulative CPU time consumed by the container, in seconds (derived from the Docker stats cpu_stats.cpu_usage.total_usage nanosecond counter). |
container_memory_usage_bytes | gauge | id, name, image | Current memory usage of the container in bytes. |
container_spec_memory_limit_bytes | gauge | id, name, image | Memory limit configured for the container. Omitted for containers with no memory limit set. |
container_network_receive_bytes_total | counter | id, name, image | Cumulative bytes received by the container across all its network interfaces. |
container_network_transmit_bytes_total | counter | id, name, image | Cumulative bytes transmitted by the container across all its network interfaces. |
The container_ prefix is intentionally compatible with cAdvisor label conventions so the metrics slot into existing Grafana dashboards without relabelling.
Request metrics
These metrics track HTTP request activity and are the most operationally relevant signals for latency alerting and security monitoring.
| Metric | Type | Labels | Description |
|---|---|---|---|
portwing_http_requests_total | counter | method, code | Total HTTP requests completed, by method and response status code. |
portwing_http_request_duration_seconds | histogram | — | HTTP request latency in seconds. Exposes _bucket, _sum, and _count series. |
portwing_http_requests_in_flight | gauge | — | Number of HTTP requests currently being processed. |
portwing_auth_failures_total | counter | reason | Total authentication failures, labelled by failure reason: bad_token, no_credentials, or (for Ed25519 signature failures) timestamp-skew, replay, unknown-key, invalid-signature. |
portwing_rate_limited_total | counter | — | Total requests blocked by the IP rate limiter. |
Sample lines:
portwing_http_requests_total{method="POST",code="200"} 42
portwing_auth_failures_total{reason="bad_token"} 3Edge connection and audit metrics
These series make controller connectivity, reconnect churn, slow-consumer backpressure, and audit exporter health directly alertable.
| Metric | Type | Mode | Description |
|---|---|---|---|
portwing_edge_controller_connected | gauge | edge | 1 while the controller WebSocket is connected, otherwise 0. |
portwing_edge_reconnects_total | counter | edge | Reconnect attempts scheduled after a controller disconnect or dial failure. |
portwing_edge_backpressure_events_total | counter | edge | Connections evicted because the bounded outbound frame queue filled. |
portwing_audit_buffer_records | gauge | both | Records currently retained by the in-memory audit ring. |
portwing_audit_buffer_capacity | gauge | both | Configured audit ring capacity. |
portwing_audit_sink_enabled | gauge | both | 1 when a file/stdout/stderr audit sink is configured. |
portwing_audit_exports_total{outcome} | counter | both | JSON/NDJSON export requests split into success and error. |
Recommended alerts include sustained
portwing_edge_controller_connected == 0, a reconnect rate above the expected
maintenance baseline, any increase in backpressure events, and an audit buffer
that remains at capacity while the exporter error counter rises.
Sample output
# HELP portwing_build_info Portwing agent build metadata.
# TYPE portwing_build_info gauge
portwing_build_info{version="0.9.1"} 1
# HELP portwing_uptime_seconds Seconds since the agent started.
# TYPE portwing_uptime_seconds gauge
portwing_uptime_seconds 3842.7
# HELP portwing_host_cpu_usage_percent Host CPU usage percentage.
# TYPE portwing_host_cpu_usage_percent gauge
portwing_host_cpu_usage_percent 4.2
# HELP portwing_host_memory_total_bytes Host total memory in bytes.
# TYPE portwing_host_memory_total_bytes gauge
portwing_host_memory_total_bytes 8589934592
# HELP portwing_host_memory_used_bytes Host used memory in bytes.
# TYPE portwing_host_memory_used_bytes gauge
portwing_host_memory_used_bytes 3145728000
# HELP portwing_host_disk_total_bytes Host total disk space in bytes.
# TYPE portwing_host_disk_total_bytes gauge
portwing_host_disk_total_bytes 107374182400
# HELP portwing_host_disk_used_bytes Host used disk space in bytes.
# TYPE portwing_host_disk_used_bytes gauge
portwing_host_disk_used_bytes 32212254720
# HELP portwing_host_network_receive_bytes_total Host network bytes received (all non-lo interfaces).
# TYPE portwing_host_network_receive_bytes_total counter
portwing_host_network_receive_bytes_total 1073741824
# HELP portwing_host_network_transmit_bytes_total Host network bytes transmitted (all non-lo interfaces).
# TYPE portwing_host_network_transmit_bytes_total counter
portwing_host_network_transmit_bytes_total 536870912
# HELP container_cpu_usage_seconds_total Cumulative CPU time consumed by the container in seconds.
# TYPE container_cpu_usage_seconds_total counter
container_cpu_usage_seconds_total{id="a1b2c3d4e5f6",name="myapp",image="myapp:latest"} 120.5
# HELP container_memory_usage_bytes Current memory usage of the container in bytes.
# TYPE container_memory_usage_bytes gauge
container_memory_usage_bytes{id="a1b2c3d4e5f6",name="myapp",image="myapp:latest"} 134217728Liveness, readiness, and info endpoints
GET /health
No authentication required. Intended for Docker/Kubernetes liveness probes. It does not contact Docker or the edge controller, so a dependency outage cannot put the process into a restart loop.
curl http://your-host:3000/healthReturns HTTP 200 with the full operational shape:
{"status":"ok","live":true,"ready":false,"mode":"standard","version":"0.9.1","uptimeSeconds":3842.7,"docker":"unknown","controller":"not_applicable"}GET /ready
No authentication required. In standard mode, readiness requires Docker. In edge mode, readiness requires both Docker and the Drydock controller WebSocket. It returns HTTP 503 until every required dependency is connected.
curl http://your-host:3000/readyStandard-mode ready response:
{"status":"healthy","live":true,"ready":true,"mode":"standard","version":"0.9.1","uptimeSeconds":3842.7,"docker":"connected","controller":"not_applicable"}Edge-mode not-ready response:
{"status":"unhealthy","live":true,"ready":false,"mode":"edge","version":"0.9.1","uptimeSeconds":42.1,"docker":"connected","controller":"disconnected"}GET /_portwing/health remains as a compatibility alias for /ready.
The container image's built-in healthcheck uses /health and automatically
selects HTTP or HTTPS from TLS_CERT, while the Kubernetes examples use
/health for liveness and /ready for readiness.
GET /_portwing/info
Bearer auth required. Returns agent version, Docker version, connection mode, uptime, hostname, and the capability list.
curl -H "Authorization: Bearer YOUR_PORTWING_TOKEN" \
https://your-host:3000/_portwing/info/_portwing/info is useful for verifying that a newly deployed agent is running the expected version before wiring it into Drydock.
Related
- Audit Logging — structured audit events complement these time-series metrics for the security side of observability.
- API Reference — full OpenAPI spec for all endpoints including metrics and health.
- Configuration — token configuration options referenced in the scrape config above.
- Deployment examples — pinned Prometheus and Fluent Bit topologies for Compose and Kubernetes.