Skip to content

Observability

Prometheus metrics, health checks, and the info endpoint exposed by the Portwing agent.

Portwing ships a zero-dependency Prometheus metrics endpoint plus explicit liveness and readiness routes in both standard and edge modes. No instrumentation library is embedded — the agent writes the exposition format directly.

Metrics endpoint

Metrics are available at two paths; both serve identical output:

PathPurpose
/_portwing/metricsCanonical path. Prefixed portwing_ and container_ metrics only.
/metricsAlias for compatibility with default Prometheus scrape configs.

In standard mode both paths require a valid Bearer token. In edge mode the local operations listener exposes /health, /ready, /_portwing/health, /metrics, and /_portwing/audit/export; keep that listener on a private container or pod network because the metrics and audit-export routes have no inbound credential surface. The audit export can contain client addresses, request paths, stack names, and container identifiers, so restrict listener access to the monitoring/exporter workload rather than the whole cluster. The response content-type is text/plain; version=0.0.4; charset=utf-8 (Prometheus text exposition format 0.0.4).

Prometheus scrape config

scrape_configs:
  - job_name: portwing
    scheme: https          # use http if TLS is not configured
    static_configs:
      - targets: ["your-host:3000"]
    authorization:
      type: Bearer
      credentials: YOUR_PORTWING_TOKEN
    tls_config:
      # ca_file: /etc/prometheus/portwing-ca.crt  # uncomment if using a custom CA
      insecure_skip_verify: false

If you scrape multiple agents, give each a distinct job_name or add a relabel_configs block to attach a host label so time series from different agents stay identifiable in your dashboards.

Edge-mode scrape config on the same private container or pod network:

scrape_configs:
  - job_name: portwing-edge
    scheme: http
    static_configs:
      - targets: ["portwing-edge:3000"]

Exported metrics

Agent metrics

These metrics are always present and use the portwing_ prefix.

MetricTypeDescription
portwing_build_infogaugeAlways 1. Carries a version label with the agent version string.
portwing_uptime_secondsgaugeSeconds elapsed since the agent process started.
portwing_host_cpu_usage_percentgaugeHost CPU usage percentage (delta between consecutive /proc/stat samples). Returns 0 on the first scrape — a second sample is needed for a meaningful value.
portwing_host_memory_total_bytesgaugeTotal host memory in bytes (from /proc/meminfo MemTotal).
portwing_host_memory_used_bytesgaugeUsed host memory in bytes (MemTotal minus available).
portwing_host_disk_total_bytesgaugeTotal disk capacity in bytes on the Docker data-root filesystem.
portwing_host_disk_used_bytesgaugeUsed disk space in bytes on the Docker data-root filesystem.
portwing_host_network_receive_bytes_totalcounterCumulative bytes received across all non-loopback host interfaces (from /proc/net/dev).
portwing_host_network_transmit_bytes_totalcounterCumulative bytes transmitted across all non-loopback host interfaces.

Disk metrics (portwing_host_disk_total_bytes, portwing_host_disk_used_bytes) reflect the filesystem that contains the Docker data root — typically /var/lib/docker. They are omitted when the agent is started with disk collection disabled.

Per-container metrics

These metrics are emitted once per running container and carry id, name, and image labels. They are absent from the output if no containers are running or if Docker is unreachable at scrape time.

MetricTypeLabelsDescription
container_cpu_usage_seconds_totalcounterid, name, imageCumulative CPU time consumed by the container, in seconds (derived from the Docker stats cpu_stats.cpu_usage.total_usage nanosecond counter).
container_memory_usage_bytesgaugeid, name, imageCurrent memory usage of the container in bytes.
container_spec_memory_limit_bytesgaugeid, name, imageMemory limit configured for the container. Omitted for containers with no memory limit set.
container_network_receive_bytes_totalcounterid, name, imageCumulative bytes received by the container across all its network interfaces.
container_network_transmit_bytes_totalcounterid, name, imageCumulative bytes transmitted by the container across all its network interfaces.

The container_ prefix is intentionally compatible with cAdvisor label conventions so the metrics slot into existing Grafana dashboards without relabelling.

Request metrics

These metrics track HTTP request activity and are the most operationally relevant signals for latency alerting and security monitoring.

MetricTypeLabelsDescription
portwing_http_requests_totalcountermethod, codeTotal HTTP requests completed, by method and response status code.
portwing_http_request_duration_secondshistogramHTTP request latency in seconds. Exposes _bucket, _sum, and _count series.
portwing_http_requests_in_flightgaugeNumber of HTTP requests currently being processed.
portwing_auth_failures_totalcounterreasonTotal authentication failures, labelled by failure reason: bad_token, no_credentials, or (for Ed25519 signature failures) timestamp-skew, replay, unknown-key, invalid-signature.
portwing_rate_limited_totalcounterTotal requests blocked by the IP rate limiter.

Sample lines:

portwing_http_requests_total{method="POST",code="200"} 42
portwing_auth_failures_total{reason="bad_token"} 3

Edge connection and audit metrics

These series make controller connectivity, reconnect churn, slow-consumer backpressure, and audit exporter health directly alertable.

MetricTypeModeDescription
portwing_edge_controller_connectedgaugeedge1 while the controller WebSocket is connected, otherwise 0.
portwing_edge_reconnects_totalcounteredgeReconnect attempts scheduled after a controller disconnect or dial failure.
portwing_edge_backpressure_events_totalcounteredgeConnections evicted because the bounded outbound frame queue filled.
portwing_audit_buffer_recordsgaugebothRecords currently retained by the in-memory audit ring.
portwing_audit_buffer_capacitygaugebothConfigured audit ring capacity.
portwing_audit_sink_enabledgaugeboth1 when a file/stdout/stderr audit sink is configured.
portwing_audit_exports_total{outcome}counterbothJSON/NDJSON export requests split into success and error.

Recommended alerts include sustained portwing_edge_controller_connected == 0, a reconnect rate above the expected maintenance baseline, any increase in backpressure events, and an audit buffer that remains at capacity while the exporter error counter rises.

Sample output

# HELP portwing_build_info Portwing agent build metadata.
# TYPE portwing_build_info gauge
portwing_build_info{version="0.9.1"} 1
# HELP portwing_uptime_seconds Seconds since the agent started.
# TYPE portwing_uptime_seconds gauge
portwing_uptime_seconds 3842.7
# HELP portwing_host_cpu_usage_percent Host CPU usage percentage.
# TYPE portwing_host_cpu_usage_percent gauge
portwing_host_cpu_usage_percent 4.2
# HELP portwing_host_memory_total_bytes Host total memory in bytes.
# TYPE portwing_host_memory_total_bytes gauge
portwing_host_memory_total_bytes 8589934592
# HELP portwing_host_memory_used_bytes Host used memory in bytes.
# TYPE portwing_host_memory_used_bytes gauge
portwing_host_memory_used_bytes 3145728000
# HELP portwing_host_disk_total_bytes Host total disk space in bytes.
# TYPE portwing_host_disk_total_bytes gauge
portwing_host_disk_total_bytes 107374182400
# HELP portwing_host_disk_used_bytes Host used disk space in bytes.
# TYPE portwing_host_disk_used_bytes gauge
portwing_host_disk_used_bytes 32212254720
# HELP portwing_host_network_receive_bytes_total Host network bytes received (all non-lo interfaces).
# TYPE portwing_host_network_receive_bytes_total counter
portwing_host_network_receive_bytes_total 1073741824
# HELP portwing_host_network_transmit_bytes_total Host network bytes transmitted (all non-lo interfaces).
# TYPE portwing_host_network_transmit_bytes_total counter
portwing_host_network_transmit_bytes_total 536870912
# HELP container_cpu_usage_seconds_total Cumulative CPU time consumed by the container in seconds.
# TYPE container_cpu_usage_seconds_total counter
container_cpu_usage_seconds_total{id="a1b2c3d4e5f6",name="myapp",image="myapp:latest"} 120.5
# HELP container_memory_usage_bytes Current memory usage of the container in bytes.
# TYPE container_memory_usage_bytes gauge
container_memory_usage_bytes{id="a1b2c3d4e5f6",name="myapp",image="myapp:latest"} 134217728

Liveness, readiness, and info endpoints

GET /health

No authentication required. Intended for Docker/Kubernetes liveness probes. It does not contact Docker or the edge controller, so a dependency outage cannot put the process into a restart loop.

curl http://your-host:3000/health

Returns HTTP 200 with the full operational shape:

{"status":"ok","live":true,"ready":false,"mode":"standard","version":"0.9.1","uptimeSeconds":3842.7,"docker":"unknown","controller":"not_applicable"}

GET /ready

No authentication required. In standard mode, readiness requires Docker. In edge mode, readiness requires both Docker and the Drydock controller WebSocket. It returns HTTP 503 until every required dependency is connected.

curl http://your-host:3000/ready

Standard-mode ready response:

{"status":"healthy","live":true,"ready":true,"mode":"standard","version":"0.9.1","uptimeSeconds":3842.7,"docker":"connected","controller":"not_applicable"}

Edge-mode not-ready response:

{"status":"unhealthy","live":true,"ready":false,"mode":"edge","version":"0.9.1","uptimeSeconds":42.1,"docker":"connected","controller":"disconnected"}

GET /_portwing/health remains as a compatibility alias for /ready.

The container image's built-in healthcheck uses /health and automatically selects HTTP or HTTPS from TLS_CERT, while the Kubernetes examples use /health for liveness and /ready for readiness.

GET /_portwing/info

Bearer auth required. Returns agent version, Docker version, connection mode, uptime, hostname, and the capability list.

curl -H "Authorization: Bearer YOUR_PORTWING_TOKEN" \
  https://your-host:3000/_portwing/info

/_portwing/info is useful for verifying that a newly deployed agent is running the expected version before wiring it into Drydock.

  • Audit Logging — structured audit events complement these time-series metrics for the security side of observability.
  • API Reference — full OpenAPI spec for all endpoints including metrics and health.
  • Configuration — token configuration options referenced in the scrape config above.
  • Deployment examples — pinned Prometheus and Fluent Bit topologies for Compose and Kubernetes.