Skip to main content

Metrics Reference

Stealthium turns the telemetry streamed from your hosts into a set of numeric metrics that it exports to your own observability stack. Metrics are pushed on a rolling ~15-second cadence to whichever sink you configure:

  • OpenTelemetry (OTLP) — over HTTP or gRPC to any OTLP-compatible endpoint (e.g. an OpenTelemetry Collector, Axiom).
  • Prometheus — via a Pushgateway (including Grafana Cloud).

This page documents every metric Stealthium exports, what it measures, and why it matters.

Naming and grouping​

Every metric belongs to a domain (gpu, cpu, memory, network, container, session, vm, controlplane) and has a short name within that domain. The exported metric name is the two joined with the stealthium_ prefix:

stealthium_<domain>_<name>

For example, the GPU core-utilization metric is exported as stealthium_gpu_utilization. The domain is the primary grouping — it's how metrics are organized on export and the natural way to build dashboards (all stealthium_gpu_* series for a GPU panel, all stealthium_container_* for containers, and so on). The sections below mirror these domains.

Labels and per-host scope​

Metrics are exported per host: each host becomes its own resource carrying host.name and a stable service.instance.id, all under the stealthium service. On top of that, individual samples carry these labels (dimensions):

LabelMeaning
machine_typeThe class of machine the sample came from.
gpu_idThe GPU's PCI address (e.g. 0000:01:00.0), or a per-link suffix like 0000:01:00.0/link/3 for NVLink metrics. Present only on per-GPU metrics. For controlplane metrics this column instead carries the Kubernetes scrape job (etcd, kube-scheduler, kube-apiserver) — there is no GPU association.
gpu_uuidThe canonical NVML UUID (GPU-…) for that GPU, when known. Present only on per-GPU metrics.

Host-level metrics (counts and rates that aren't tied to one GPU) are emitted with an empty gpu_id/gpu_uuid.

Units​

Units follow the UCUM convention used by OpenTelemetry:

UnitMeaning
1Dimensionless count
%Percentage
CelDegrees Celsius
Cel/sDegrees Celsius per second (rate of change)
WWatts
ByBytes
By/sBytes per second
MHzMegahertz
{event}/sEvents per second
msMilliseconds
sSeconds

All metrics are exported as gauges — each sample is the latest value from a sliding window, not a monotonic total.

GPU metrics​

Exported as stealthium_gpu_*. Most are reported per GPU (labelled with gpu_id/gpu_uuid); counts and rates that describe the whole host are emitted without a GPU label.

stealthium_gpu_utilization​

  • Unit: %

Core (SM) utilization of the GPU, sourced from DCGM.

Why this matters: This is the headline "how busy is the GPU" signal and the first thing to check when workloads run slow or hardware sits idle. Sustained low utilization on an expensive accelerator often points to a pipeline bottleneck elsewhere, while pinned-at-100% utilization helps confirm a GPU is saturated.

stealthium_gpu_memory​

  • Unit: By

GPU memory currently in use, sourced from DCGM.

Why this matters: GPU memory is usually the first resource a workload exhausts, and running out causes out-of-memory crashes rather than graceful slowdown. Tracking it reveals how close jobs run to the limit and helps right-size models and batch sizes.

stealthium_gpu_memory_utilization​

  • Unit: %

Memory-copy utilization (the fraction of time the memory subsystem is busy moving data), sourced from DCGM.

Why this matters: High memory-bandwidth utilization alongside modest core utilization is the classic signature of a memory-bound workload. It tells you whether more compute would actually help or whether the bottleneck is data movement.

stealthium_gpu_temperature​

  • Unit: Cel

GPU die temperature, per GPU.

Why this matters: Temperature is the leading indicator of thermal throttling and long-term hardware wear. Watching it catches cooling problems before they degrade performance or shorten the life of the card.

stealthium_gpu_thermal_slope​

  • Unit: Cel/s

The rate at which a GPU's temperature is changing, per GPU.

Why this matters: A sharp positive slope warns of a runaway thermal event sooner than the absolute temperature crossing a threshold does. It gives you lead time to react before a card hits its throttle or shutdown limit.

stealthium_gpu_power​

  • Unit: W

Instantaneous GPU power draw, sourced from DCGM.

Why this matters: Power draw tracks both real workload intensity and total energy cost, and unexpected draw can reveal hidden or unauthorized compute. Aggregated across a fleet it's also a direct input to datacenter power and cooling planning.

stealthium_gpu_power_headroom​

  • Unit: W

The difference between the GPU's configured power limit and its current draw — i.e. how much power budget remains before the hardware governor clamps the clock.

Why this matters: A shrinking headroom warns that a GPU is approaching its power cap before throttling actually starts. Tracking it alongside raw power draw lets you detect workloads that are consistently power-limited without waiting for the throttle-reason metric to go non-zero.

stealthium_gpu_fan_speed​

  • Unit: %

GPU fan speed as a percentage of maximum, per GPU.

Why this matters: Fan speed is the primary indicator of active thermal management and how hard the cooling system is working. A fan pinned at 100% is a signal the GPU is running hot and may be approaching thermal throttling, while a fan that stops spinning entirely on an active GPU often precedes a hardware failure.

stealthium_gpu_throttle​

  • Unit: 1

NVIDIA throttle-reason bitmask indicating which hardware governor(s) are currently limiting GPU clock speed, per GPU. A value of 0 means no throttling is active.

Why this matters: Throttle reasons translate a performance drop into a root cause — power limit, thermal limit, voltage swing, etc. — without guesswork. A non-zero value tells you why a GPU is underclocking, which you cannot read from clock speed or temperature alone.

stealthium_gpu_clock_speed​

  • Unit: MHz

Streaming-multiprocessor (SM) clock speed, sourced from DCGM.

Why this matters: A clock that drops below its expected boost range is the clearest evidence of thermal or power throttling in action. Correlating it with temperature and power confirms why a GPU is underperforming.

stealthium_gpu_memory_clock​

  • Unit: MHz

Memory clock speed, sourced from DCGM.

Why this matters: Memory-clock throttling throttles memory-bound workloads specifically, separately from the core clock. Tracking it isolates memory-side performance limits from compute-side ones.

  • Unit: 1

PCIe link generation negotiated between the GPU and the host, per GPU (e.g. 4 for PCIe Gen 4).

Why this matters: A GPU negotiating a lower link generation than its slot supports means bandwidth is being left on the table and the GPU-to-CPU data path may be a bottleneck. On stable infrastructure the generation should never change, so any drop is a signal the hardware is mis-seated or the driver has downgraded it.

stealthium_gpu_pcie_throughput​

  • Unit: By/s

Combined PCIe receive + transmit throughput, per GPU.

Why this matters: PCIe bandwidth is the ceiling on how fast the host can feed data to the GPU and read results back. High throughput relative to the link's rated bandwidth pinpoints a PCIe bottleneck that adding more compute cores cannot solve.

stealthium_gpu_encoder_utilization​

  • Unit: %

Hardware video-encoder (NVENC) utilization, sourced from DCGM.

Why this matters: For transcoding and streaming workloads the encoder, not the compute cores, is the resource that saturates first. This metric shows encoder headroom that core utilization alone would miss.

stealthium_gpu_decoder_utilization​

  • Unit: %

Hardware video-decoder (NVDEC) utilization, sourced from DCGM.

Why this matters: Decode-heavy pipelines (e.g. video analytics) bottleneck on the decoder independently of compute. Watching it tells you whether to scale out decode capacity rather than add more GPU compute.

stealthium_gpu_health_score​

  • Unit: 1 (0–1 score)

A computed composite health score for each GPU, where higher is healthier.

Why this matters: It distills temperature, errors, and other signals into one number you can alert and dashboard on without watching every raw metric. A declining score is an early, single-glance warning that a specific card is trending toward trouble.

stealthium_gpu_ecc_errors​

  • Unit: 1

Count of uncorrectable ECC memory errors, per GPU.

Why this matters: Uncorrectable ECC errors corrupt computation and are a strong predictor of imminent hardware failure. Any non-zero value is a reason to drain and inspect the card before it produces silently wrong results.

stealthium_gpu_xid_error_count​

  • Unit: 1

Count of NVIDIA Xid errors observed for a GPU within each reporting window.

Why this matters: Xid errors are the driver's own report of GPU faults — from application bugs to failing hardware — and frequently precede crashes. Tracking their rate helps you tell a one-off application error from a card that's actively failing.

stealthium_gpu_crash_chain_count​

  • Unit: 1 (0 or 1)

A per-GPU indicator that is 1 while a correlated chain of crash-related events is active and 0 otherwise.

Why this matters: It flags GPUs caught in a cascading failure rather than a single isolated error, which is the pattern that takes whole jobs down. Use it to prioritize which card to intervene on first during an incident.

stealthium_gpu_count​

  • Unit: 1

Number of GPUs detected on the host.

Why this matters: A count that drops below the expected number means a GPU has fallen off the bus or the driver has lost it — often before any workload reports an error. It's a cheap, host-level integrity check for your accelerators.

stealthium_gpu_process_count​

  • Unit: 1

Number of processes currently holding GPU access on the host.

Why this matters: Knowing how many processes share each GPU reveals contention and unexpected tenants competing for the same hardware. An unexplained process on a GPU can also be the first sign of unauthorized use.

stealthium_gpu_rpc_gpu_to_cpu_rate​

  • Unit: {event}/s

Rate of GPU→CPU RPC calls.

Why this matters: This captures how chatty the GPU-to-host control path is, which is a low-level driver of overhead in many workloads. Sudden changes can reveal a behavioral shift in what's running on the GPU.

stealthium_gpu_rpc_cpu_to_gpu_rate​

  • Unit: {event}/s

Rate of CPU→GPU RPC calls.

Why this matters: The host-to-GPU command rate reflects how hard the CPU side is driving the accelerator. Anomalous spikes are useful both for performance tuning and for spotting workloads that don't match their expected profile.

stealthium_gpu_context_create_rate​

  • Unit: {event}/s

Rate at which GPU contexts (TSGs/channels — the lowest-level "running on the GPU" primitive) are created.

Why this matters: Context creation is normally steady, so a sustained spike can indicate fuzzing, resource abuse, or a misbehaving application thrashing the GPU. It's a fine-grained observability signal independent of the alerting path.

stealthium_gpu_context_destroy_rate​

  • Unit: {event}/s

Rate at which GPU contexts (TSGs/channels) are destroyed.

Why this matters: Paired with the create rate, it shows whether contexts are churning rapidly or leaking. A large or growing gap between create and destroy rates points to a resource leak on the GPU.

  • Unit: By/s

Per-link NVLink receive bandwidth (the gpu_id label carries a /link/N suffix).

Why this matters: In multi-GPU training, NVLink bandwidth between GPUs is often the real bottleneck rather than per-GPU compute. Per-link visibility helps you spot an underperforming link or imbalanced traffic across the interconnect.

  • Unit: By/s

Per-link NVLink transmit bandwidth (the gpu_id label carries a /link/N suffix).

Why this matters: Transmit bandwidth completes the picture of how data flows between GPUs over NVLink. Asymmetry between TX and RX, or a link well below its peers, points to a topology or hardware problem.

  • Unit: 1 (0–4 enum)

Per-link NVLink state (the gpu_id label carries a /link/N suffix): 0 Inactive, 1 Active, 2 Fault, 3 Sleep, 4 Recovery.

Why this matters: A link that isn't Active is unavailable for GPU-to-GPU traffic, which silently degrades multi-GPU workloads to a slower fallback path instead of failing outright. Fault or Recovery on a link that should be steady is the clearest sign of a failing NVLink connection or cable.

  • Unit: {event}/s

Per-link rate of data-link-layer CRC/retransmit errors on NVLink (the gpu_id label carries a /link/N suffix), from NVML's cumulative DL_CRC_DATA counter.

Why this matters: A rising CRC error rate on a link means data is being corrupted in transit and retransmitted, which both slows GPU-to-GPU communication and points to a physical-layer problem (cabling, connector, or hardware fault) on that specific link before it degrades further.

  • Unit: {event}/s

Per-link rate of NVLink recovery events (the gpu_id label carries a /link/N suffix) — how often the link has to re-train itself back to an active state.

Why this matters: Occasional recoveries are normal, but a climbing rate means a link is repeatedly dropping and re-establishing itself, which stalls the collective operations that depend on it. It's an early warning for a link headed toward a persistent fault.

  • Unit: {event}/s

Per-link rate of uncorrected flit-level CRC errors on NVLink (the gpu_id label carries a /link/N suffix), from NVML's cumulative DL_CRC_FLIT counter.

Why this matters: Uncorrected flit errors are a finer-grained view of link integrity than the data-layer CRC counter, catching corruption the retransmit path didn't fully absorb. A sustained rate here, alongside CRC errors and recoveries, confirms a link is degrading rather than seeing a one-off glitch.

stealthium_gpu_uvm_allocation_rate​

  • Unit: {event}/s

Rate of NVIDIA Unified Virtual Memory (UVM) allocation events, aggregating user- and kernel-initiated allocations.

Why this matters: UVM allocation churn drives page-migration overhead that can quietly dominate the runtime of managed-memory workloads. Tracking it surfaces memory-management pressure that GPU utilization alone hides.

stealthium_gpu_uvm_free_rate​

  • Unit: {event}/s

Rate of NVIDIA UVM free events.

Why this matters: Compared against the allocation rate, it shows whether managed memory is being released in step with allocation. A persistent imbalance is an early sign of a memory leak in a UVM workload.

stealthium_gpu_uvm_eviction_rate​

  • Unit: {event}/s

Rate of NVIDIA UVM eviction (EvictStart) events — i.e. attempted evictions.

Why this matters: Frequent evictions mean the GPU is thrashing managed memory between device and host, which devastates performance. A rising eviction rate is a direct signal that a workload's memory footprint exceeds what fits on the GPU.

stealthium_gpu_uvm_bytes_allocated_rate​

  • Unit: By/s

Rate of bytes allocated through NVIDIA UVM.

Why this matters: The byte rate quantifies UVM pressure in capacity terms that the event rate can't, since allocation sizes vary widely. It's the better signal for correlating managed-memory activity with actual GPU memory consumption.

stealthium_gpu_uvm_bytes_evicted_rate​

  • Unit: By/s

Rate of bytes evicted by NVIDIA UVM.

Why this matters: This measures the real volume of data being shuttled off the GPU under memory pressure, which is what actually consumes PCIe bandwidth and stalls the workload. Sustained high values confirm a workload is memory-overcommitted.

stealthium_gpu_gpu_sm_active​

  • Unit: %

GPU-wide streaming-multiprocessor (SM) active percentage from NVIDIA's GPU Performance Monitoring (GPM) API, averaged over the reporting window, per GPU.

Why this matters: SM-active from GPM is a higher-fidelity compute signal than the DCGM utilization rate: it measures the fraction of time at least one warp is active on each SM, making it sensitive to memory stalls that the coarser utilization metric averages away. Comparing it with tensor-active isolates compute-bound versus tensor-bound workloads.

stealthium_gpu_gpu_tensor_active​

  • Unit: %

GPU-wide any-tensor-active percentage from NVIDIA GPM, averaged over the reporting window, per GPU. Reports the fraction of time any tensor engine is executing.

Why this matters: Tensor-core utilization is the key signal for AI/ML workloads: a large gap between SM-active and tensor-active means the GPU has compute capacity but is not keeping its tensor engines fed. Closing that gap through better kernel selection or data movement typically yields the largest throughput gains.

stealthium_gpu_gpu_dram_bandwidth​

  • Unit: %

GPU DRAM bandwidth utilization from NVIDIA GPM, averaged over the reporting window, per GPU — the fraction of theoretical peak DRAM bandwidth in use.

Why this matters: DRAM bandwidth saturation is the most common bottleneck in large-model inference and training: when this metric approaches 100%, adding more tensor-core occupancy will not improve throughput. It provides a clear memory-side ceiling signal distinct from both compute utilization and memory capacity.

stealthium_gpu_cuda_kernel_launch_rate​

  • Unit: {event}/s

Rate of CUDA kernel launches across all processes on the GPU, per GPU.

Why this matters: Kernel launch rate is a direct measure of how aggressively work is being submitted to the GPU. An unexpectedly high rate can indicate fine-grained launches that waste CPU-side overhead on dispatch, while a sudden drop may signal a workload stall or crash before any other metric reflects it.

CPU & process metrics​

Exported as stealthium_cpu_*. These describe host process activity; load and utilization are reported from guest VMs.

stealthium_cpu_process_count​

  • Unit: 1

Number of running processes on the host.

Why this matters: A baseline process count makes runaway process growth and fork bombs immediately visible. Sudden jumps are also a common side effect of misconfigured jobs or compromise.

stealthium_cpu_process_fork_rate​

  • Unit: {event}/s

Rate of process creation (fork) events.

Why this matters: A spike in fork rate is a classic signature of fork bombs, crash-loops, and certain malware. It catches abusive process churn that a steady process count would average away.

stealthium_cpu_process_exit_rate​

  • Unit: {event}/s

Rate of process exit events.

Why this matters: Read alongside the fork rate, it shows whether processes are short-lived churn or accumulating. A high exit rate often signals a crash-looping service.

stealthium_cpu_process_exec_rate​

  • Unit: {event}/s

Rate of program execution (exec) events.

Why this matters: Each exec is a new program starting, so a burst of execs can indicate script-driven activity or a compromise spawning tools. It's a key behavioral signal for what's actually running on the host.

stealthium_cpu_scheduling_rate​

  • Unit: {event}/s

Rate of process scheduling events.

Why this matters: Scheduling activity reflects how heavily contended the CPU is across runnable work. Unusually high rates point to context-switch thrashing that wastes CPU on overhead rather than useful work.

stealthium_cpu_unique_binary_count​

  • Unit: 1

Cumulative number of distinct binaries executed during the session.

Why this matters: A host's normal workload uses a stable, fairly small set of binaries, so growth in unique binaries is a strong anomaly signal. A jump frequently indicates an attacker dropping and running new tools.

stealthium_cpu_load​

  • Unit: 1

1-minute load average, reported from guest VMs.

Why this matters: Load average is the standard at-a-glance measure of how overcommitted a machine is relative to its CPUs. Surfacing it from guest VMs gives visibility into tenants you don't otherwise instrument directly.

stealthium_cpu_utilization​

  • Unit: %

1-minute load average expressed as a percentage, reported from guest VMs.

Why this matters: Normalizing load to a percentage makes it comparable across machines with different core counts. It's the simplest single number for spotting saturated guest VMs.

Memory metrics​

Exported as stealthium_memory_*. System-wide host memory.

stealthium_memory_available​

  • Unit: By

Memory available for new allocations.

Why this matters: Available memory is the truest measure of headroom before the kernel starts reclaiming or killing processes. Watching it predicts out-of-memory events before they happen.

stealthium_memory_utilization​

  • Unit: %

Memory in use as a percentage of total.

Why this matters: A normalized utilization figure is the easiest threshold to alert on across heterogeneous hosts. Sustained high utilization is the precursor to OOM kills and swap-driven slowdowns.

stealthium_memory_cache​

  • Unit: By

Memory used by the page cache.

Why this matters: Cache is reclaimable, so distinguishing it from genuinely used memory prevents false "out of memory" alarms. A collapsing cache under load also explains sudden I/O slowdowns as the system loses its read cache.

stealthium_memory_rss_total​

  • Unit: By

Total resident set size summed across all processes.

Why this matters: This is the memory genuinely pinned by running processes, the part that can't simply be reclaimed. Steady growth is the signature of a memory leak somewhere on the host.

stealthium_memory_pressure​

  • Unit: %

Memory pressure from the kernel's PSI some_avg10 indicator.

Why this matters: PSI pressure captures time actually lost stalling on memory, which catches contention that raw usage percentages miss. Rising pressure is an early, direct warning that memory is hurting performance.

Network metrics​

Exported as stealthium_network_*. Host- and process-level network activity.

stealthium_network_bandwidth​

  • Unit: By/s

Combined transmit + receive bandwidth, smoothed over a short sliding window.

Why this matters: Total throughput is the baseline for spotting both saturated links and anomalous data movement such as exfiltration. A sudden sustained climb with no matching workload change warrants investigation.

stealthium_network_interface_count​

  • Unit: 1

Number of network interfaces on the host.

Why this matters: A changing interface count can reveal new tunnels, virtual interfaces, or VPNs appearing on a host. On stable infrastructure the count should rarely move, so changes are worth a look.

stealthium_network_dns_query_rate​

  • Unit: {event}/s

Rate of DNS queries.

Why this matters: DNS query rate and patterns are a well-known channel for command-and-control and data exfiltration (DNS tunneling). An unexplained surge in queries is a high-value security signal.

stealthium_network_socket_event_rate​

  • Unit: {event}/s

Rate of socket operations.

Why this matters: Socket churn reflects how actively the host is opening and closing connections, which scanning and beaconing behavior inflates. It complements bandwidth by catching activity that moves little data but many connections.

stealthium_network_http_request_rate​

  • Unit: {event}/s

Rate of HTTP requests.

Why this matters: HTTP request volume tracks application traffic and can expose automated abuse or beaconing to web endpoints. Correlated with latency, it also shows whether a service is under load.

stealthium_network_http_response_latency​

  • Unit: ms

Latency between matched HTTP request/response pairs.

Why this matters: Response latency is a direct measure of the experience a service is delivering to its callers. Rising tail latency is an early warning of an overloaded or degrading dependency.

stealthium_network_utilization​

  • Unit: By

Total bytes observed, broken down by protocol (TCP, UDP, ICMP, …).

Why this matters: A per-protocol byte breakdown shows the shape of network traffic, not just its volume, so unusual protocol mixes stand out. A surge in an otherwise-quiet protocol is a classic anomaly indicator.

stealthium_network_usage​

  • Units: By (per-process byte totals) and 1 (connection / port-activity counts)

A general network-usage metric emitted from several angles: total bytes attributed to processes (By), and connection/port-activity counts (1). Use the unit to distinguish the variant.

Why this matters: Together these show both how much data and how many connections a host is generating, which is the combination that reveals beaconing, scanning, and bulk transfers. Per-process byte attribution also helps pin unusual traffic to a specific culprit.

Container metrics​

Exported as stealthium_container_*. Container population and per-container activity rates.

stealthium_container_container_count​

  • Unit: 1

Number of running containers on the host.

Why this matters: The container count is the baseline for the host's workload density, making unexpected containers immediately visible. An unexplained new container is a common first sign of compromise or misconfiguration.

stealthium_container_gpu_container_count​

  • Unit: 1

Number of running containers that have GPU access.

Why this matters: GPU access is a privileged, expensive capability, so knowing exactly which containers hold it is important for both cost and security. A container unexpectedly gaining GPU access is worth investigating.

stealthium_container_mps_container_count​

  • Unit: 1

Number of running containers using NVIDIA Multi-Process Service (MPS).

Why this matters: MPS lets multiple containers share a GPU concurrently, which affects both performance isolation and security boundaries. Tracking MPS usage shows how aggressively GPUs are being shared.

stealthium_container_container_process_count​

  • Unit: 1

Total number of processes running across all containers.

Why this matters: This reveals how much work is packed inside containers versus on the host directly. Sudden growth can indicate a container fork bomb or a breakout-style escalation.

stealthium_container_container_churn_rate​

  • Unit: {event}/s

Rate of container lifecycle events (starts, stops, etc.).

Why this matters: Healthy clusters have a fairly stable container population, so high churn signals crash-looping workloads or an orchestration problem. Rapid start/stop cycling can also be an attacker probing the environment.

stealthium_container_container_start_rate​

  • Unit: {event}/s

Rate of container start events.

Why this matters: Start rate isolates the creation half of container churn, which is relevant both for autoscaling audits and for detecting rapid container re-spawning (crash loops or an attacker trying to establish a persistent foothold). It complements the stop rate to give directional clarity on lifecycle activity.

stealthium_container_container_stop_rate​

  • Unit: {event}/s

Rate of container stop events.

Why this matters: A spike in stop rate without a matching start rate means containers are dying faster than they are being replaced, which points to a systemic failure or a targeted shutdown. Tracking it separately from churn makes it easy to distinguish graceful scale-down from crash-driven teardown.

stealthium_container_cgroup_event_rate​

  • Unit: {event}/s

Rate of cgroup events.

Why this matters: cgroup activity underlies container resource management, so its rate reflects container creation, teardown, and resource reconfiguration. Anomalous rates can point to instability in the container runtime.

stealthium_container_container_socket_ops_rate​

  • Unit: {event}/s

Rate of socket operations attributed to containers.

Why this matters: Per-container socket activity isolates which container is driving network connections. That attribution is what lets you trace suspicious network behavior back to the responsible workload.

stealthium_container_container_dns_rate​

  • Unit: {event}/s

Rate of DNS queries attributed to containers.

Why this matters: Containerized workloads are a common origin of DNS-based C2 and exfiltration, and per-container attribution pinpoints the source. It turns a host-level DNS anomaly into an actionable "which container" answer.

stealthium_container_container_http_rate​

  • Unit: {event}/s

Rate of HTTP requests attributed to containers.

Why this matters: Per-container HTTP rates show which workloads are talking to web services and how heavily. A container suddenly making many requests can indicate abuse or a compromised image phoning home.

stealthium_container_container_file_ops_rate​

  • Unit: {event}/s

Rate of file operations attributed to containers.

Why this matters: Bursts of file activity are characteristic of ransomware, crypto-mining, and data staging inside a container. Per-container attribution makes that behavior traceable to a single workload.

stealthium_container_container_exec_rate​

  • Unit: {event}/s

Rate of process execution (exec) events attributed to containers.

Why this matters: Execs inside a container reveal what's actually being launched there, and unexpected execs are a strong indicator of interactive access or a breakout attempt. It's a key signal for detecting hands-on-keyboard activity in a container.

Session & system metrics​

Exported as stealthium_session_*. Session-wide and kernel/system-level signals.

stealthium_session_hyperprint_count​

  • Unit: 1

Number of telemetry events (hyperprints) observed in each reporting window.

Why this matters: This is the volume of raw telemetry the host is producing, a proxy for how active and how well-instrumented it is. A sudden drop can mean the agent or host has gone quiet, while a spike signals a burst of activity worth examining.

stealthium_session_ebpf_program_count​

  • Unit: 1

Cumulative number of distinct eBPF programs loaded during the session.

Why this matters: eBPF programs run in the kernel with deep visibility and power, and attackers increasingly abuse them for stealthy rootkits. Growth in loaded eBPF programs beyond your known set is a high-signal security event.

stealthium_session_kernel_module_count​

  • Unit: 1

Number of kernel modules currently loaded.

Why this matters: Loadable kernel modules can grant total control of a host, so an unexpected module is among the most serious indicators of compromise. Tracking the count surfaces rootkits and unauthorized drivers.

stealthium_session_namespace_creation_rate​

  • Unit: {event}/s

Rate of namespace creation events.

Why this matters: Namespace creation underlies containerization and sandboxing, so its rate reflects how often isolated environments are being spun up. Anomalous spikes can indicate container escapes or sandboxing abuse.

stealthium_session_kmod_load_rate​

  • Unit: {event}/s

Rate of kernel module load events.

Why this matters: Where the module count shows state, the load rate catches the act of loading in real time. Any module load on a stable host is worth scrutiny, and a burst is a strong red flag.

stealthium_session_login_event_rate​

  • Unit: {event}/s

Rate of login events.

Why this matters: Login activity is a front-line authentication signal, and spikes in failed or unusual logins are a hallmark of brute-force and lateral movement. Tracking the rate makes credential-based attacks visible.

VM metrics​

Exported as stealthium_vm_*.

stealthium_vm_vm_count​

  • Unit: 1

Number of registered guest VMs.

Why this matters: The guest-VM count is the baseline for virtualized workload density on the host. An unexpected change can reveal VMs being created or destroyed outside your normal provisioning flow.

Kubernetes control-plane metrics​

Exported as stealthium_controlplane_*. These land an allowlisted set of Kubernetes control-plane /metrics (etcd, kube-scheduler, kube-apiserver) as Orion metrics, scraped roughly every 30 seconds. Unlike the domains above, these aren't per-GPU or per-host in the usual sense: the gpu_id label carries the scrape job (etcd, kube-scheduler, or kube-apiserver) instead of a PCI address, and each value is summed or rate-computed across whatever Prometheus label combinations the underlying series had — Orion's metric model has no extra label dimension to preserve them. apiserver series are available on both managed and self-managed clusters via the in-cluster scrape; etcd and kube-scheduler series are self-managed clusters only (managed offerings like EKS/GKE/AKS don't expose those components' /metrics).

stealthium_controlplane_etcd_db_size​

  • Unit: By

Size of etcd's backing database file, scraped from kube-apiserver's apiserver_storage_size_bytes (Kubernetes' available proxy for etcd DB size).

Why this matters: etcd rejects writes once its database fills up to its configured quota, which stalls every control-plane operation across the whole cluster. Watching this size gives lead time to compact or resize etcd before that happens.

stealthium_controlplane_etcd_has_leader​

  • Unit: 1 (0 or 1)

Whether etcd currently has a leader, from etcd_server_has_leader. 1 means a leader is elected; 0 means it isn't.

Why this matters: etcd can't accept writes without a leader, so a value of 0 means the control plane has lost quorum and can't make progress. It's the most direct, earliest signal of an etcd availability incident.

stealthium_controlplane_scheduler_pending_pods​

  • Unit: 1

Number of pods waiting to be scheduled, summed across kube-scheduler's queues, from scheduler_pending_pods.

Why this matters: A growing backlog of pending pods means the scheduler can't place new work fast enough, whether from resource exhaustion, misconfigured constraints, or scheduler overload. It's the earliest sign that new workloads are stalling before they ever start running.

stealthium_controlplane_apiserver_inflight_requests​

  • Unit: 1

Number of requests kube-apiserver is currently processing, summed across request kinds, from apiserver_current_inflight_requests.

Why this matters: Inflight request count shows how close the apiserver is to its concurrency limits before it starts shedding load via API Priority & Fairness. A sustained climb is an early warning of control-plane saturation, ahead of any requests actually being rejected.

stealthium_controlplane_apiserver_request_rate​

  • Unit: {event}/s

Rate of requests handled by kube-apiserver, derived from the windowed delta of apiserver_request_total.

Why this matters: Request rate is the baseline traffic volume hitting the control plane, so an unexplained spike can point to a runaway client, a misbehaving controller loop, or unusual automation. It's also the context you need to interpret the apiserver latency and error-rate signals alongside it.

stealthium_controlplane_apiserver_request_latency_p99​

  • Unit: s

p99 latency of kube-apiserver requests, computed as a windowed quantile over apiserver_request_duration_seconds_bucket. Long-lived WATCH/CONNECT requests are excluded since their duration isn't a meaningful latency signal.

Why this matters: Rising p99 latency is the most direct sign that the Kubernetes API is becoming slow to respond, which cascades into slow deployments, autoscaling, and every controller that depends on the API. Excluding long-lived streaming verbs keeps the signal focused on genuine request-handling latency rather than connections that are expected to stay open.