Validator Monitoring: Signing, Node Health and Incident Response

A running process is not enough

A validator host can be online while the validator is no longer participating as expected. The node may be behind the chain, disconnected from its signer or unable to communicate with enough peers. Monitoring should therefore answer two questions: is the validator performing its consensus role, and can the team respond before the situation becomes more serious?

This guide focuses on Cosmos SDK networks using CometBFT. Metric names, slashing rules and operational commands depend on the chain and software version.

Watch participation and chain progress

Track signing activity, the validator's active or jailed status, and its position relative to the chain's downtime rules. Read the actual parameters from the target network rather than applying a threshold copied from another chain.

Compare the node's height with an independent reference. A single difference is not necessarily an incident: nodes can be observed between block updates. Look for a persistent gap and investigate whether the reference itself is healthy.

Monitor peer count and connectivity against the intended topology. A validator connected through two sentries needs different expectations from a public RPC node with many peers. An alert should identify which role is affected.

Connect host metrics to the workload

Collect disk capacity and latency, memory pressure, CPU load and network errors. These signals explain why a node may be struggling, but none proves that it is signing correctly.

A useful alert combines a symptom with context. For example, if node height stops advancing while an independent reference continues to move, the runbook can direct the operator to check peers, storage and recent changes. If both stop, investigate the network before treating it as an individual-node failure.

Protect monitoring endpoints from public exposure. CometBFT's metrics documentation describes available instrumentation; confirm names and availability against the version actually deployed.

Monitor the monitor

An alerting system that depends entirely on the same host or provider as the validator can disappear during the incident it should report. Use an independent observation path where practical, and alert when metrics stop arriving rather than interpreting missing data as a healthy zero.

Periodically test delivery to the people responsible for responding. A dashboard nobody sees during an incident is not an escalation plan.

Give each alert a next action

For a signing or freshness alert, begin with the chain's condition, then inspect node synchronization, peers, signer connectivity and recent changes. Preserve relevant logs before restarting services.

If a signer reports a state conflict, treat that as a reason to investigate. Deleting signing state or starting another signer to silence an alert can create a more serious problem. Follow the chain- and signer-specific recovery procedure.

DTEAM can discuss dedicated validator and supporting node infrastructure with monitoring responsibilities agreed for the deployment. Send us the network, node roles and escalation requirements so we can scope the infrastructure around them.