etcd Plugin

Basic accessibility observation

Resource overviews show accessible, inaccessible, or access unknown independently of metric sources. Successful access does not attest healthy membership, replication, partitions, or workloads; use domain diagnostics and alerts for those conditions. Existing liveness/readiness fields are domain evidence and do not drive the single core-owned access alarm. Unknown observations do not resolve firing incidents.

The resource overview separates configured connections from discovered addresses. Use View all addresses to inspect and copy full URLs. A Service/LB address is a connection path, not an additional cluster member. Endpoint diagnostics show the address source; Members provides member identities and roles.

The leader value is separate from eight cluster trends arranged four per row. Member comparisons use two columns; request throughput uses three, and peer comparisons retain full width. Existing saved layouts and query definitions remain unchanged.

The leader card translates only observed 1 and 0 values into elected and no-leader states. Missing or unexpected values stay unknown. Counts are shown as integers; scaled byte values, rates, and latency may keep fractional digits. Node totals and worst-node summaries use the aggregation named by each panel.

The etcd plugin registers an etcd cluster as one Konduo resource and exposes member, endpoint, alarm, proposal, MVCC DB size, peer, and network signals to the operator UI.

Highlights

• Uses both HTTP gateway and etcd v3 gRPC client paths for member, status, and alarm evidence.
• Connects member discovery, leader/learner state, endpoint health, HashKV, and NOSPACE alarms to diagnostics.
• Supports Prometheus mapping packs and Konduo-managed metric collection.
• Protects recovery-support actions such as compact, defrag, alarm disarm, snapshot, and leader transfer with confirmation and cooldown controls.

Before Registration

• Prepare etcd client endpoints, TLS/mTLS, basic auth, and timeout settings.
• Set metric_collection_endpoints when /metrics is exposed on a separate listener.
• When Prometheus is the managed collection write target, run Prometheus with --web.enable-remote-write-receiver.
• In gRPC-only environments, only a status-derived subset may be collected; review p99 latency and histogram panel coverage.

Operations Tips

• Distinguish client URLs from peer URLs when reading member evidence.
• Treat fallback evidence as partial when gateway discovery fails.
• Confirm revision, target endpoint, and operator intent before running defrag or compact actions.

Operator Views

• The dashboard keeps 26 stable catalog panels and selects 17 for a fresh layout. The overview highlights leader state, proposal delay/failure, DB capacity, and disk latency; member comparisons and proposal/gRPC/peer traffic use separate detail subpages.
• The binary leader signal is OK for 1 and FAIL for 0. It remains a compact current-status card, while aggregate trends use four columns and labeled member comparisons use half width.
• Less common aggregate, member, and peer panels remain available through personalization. Existing saved visibility, row placement, order, and size take precedence, and the reorganization does not change query scope, units, filters, sampling, or aggregation.
• The peer detail row starts collapsed. Query and failure evidence continues to run, so a collection or query failure is not hidden by that initial state.
• The member view shows member id, name, client and peer URLs, leader/learner state, and endpoint health together.
• The alarm view exposes cluster alarms such as NOSPACE and helps decide whether disarm is appropriate.
• Diagnostics combine HashKV, proposals, DB size, quota, and peer/client latency evidence for consistency and capacity risk.
• Snapshot, compact, defrag, and leader-transfer actions are recovery helpers that run through explicit confirmation.

Diagnostics And Alerts

• etcd availability should be read through endpoint health, member quorum, leader state, and alarm state together.
• Alert rules cover etcd-domain risks such as leader loss, proposal failures, DB quota pressure, peer/client latency, and NOSPACE alarms.
• Both managed collection and Prometheus mapping packs rely on member-level labels, so endpoint/member label mismatches become partial evidence.
• HashKV differences should be checked with revision, endpoint, and member state before being treated as data corruption.

Management Boundary

• Compact and defrag can affect performance and storage state, so target member and revision must be explicit.
• Alarm disarm should happen after removing the cause; repeatedly disarming NOSPACE is not a recovery plan.
• Snapshot actions provide recovery evidence but do not replace a full backup policy.
• Member add/remove or cluster reconfiguration automation is outside the default CE management scope.

Example Workflows

• For suspected leader instability, inspect leader, learner, endpoint health, and quorum viability first.
• When DB size approaches quota, review compaction revision, defrag candidate members, and NOSPACE alarm state.
• For latency alerts, compare client request latency, peer round-trip, and proposal pending/failure evidence.
• If member HashKV values differ, align revision and endpoint state before retesting, then escalate to upstream etcd procedures when needed.

Prometheus mapping safety: HashKV consistency, learner promotion readiness, snapshot age, and maintenance history require operation evidence that the exporter does not provide. They are not offered as Prometheus logical metrics; resolving an old saved key returns unavailable rather than a constant success or zero age. Use the corresponding live operation/diagnostic evidence instead.

---

etcd Enterprise Overlay

Basic accessibility observation

Resource overviews show accessible, inaccessible, or access unknown independently of metric sources. Successful access does not attest healthy membership, replication, partitions, or workloads; use domain diagnostics and alerts for those conditions. Existing liveness/readiness fields are domain evidence and do not drive the single core-owned access alarm. Unknown observations do not resolve firing incidents.

The resource overview separates configured connections from discovered addresses. Use View all addresses to inspect and copy full URLs. A Service/LB address is a connection path, not an additional cluster member. Endpoint diagnostics show the address source; Members provides member identities and roles. This shared Community behavior also applies to the Enterprise overview.

The leader value is separate from eight cluster trends arranged four per row. Member comparisons use two columns; request throughput uses three, and peer comparisons retain full width. Existing saved layouts and query definitions remain unchanged.

The leader card translates only observed 1 and 0 values into elected and no-leader states. Missing or unexpected values stay unknown. Counts are shown as integers; scaled byte values, rates, and latency may keep fractional digits. Node totals and worst-node summaries use the aggregation named by each panel.

The dashboard retains all 26 catalog panels and selects 17 defaults. Overview keeps leader, proposal, disk latency and DB capacity evidence; member comparisons and traffic details have separate subpages. Labeled member trends use half width, and saved layouts take precedence.

The etcd Enterprise overlay adds MCP descriptors, anomaly rules, and plugin-executed detectors on top of the Community etcd plugin. Its goal is to surface cluster risk earlier by correlating quorum, leader, proposal, WAL, and fsync signals.

Metric interpretation

• The dashboard shows the binary leader signal as OK for 1 and FAIL for 0, alongside the trend panels used to investigate instability.

Highlights

• Exposes etcd resources, monitoring overview, diagnostics, metrics mapping, and anomaly rules through the MCP catalog.
• Plugin-executed anomaly detectors interpret etcd API evidence together with metric evidence.
• Declarative metric rules resolve logical metric keys through a Prometheus metric source.
• Simulator fixtures cover leader instability, quorum risk, disk latency, and proposal backlog scenarios.

Before Registration

• Community etcd connectivity and TLS settings must be healthy.
• Anomaly analysis needs a Prometheus metric source and mapping pack linkage.
• MCP users need read scopes and resource access.

Operations Tips

• Interpret leader changes, proposal backlog, and fsync latency together with quorum state instead of as isolated signals.
• When plugin detector and declarative metric results differ, check metric-source coverage and etcd API permissions first.

Operational Scenarios

• During control-plane incidents, inspect leader stability, quorum member health, and proposal apply backlog together to judge write-path risk.
• When disk latency rises, separate WAL fsync and backend commit delay to distinguish storage bottlenecks from consensus bottlenecks.
• MCP catalogs give incident-analysis agents a read-only entry point for etcd state, metric coverage, and anomaly rules.

Boundaries

• Quorum interpretation and anomaly metadata stay in the EE overlay; shared etcd connectivity and base health behavior stay in the CE plugin.
• Plugin-mode detectors run only for rules that need etcd API evidence. Metric-threshold rules are evaluated by the core declarative evaluator.

Prometheus mapping safety: HashKV consistency, learner promotion readiness, snapshot age, and maintenance history require operation evidence that the exporter does not provide. They are not offered as Prometheus logical metrics; resolving an old saved key returns unavailable rather than a constant success or zero age. Use the corresponding live operation/diagnostic evidence instead.

The Enterprise anomaly catalog no longer registers Member Divergence, Learner Stuck, Backup Staleness, or Maintenance Aftereffect patterns: their required operation evidence is not available as a Prometheus metric. Learner membership alone does not prove failed catch-up or promotion readiness. HashKV inspection, member/learner operations, snapshot inspection, maintenance history, and their diagnostic/MCP routes remain available; use those observed results for investigation.

Saved overrides cannot reactivate these removed built-in rules. Previously firing states remain historical observations with no new evaluation; removal does not automatically mark them resolved or prove recovery. Check their observation time and current operation evidence when reviewing an older state.

MCP diagnostic execution effects

The run_diagnostics tool changes diagnostic execution and cooldown state. Its descriptor declares operation_class: diagnostic_lifecycle; MCP advertises readOnlyHint=false, destructiveHint=false and idempotentHint=false. Diagnostic queries retain read-only hints. Both queries and actions continue to require mcp:invoke and the existing resource permissions/audit checks. Safety hints do not change authorization or guarantee safe retries.

MCP tool input contracts

Alarm inventory, hash-KV inspection, failover planning, static catalogs, and the diagnostic refresh action accept {}. Summary/history and monitoring expose the metric source and lookback inputs they consume. Member and topology tools expose topology_host_map_json, a JSON object mapping members to host resource instances. Dynamic metric selector and request aliases remain supported on their owning routes.

The advertised JSON Schema is checked before plugin dispatch. Omitted optional values retain plugin defaults. Arguments do not replace resource-instance authorization; stored connection credentials are supplied by Konduo. Open schemas are retained only for existing dynamic filters or aliases, and each advertised property describes its caller-facing meaning. Plugin result shapes can vary with product/version, available evidence, and partial failures; no stable outputSchema is promised for these dynamic responses. Client and resource coverage rules are defined in enterprise/backend/docs/plugin-contract.md in the Enterprise source repository.