RustFS Plugin

The RustFS plugin provides read-only visibility into RustFS readiness, nodes, drives, capacity, requests, object scanning, and background healing.

Highlights

• Distinguishes liveness from readiness with RustFS health and admin APIs.
• Reads node, drive, pool/set placement, and software version evidence.
• Uses a linked Prometheus resource for capacity, request, process, and scanner metrics.
• Presents structured evidence for storage, scanner, and background-heal diagnostics.
• Provides managed alert rules for capacity, drives, request latency, and process restarts.

Before Registration

1. Prepare a RustFS S3/API address reachable from the Konduo backend. The typical API port is 9000.
2. To use diagnostics, topology, and software inventory, prepare an access key and secret key that can read /rustfs/admin/v3/info, /storageinfo, /scanner/status, and /background-heal/status.
3. Configure Prometheus to collect RustFS metrics through an OpenTelemetry Collector or another supported collection path.
4. Prepare a Prometheus resource in Konduo that can be linked to the RustFS resource as its metric source.
5. For self-signed certificates, prefer distributing a trusted CA. Skip certificate verification only in a controlled test environment.

Settings

Field	Required	Default	Description
endpoint	yes	http://localhost:9000	RustFS S3/API base address reachable from the Konduo backend
timeout_seconds	no	10	Per-request limit from 1 to 60 seconds
access_key	conditional	none	Access key for read-only admin APIs; configure with secret_key
secret_key	conditional	none	Secret paired with the access key
region	no	us-east-1	Region used in the AWS Signature Version 4 credential scope
tls_insecure_skip_verify	no	false	Disables HTTPS server certificate verification

localhost refers to the host or container running the Konduo backend. A URL that works in a browser can still fail when it is not reachable from the backend network.

Connection And Version Compatibility

Configure access_key and secret_key together. If they are omitted, the public /health/ready probe is used, with a /health fallback for 1.0.0-alpha.83. Without admin credentials, admin-backed diagnostics, topology, and software inventory are unavailable.

Grant the key only the permissions needed to read these routes:

• GET /rustfs/admin/v3/info
• GET /rustfs/admin/v3/storageinfo
• GET /rustfs/admin/v3/scanner/status
• POST /rustfs/admin/v3/background-heal/status

The background-heal status API is read-only but uses POST in the RustFS protocol. The plugin never calls the manual-heal start API. If exact permission names are not established by the local documentation, do not invent policy actions. Confirm the actions required for these routes in the target RustFS release documentation.

Alpha.83 /health proves endpoint liveness but does not return a dedicated readiness result. With admin credentials, the plugin uses online node state from /info as additional readiness evidence. Without credentials, liveness can be healthy while readiness remains unknown.

The plugin accepts both direct JSON payloads returned by 1.0.0-alpha.83 and the info-wrapped responses used by later releases. Alpha.83 does not expose scanner status and its background-heal status handler is not implemented, so those diagnostics remain unavailable until RustFS is upgraded. Validate connection, diagnostics, and metrics against the exact RustFS release before production use.

The region is part of the SigV4 credential scope. A mismatch with the region expected by RustFS or its front proxy can produce 403 even with the correct keys. Also verify system clocks, key state, request host, and proxy path rewrites.

Monitoring Setup

Export RustFS observability metrics to a Prometheus-compatible source and link that prometheus-plugin instance to the RustFS resource. The v4 mapping in rustfs-plugin-prometheus-pack-v1 supports both current RustFS metric names and the OTLP names emitted by 1.0.0-alpha.83.

When the OpenTelemetry Collector Prometheus exporter is configured with namespace: rustfs, the alpha.83 native rustfs_* instruments are exposed as rustfs_rustfs_*. The v4 mapping handles both forms, but avoiding an unnecessary duplicate prefix makes new configurations easier to inspect.

A Collector exporter such as :8889/metrics is a Prometheus scrape target, not a Prometheus query API. Configure Prometheus to scrape it, then link that Prometheus instance through prometheus-plugin. The admin /rustfs/admin/v3/metrics response is not used as a Prometheus text scrape endpoint.

Alpha.83 exports capacity, bucket/object, request-rate, and request-latency metrics, but does not export drive-health or scanner metrics. Missing series are not converted to zero; dependent panels and alerts show no data.

Registration And Validation

1. Enter the endpoint, request timeout, and security settings for the RustFS resource.
2. If admin-backed features are required, enter the access key, secret key, and signing region together.
3. Run Connection Test. On failure, inspect the response status and backend failure classification before repeatedly saving the same settings.
4. Save the resource after the test succeeds, then confirm that the last-check and next-refresh times advance on the summary page.
5. On alpha.83, if readiness is unknown, add admin credentials and confirm that online-node evidence is read from /info.
6. In Prometheus, confirm that the Collector or RustFS reporter target is UP.
7. Link that Prometheus resource to RustFS and confirm that capacity and runtime panels contain current samples.
8. In storage diagnostics, compare node/drive placement and capacity with the admin API evidence.
9. Review managed alerts and tune thresholds to expansion lead time and latency objectives.

A successful connection test, a successful health probe, and successful Prometheus collection validate different paths. Do not use one result as proof that the other two paths work.

Operations View

• Summary: key metrics and action-oriented checks for readiness, nodes, capacity, drives, scanner, and healing
• Capacity: raw/used/free comparison and per-drive utilization
• Workload: objects, per-bucket usage, total/failed request rate, and P95 latency
• Runtime: RustFS process CPU and resident memory
• Storage diagnostics: stacked drive cards grouping state, capacity, pool/set/disk placement, observation freshness, and scan/heal state
• Scanner diagnostics: a summary, enablement/freshness evidence cards, and current counters in a structured table
• Background-heal diagnostics: a summary and evidence cards for state, queue, active/retrying work, failures, and object progress
• Topology: cluster nodes and their drive relationships

Diagnostic tabs use the same key-metric, summary-table, and stacked evidence-card patterns as other built-in resource plugins. They do not display scanner or background-heal responses as undifferentiated raw JSON.

Alert Rules

rustfs-alert-rules-v1 enables these eight rules by default.

Rule	Severity	Default condition	Pending duration	First checks
Usable capacity high	warning	at least 85% over a 15-minute window	10 minutes	growth, lifecycle policy, expansion lead time
Usable capacity critical	critical	at least 95% over a 5-minute window	3 minutes	preserve write/heal headroom, expand or safely remove data
Drive offline	critical	at least one over a 2-minute window	1 minute	node, disk path, filesystem, hardware, healing
Drive utilization high	warning	per-drive 85% over a 15-minute window	10 minutes	skew, placement, expansion headroom
Drive utilization critical	critical	per-drive 95% over a 5-minute window	3 minutes	write and healing headroom on the affected drive
Request P95 latency high	warning	at least 1 second over a 10-minute window	5 minutes	request rate, disks, scan/heal activity, network
Request P95 latency critical	critical	at least 3 seconds over a 5-minute window	2 minutes	storage saturation, network errors, client timeouts
Process restart loop	warning	at least two restarts in 30 minutes	1 minute	exit reason, memory pressure, storage initialization

Generic liveness and readiness alerts are Core-owned and use direct health/runtime probes instead of Prometheus up. Drive metrics missing from alpha.83 remain no-data and do not create a zero-valued condition.

Diagnostic Interpretation

State or evidence	Meaning	Next check
Healthy liveness, unknown readiness	The health route responds but does not prove storage readiness	Admin credentials and online-node evidence from /info
Admin API unavailable	Credentials are absent, or permission, signature, or version compatibility failed	Response status, signing region, system clock, API route, and RustFS version
Drive data absent	The release does not export the metric or the Prometheus mapping does not match	/storageinfo, raw Prometheus names, and mapping rules
Scanner or healing unavailable	The release lacks the status API or handler	RustFS release support for that API
Stale metric evidence	Collection or querying is delayed independently of RustFS state	Prometheus target, scrape interval, and last sample time

Unknown, unavailable, and no-data do not mean healthy or zero. Evaluate direct API state separately from time-series collection state.

Example Workflows

Capacity alert

1. Read raw, used, and free capacity together with usable-capacity utilization.
2. Check per-drive skew and offline drives.
3. Compare request volume and object growth with scanner and healing activity.
4. Preserve write and healing headroom when scheduling expansion or eligible data removal.

Offline drive

1. Locate the affected node, pool, set, and disk path in storage diagnostics.
2. Check node state, mounts, filesystem, hardware, and network.
3. After recovery, inspect background-heal queue, failures, and progress.
4. Watch capacity alerts and request errors until healing completes.

Increasing request latency

1. Compare total request rate, failed request rate, and P95 latency over the same interval.
2. Check drive utilization, offline state, and host CPU/memory.
3. Correlate latency with scanner or healing activity.
4. Check network errors and client timeouts, then act on the confirmed bottleneck.

Troubleshooting

Symptom	Check in this order
Connection Test returns 400	Validate the endpoint format and required field combinations; inspect the API response body and Konduo backend validation log instead of relying only on the browser status code
Admin API returns 403	Check the access/secret pair, signing region, clock synchronization, proxy Host/path rewriting, and permission for the target route
Resource readiness is unknown	Check the alpha.83 /health limitation and whether admin credentials can read online-node evidence from /info
Every metric panel is empty	Confirm that Collector :8889/metrics was not registered as a Prometheus query resource; check the Prometheus target, resource link, and mapping pack
Only rustfs_rustfs_* names exist	The Collector namespace: rustfs duplicated the prefix; confirm v4 mapping and consider removing the namespace for new configurations
Only some panels or alerts have no data	Confirm that the RustFS release exports those metrics, then compare raw names/labels with mapping rules
Scanner or healing diagnostics are unavailable	Check whether alpha.83 is in use; upgrade to a release that implements the required API and rerun diagnostics
Only HTTPS fails	Fix the certificate chain and server name first; skip verification only in a controlled test environment

Safety Boundary

The CE plugin is read-only. It does not restart or stop RustFS, mutate pools, change buckets or objects, start manual healing, or alter lifecycle settings. Access keys and secret keys are never returned in diagnostic payloads.

---

RustFS Enterprise Overlay

The RustFS Enterprise overlay adds multi-signal anomaly rules and read-only MCP descriptors to the Community RustFS resource plugin. Community continues to own connectivity, SigV4 admin reads, health checks, diagnostics, topology, metrics, dashboards, and managed alert rules.

Before Use

• The Community RustFS resource connection must be healthy. Configure read-only admin API credentials when detailed operational evidence is needed.
• Link a Prometheus-compatible metric source through the RustFS mapping pack.
• The Enterprise mcp.gateway and anomaly.engine features must be enabled.
• Reading anomaly rules requires the viewer role and the rustfs-plugin.anomaly.read permission.
• MCP callers need both an allowed MCP read scope and access to the target RustFS resource instance.

RustFS releases that do not export optional drive-health or scanner metrics leave dependent anomaly rules unevaluated. Missing series are not interpreted as healthy or zero values.

Anomaly Rules

The Enterprise pack provides six multi-signal rules.

Rule key	Severity/score	Evaluation condition
rustfs.storage_resilience_cascade	critical/0.97	cluster capacity utilization >= 85% and offline drives > 0
rustfs.drive_imbalance_pressure	warning/0.86	cluster capacity utilization < 85% and per-drive utilization >= 95%
rustfs.capacity_observation_gap	warning/0.79	stale drive-capacity observations > 0 or missing observations > 0
rustfs.request_path_degradation	critical/0.94	request failure rate > 0 and P95 latency >= 1,000ms
rustfs.restart_failure_cascade	critical/0.92	process restarts in 30 minutes >= 1 and request failure rate > 0
rustfs.scanner_request_contention	warning/0.84	scanner object rate > 0, cycle duration >= 3,600s, and P95 latency >= 1,000ms

These rules provide triage evidence. They do not replace single-series managed alerts. Confirm affected nodes, drives, requests, and time ranges before changing capacity, placement, scanner load, or process settings.

MCP Resources and Tools

The MCP catalog exposes 12 read-only resources:

• runtime_probe
• monitoring_overview
• diagnostics_summary
• diagnostics_storage
• diagnostics_scanner
• diagnostics_heal
• topology_summary
• metrics_catalog
• mapping_pack_catalog
• alert_rules
• anomaly_rules
• software_inventory

It also exposes 11 read-only tools:

• runtime_probe
• monitoring_overview
• diagnostics_summary
• diagnostics_storage
• diagnostics_scanner
• diagnostics_heal
• topology_summary
• metrics_query_resolve
• mapping_pack_resolve
• alert_rules
• anomaly_rules

metrics_query_resolve requires logical_metric_key and accepts optional query_mode (instant or range) and instance_selector. mapping_pack_resolve accepts optional target_metric_source_plugin_id, whose only current value is prometheus-plugin.

MCP does not expose service control, bucket or object mutation, manual healing, lifecycle changes, diagnostic refresh, or credentials.

Operations and Incident Response

1. Use runtime_probe and monitoring_overview to verify connectivity, metric linkage, and the latest collection time.
2. Read managed alerts and anomaly rules together, comparing evidence over the same time window.
3. Use storage, scanner, healing, and topology diagnostics to identify affected nodes and drives.
4. Resolve the relevant logical metrics with metrics_query_resolve and inspect their raw series.
5. Preserve diagnostic evidence before performing recovery, placement, or capacity changes outside Konduo.

If anomaly rules are absent, verify the Enterprise license, anomaly.engine, and the caller's rustfs-plugin.anomaly.read permission. If the MCP catalog is absent, verify mcp.gateway, the MCP read scope, and resource-instance access. If a rule remains unevaluated, inspect the linked Prometheus source and its required logical metrics. Drive-capacity observation and scanner metrics can be optional depending on the RustFS release.

Edition Boundary

The anomaly rule pack, MCP descriptors, and Enterprise localization remain in the Enterprise overlay. The Community plugin remains independently installable and usable without these commercial contributions.