Apache Fluss Plugin

The Apache Fluss plugin registers a Fluss cluster as a Konduo resource. It monitors CoordinatorServer, TabletServer, bucket, replication, request path, storage, and JVM state.

Highlights

• Negotiates the Fluss protocol and discovers the coordinator and alive tablet servers through multiple bootstrap addresses.
• Provides 27 Fluss logical metrics through a Prometheus mapping pack.
• Uses 27 default panels split between Essential and Detail for cluster, throughput, replication reliability, request pressure, storage, and JVM resources.
• Uses category rollups in the summary and decision evidence with recommended actions in detail views.
• Separates direct connection evidence from time-window metrics supplied by a linked Prometheus resource.
• Selects databases and tables from the catalog for bounded Log and Primary-key data operations, then manages configuration, ACLs, server tags, and rebalance through permission-aware safeguards.

Before Registration

1. Prepare one or more Fluss CoordinatorServer or TabletServer host:port addresses that are reachable from the Konduo backend.
2. Enable the Prometheus reporter on every CoordinatorServer and TabletServer.
3. Confirm that Prometheus scrapes each server's metric endpoint.
4. Link the target Prometheus resource to the Fluss resource as its metric source.
5. If the Fluss version or reporter scope differs from the default mapping, inspect actual metric names and labels and prepare environment-specific mapping rules.

Connection Settings

Field	Required	Default	Description
endpoint	yes	localhost:9123	Comma-separated CoordinatorServer or TabletServer host:port bootstrap addresses
security_protocol	yes	PLAINTEXT	PLAINTEXT, TLS, SASL_PLAINTEXT, or SASL_TLS
sasl_mechanism	for SASL	PLAIN	Fluss 0.9.1 supports PLAIN
sasl_username, sasl_password	for SASL	none	PLAIN credentials; the password is stored as a secret
tls_ca_cert	optional TLS	system roots	PEM CA certificate
tls_client_cert, tls_client_key	optional TLS	none	PEM client certificate pair
tls_server_name	optional TLS	derived	Certificate server-name override
tls_skip_verify	optional TLS	false	Disables certificate verification; use only for isolated testing
dial_timeout_seconds	no	10	Per-connection dial timeout, 1–300 seconds
request_timeout_seconds	no	15	Protocol request timeout, 1–300 seconds
read_max_attempts	no	2	Safe-read attempts, 1–3; mutations are not automatically retried
Metric-source link	for metrics	none	prometheus-plugin resource that scrapes every Fluss server

Connection Test uses fluss-go v0.1.0-beta.10 to negotiate with Apache Fluss 0.9.1-incubating, then retrieves CoordinatorServer and alive TabletServer metadata. A TCP handshake without protocol negotiation is not considered healthy.

Multiple bootstrap addresses preserve a discovery path when one server fails. Separate addresses with commas and use the actual server host:port, not an HTTP scheme or URL path.

Registration And Validation

1. When possible, enter at least two CoordinatorServer or TabletServer addresses from separate failure domains in endpoint.
2. Run Connection Test and confirm protocol negotiation, exactly one coordinator, and the expected number of alive tablet servers.
3. Confirm that Prometheus scrapes the reporter endpoint on every CoordinatorServer and TabletServer.
4. Link that Prometheus resource to Fluss as its metric source.
5. In cluster overview, compare active CoordinatorServer/TabletServer and offline-bucket counts with the actual deployment.
6. Confirm that CoordinatorServer and TabletServer CPU/heap panels preserve distinct server labels.
7. If replication or request diagnostics report unavailable, inspect the logical-metric route and actual reporter scope instead of treating it as healthy.
8. CE has no managed alert pack, so explicitly add required logical metrics to the general alert configuration.

Protocol health, per-server Prometheus collection, and logical-metric mapping validate separate paths. Protocol discovery does not by itself prove healthy buckets, metrics, or application data.

First-time Operator Path

1. Register the resource and use Connection Test to verify protocol negotiation plus Coordinator and alive TabletServer discovery.
2. Link the Prometheus resource that scrapes every Fluss server. Start at Dashboard > Essential; when it shows risk, use Detail to correlate request pressure and storage evidence.
3. In Diagnostics > Summary, distinguish ok, partial, and unavailable. Use category details and the History window to narrow the cause.
4. In Operations > Catalog, select the table action on a database row, then open Log Data or Primary-key Data from the applicable table row. The chosen database and table carry into the workspace.
5. Review the preflight result before a mutation. The web UI displays the target-derived confirmation value read-only, and dangerous actions require one additional final confirmation.
6. If the outcome is unknown or reconciliation remains pending, do not repeat the mutation immediately. Read authoritative catalog, bounded data, or task detail state first.

The Operations page is not a general SQL console. Every inventory, read, and mutation runs against the selected target and within the documented bounds.

Read-only Operations

Operators with the corresponding fluss-plugin.operations.*.read permission can inspect Coordinator-authoritative cluster and catalog data:

Area	Routes	Notes
Cluster	operations/cluster, operations/servers	Coordinator and alive TabletServer identity, role, address, and rack
Databases	operations/databases, operations/databases/detail	Comment, properties, timestamps, and table count
Tables	operations/tables, operations/tables/detail	Kind, buckets, row count, compact per-bucket offset range, complete logical schema, and properties
Partitions	operations/partitions	Optional partial partition_spec JSON filter
Offsets and statistics	operations/tables/offsets, operations/tables/stats	Per-bucket offset ranges, optional timestamp resolution, and row counts
Configuration	operations/cluster-config	Effective value and server-reported source
ACLs	operations/acls	Filtered access-control entries with human-readable enums

List routes accept search, offset, and limit; the maximum result limit is 1,000. Table routes require database and table. A partition target uses a JSON string map such as {"region":"kr"}. Bucket failures are returned individually with partial status. unavailable, permission denied, invalid input, and empty results are distinct states.

The catalog UI keeps only search above database and table inventories because the table component owns interactive pagination. API clients can still use offset and limit. Delete database/table controls are attached to their rows. The selected target, expected table/schema IDs, and confirmation phrase are filled read-only from that row; operators still run dry-run and approve the final destructive confirmation. Table alteration and partition navigation also start from the selected table row. Partitions are a hidden table-scoped drill-down, and partition deletion is attached to the selected partition row.

The table inventory shows row counts and offset ranges directly for both Log and Primary-key tables. It sums bucket row statistics, summarizes the earliest/latest boundaries for up to three buckets, and shows the numeric offset span summed across all buckets. The offset span is not an exact row count. Partitioned tables show Per partition for both values because no single table-wide result exists. Automation that needs an offset at a specific time can still call operations/tables/offsets with an RFC3339 timestamp. Runtime enrichment is not polled every 30 seconds because it requires multiple Coordinator requests per table. It runs on entry or manual refresh, and every row labels the evidence as OK, partial, unavailable, or partition-scoped.

Cluster Configuration And ACL Operations

operations/cluster-config returns effective server values with their source and always redacts values whose keys resemble credentials. An administrator with fluss-plugin.operations.config.write can send up to 20 environment-allowlisted changes to operations/cluster-config/apply. SET, DELETE, APPEND, and SUBTRACT are supported. Dry-run can compare optional expected_value and expected_source preconditions and records fingerprints instead of echoing values. A real mutation requires APPLY CLUSTER CONFIG <environment>, is submitted once, and is verified by an authoritative configuration read. The default UI uses a key/value editor for SET changes selected from the current inventory. DELETE, APPEND, SUBTRACT, and optimistic expected-value or source checks remain available through Advanced JSON mode.

operations/acls filters by resource, principal, host, operation, and permission. operations/acls/create accepts explicit ALLOW entries, while operations/acls/drop rejects an all-wildcard deletion. Real changes require CREATE ACLS <count> or DROP ACLS <count>. Per-item outcomes are preserved and successful requests are verified by listing ACLs again. User, Group, and Role principals use the typed, canonical ACL contracts provided by fluss-go. The UI creates a single ACL through structured fields and reserves batch JSON for advanced creation. Exact ACL deletion starts from the selected row, which prefills the explicit filter and confirmation phrase. A disabled Fluss authorizer is reported separately from permission denial. Production, staging, and development use per-resource cooldowns; test mode does not. The page shows ACL, principal, resource, and broad all-operation counts before the filters. Resource and operation are server-side filters; the table search matches principal, resource, host, and other visible text. The compact list is ordered as principal, resource, operation, and source host. Since this Fluss contract only creates ALLOW entries, permission remains visible in the detail sheet instead of consuming a list column.

Server Maintenance And Rebalance

Admin-only maintenance operations accept official Fluss 0.9 names rather than raw protocol integers. Server tags are PERMANENT_OFFLINE and TEMPORARY_OFFLINE; rebalance goals are REPLICA_DISTRIBUTION, LEADER_DISTRIBUTION, and RACK_AWARE. Tag changes use operations/servers/tags/add and operations/servers/tags/remove. Since Fluss has no server-side preview, dry-run is explicitly reported as a local preflight of current TabletServers, racks, and target IDs.

The UI exposes rebalance-policy presets instead of accepting free-form goal names. The default balances replicas and leaders. For a multi-rack cluster, select rack-aware placement followed by replica and leader balancing. RACK_AWARE must be first so later goals preserve rack constraints; the API rejects an unsafe order.

The Fluss 0.9 protocol has no API for reading current tags. Permanent-offline add/remove can be reconciled when Coordinator metadata changes accordingly. Temporary-offline and other inconclusive results remain reconciliation-pending and are never retried blindly.

operations/rebalances/start requires at least three alive TabletServers and checks rack labels plus supplied offline/under-replicated bucket evidence. The server-issued rebalance ID is retained as the task reference, and a second local start is blocked while the known task is active. Poll operations/rebalances/detail with the ID for deterministic table/bucket status, original/new leaders, and replicas. operations/rebalances/cancel distinguishes active and already-final tasks and warns that completed moves are not rolled back. Every real action uses an exact confirmation phrase and cooldown, is submitted once, and reads Coordinator state again. The operations page automatically follows the ID tracked for that resource instance and displays it as read-only progress metadata. Operators select the rebalance goal priority; the Coordinator calculates the bucket moves and the page renders that returned plan as a table. An explicit ID remains available to API callers and operators inspecting another task.

Operations Page, Access, And Audit

The resource dashboard exposes a read/write operations workspace at /resources/{resourceInstanceId}/manage/operations. The page reads the declarative operations/schema contract and presents four visible tabs: Cluster, Catalog, Rebalance, and Security. Cluster presents the Coordinator and TabletServer counts together with the server inventory, and opens Cluster Config as a contextual child workspace. Select a database row in Catalog to inspect its metadata and properties in a compact sheet, or use its table action to list its tables. The table inventory obtains the authoritative table type, shows its row count and offset summary, and offers Log Data or Primary-key Data actions only when applicable; the selected database and table are carried into the target workspace automatically. Table detail, partition, and point/prefix lookup views are hidden drill-down tabs that retain the selected context and provide a route back to their parent.

Tabs and actions reference the manifest access catalog. Viewer can inspect cluster, catalog, configuration, and bounded data results; editor can perform non-destructive catalog and data writes; admin is required for maintenance, ACL and configuration changes, catalog deletion, and data deletion. The core proxy applies the linked capability descriptor to direct API calls as well as the page, so hiding a control is never the only permission check.

The minimum roles and permission patterns are listed below. A role alone does not grant an action when its corresponding permission pattern is absent.

Scope	Minimum role	Representative permission patterns
Cluster, catalog, configuration, and bounded-data reads	viewer	operations.cluster.read, operations.catalog.read, operations.config.read, operations.data.read
Database/table/partition create or alter and Log/KV writes	editor	operations.catalog.write, operations.data.write
ACL and rebalance reads	admin	operations.security.read, operations.maintenance.read
Configuration, ACL, server-tag, and rebalance mutations	admin	operations.config.write, operations.security.write, operations.maintenance.write
Catalog and Primary-key data deletion	admin	operations.catalog.delete, operations.data.delete

Every pattern in the table has the fluss-plugin. prefix in the complete permission. For example, bounded data reads use fluss-plugin.operations.data.read.

Creation forms leave new object names and free-form payloads empty. Only explicit parent context, such as the current database for a new table, is prefilled; the first or previously selected inventory row is never copied into a create form.

Queries use bounded pagination, search, and refresh intervals. Explicit Log and Primary-key data queries prevent duplicate execution only while a request is in flight and do not add a UI cooldown after completion. Write forms start in dry-run and require the exact target-derived confirmation phrase before a real request. A successful action triggers only a bounded reload. Dry-run feedback remains in the action dialog for review. A completed mutation uses a transient success notification without duplicating the same message in the result-table header. Validation and execution errors stay in the action dialog and also use error notifications. After the dialog closes, block headers, row-action areas, and preview sheets do not retain old feedback beside their launch controls. The core audit request ID becomes the execution ID when the caller does not supply one. Audit evidence keeps target, route, request, result, and execution references while credentials, row and key payloads, ACL bodies, configuration changes, Arrow batches, and client keys are redacted and never included in operator-facing action messages.

The web UI derives the route-specific confirmation value from the current parameters or selected row and displays it read-only. Operators do not need to retype a command-like phrase. Direct API callers must still send the exact value, and destructive operations retain a separate final confirmation step.

The Filter above a result table narrows the already returned browser page. It is not a server predicate and does not execute arbitrary Fluss WHERE conditions or a cluster-wide table search. Long record values are truncated in the table; select a row to inspect and copy the complete structured value in the detail sheet.

Log Table Data Operations

With fluss-plugin.operations.data.read, operations/tables/log/scan performs a bounded Log Table read. The default is 100 rows and the hard limits are 1,000 rows, 16 MiB, and 30 seconds. start_kind supports latest_backward, latest_forward, earliest, timestamp, and offset (latest remains an API alias for latest_forward); the API accepts RFC3339 timestamps and offsets are non-negative. In the web UI, enter the timestamp in the application timezone. Konduo converts it to UTC RFC3339 for the request and renders returned record times in the same application timezone. Optional projection, partition_spec_json, and per-bucket exclusive stopping_offsets_json values are supported. Projection names are validated against the table schema and applied to the bounded row and Arrow results in the plugin; partition and stopping-offset options are pushed into the scanner. Snapshot modes capture the current per-bucket log ends before reading, so a 100-row limit returns immediately with fewer rows when fewer are currently available. Results retain bucket, offset, and high-watermark evidence plus scan_mode and completion_reason. Bucket-local failures remain visible as partial; this route is neither an unbounded tail nor a cluster-wide search.

Use the web UI's Read Start controls as follows.

Choice	Behavior	Additional input
From Beginning	Reads from each selected bucket's current earliest offset to its request-time latest offset	None; this is the default
Recent Rows	Distributes the limit across selected buckets and returns the request-time newest rows first without waiting	None
Wait for New Rows	Starts at the request-time latest offsets and waits forward for appended rows	None; returns on row, byte, or time limit
Offset	Applies the same non-negative start offset to each selected bucket	Start Offset
Timestamp	Resolves a per-bucket offset at or after the specified time	Local date and time in the web UI; RFC3339 with timezone in the API

Read Columns accepts comma-separated returned column names such as clerk,customer_id. It is a projection, not a row predicate. Entering clerk returns only that column from every selected row; it does not search for a clerk value. Leave it empty to return every column.

operations/tables/log/append validates rows_json against the authoritative table schema, or accepts one schema-matching Arrow IPC record batch through arrow_ipc_base64 and an explicit bucket. JSON rows use a non-empty object array such as [{"id":1,"payload":"value"}], never a single object. It supports auto/key-hash, sticky, and round-robin assignment; auto/indexed/compacted row formats; and NONE/LZ4/ZSTD Arrow compression.

For example, when the selected table schema requires id, event_type, and payload, enter an array:

[
  {"id": 1, "event_type": "CREATED", "payload": "value"}
]

A single {...} object is invalid. Field names, required columns, and value types come from the selected table's authoritative schema, not from this example. Only Partition Spec JSON uses one object such as {"region":"kr"}; do not place it inside the row array.

Append requires fluss-plugin.operations.data.write and the editor role. It always starts with dry_run=true; a real write requires the exact APPEND LOG <database.table> phrase. The default is one attempt. An operator may choose two or three write_max_attempts; this enables fluss-go's idempotent writer retry while preserving writer identity, bucket sequence, and encoded bytes, and requires acks=-1. Accepted records remain serial across buckets to preserve operator-visible result order and partial-failure evidence. An uncertain result after the selected attempts is outcome-unknown; inspect offsets or run a bounded scan before deciding what to do next.

Primary-key Table Data Operations

operations/tables/kv/lookup and operations/tables/kv/prefix-lookup preserve the order of composite-key objects in keys_json. Point lookup requires the complete primary key; prefix lookup accepts only a non-empty leading subset. Every result retains input_index, key or prefix, bucket, and an explicit found, not_found, or per-item error state. The bounded scheduler uses a maximum 1,000-key queue, a fixed one-millisecond batch delay, the configured request timeout, and read_max_attempts. Atomic insert-if-not-exists remains single-attempt because it may change state.

The web UI exposes these read modes:

Mode	Input	Purpose
Browse	Optional returned columns and row limit	Walks buckets for a bounded view of current rows
Point lookup	Complete Primary Key object array such as [{"customer_id":0}]	Retrieves the current row for known keys
Prefix lookup	Leading-key object array such as [{"region":"kr"}]	Restricts a lookup by the first columns of a composite Primary Key

Key input is an object array, not one object. For a composite Primary Key ordered as region, customer_id, point lookup needs both columns and prefix lookup may use region alone. Supplying only customer_id skips the leading column and is invalid.

operations/tables/kv/scan takes a stable snapshot of current bucket leaders and scans each bucket within the 100-row default, 1,000-row/16 MiB/30-second hard limits and optional projection. A failed bucket remains visible as partial. Partitioned tables require an exact partition_spec_json, so an operation never silently crosses a dynamic partition boundary.

Send complete rows to operations/tables/kv/upsert and rows containing the primary key plus changed columns to operations/tables/kv/partial-upsert. The configured merge engine is the default; explicit overwrite requires a table merge-engine setting. operations/tables/kv/insert-if-not-exists is a write action rather than a read query because it can atomically create a missing key. Upsert and insert accept an object array whose entries contain the complete Primary Key and every required column. Partial upsert accepts an object array whose entries contain the complete Primary Key and columns to change. Delete uses an array of complete Primary Key objects such as [{"primary_key":1}]. Only the optional partition specification is one JSON object rather than an array.

The following shapes assume customer_id is the Primary Key. Use the selected table's actual columns and value types.

Action	Required row shape	Example
Upsert	Complete Primary Key and every required column	[{"customer_id":0,"name":"Alice","balance":"146.30"}]
Partial upsert	Complete Primary Key and at least one changed column	[{"customer_id":0,"balance":"200.00"}]
Insert if missing	The same complete row required by upsert	[{"customer_id":1,"name":"Bob","balance":"0.00"}]
Delete	Complete Primary Key only	[{"customer_id":1}]

The web UI distinguishes Row Array JSON from Primary Key Array JSON and rejects a single object before submission. Success uses a transient notification and reloads the inventory. Errors remain with the input in the action dialog; long server messages are not left in result-table headers.

Upsert, partial upsert, and insert require operations.data.write; delete separately requires operations.data.delete and the admin role. Every action uses dry-run and a target-bound phrase; delete requires DELETE KV <database.table>. KV writer actions default to one attempt and may use up to three explicitly selected idempotent attempts only with acks=-1. Insert-if-not-exists is never retried. Every accepted mutation is verified through point lookup; an inconclusive verification returns reconciliation-pending instead of claiming success or blindly repeating it.

Catalog Lifecycle

Database, table, and partition create/alter/drop actions default to dry_run=true. Review the preflight summary, refresh the authoritative detail, then execute the action. The web UI sets the exact target-derived confirmation value read-only; operators do not retype it. The phrases below are the confirmation_phrase contract for direct API callers:

• CREATE DATABASE <database> or DROP DATABASE <database>
• CREATE TABLE <database.table>, ALTER TABLE <database.table>, or DROP TABLE <database.table>
• CREATE PARTITION <database.table> or DROP PARTITION <database.table> <partition-name>

For table alter/drop and partition actions, send the table ID and schema ID observed during preflight as expected_table_id and expected_schema_id. Mismatch rejects stale work without sending a mutation. schema_json preserves the Fluss logical schema; top-level partition_key and bucket_key arrays define the table keys and must reference declared columns. alter_json groups config set/delete/append/subtract, column add/drop, and rename changes. partition_spec must match the ordered table partition key.

The Create Table dialog shows this validated Primary-key Table example. For a Log Table, keep the columns and keys you need but set primary_key to [].

{
  "version": 1,
  "columns": [
    {"name": "id", "data_type": {"type": "BIGINT", "nullable": false}, "id": 0},
    {"name": "value", "data_type": {"type": "STRING", "nullable": true}, "id": 1}
  ],
  "primary_key": ["id"],
  "partition_key": [],
  "bucket_key": ["id"],
  "auto_increment": [],
  "highest_field_id": 1
}

Konduo never automatically retries these mutations. It reads the target again after one request. If a response is lost, the result states that the outcome is unknown; inspect the corresponding read route before deciding whether to retry. Database cascade and table/partition deletion require an administrator and are recorded as dangerous actions.

The current release uses the following integration baseline.

Area	Tested baseline	Interpretation
Apache Fluss	0.9.1-incubating, one CoordinatorServer and three TabletServers	This is the current compatibility baseline for operational features.
Go client	github.com/pletorco/fluss-go v0.1.0-beta.10	Both the fgo data protocol and fadm administration protocol are exercised. Optional storage and observability adapter modules are not part of the plugin dependency graph.
Plain and authenticated connections	PLAINTEXT and SASL/PLAIN	These use the native Fluss 0.9.1 listeners.
TLS	PLAINTEXT and SASL/PLAIN through TLS termination	Do not interpret this as native TLS support in Fluss 0.9.1.
Failure scenarios	TabletServer loss and recovery; CoordinatorServer loss	TabletServer recovery and serving-state refresh are verified. Fluss 0.9.1 has no CoordinatorServer HA, so coordinator loss must report unavailable rather than a successful failover.

Integration coverage includes catalog lifecycle, Log and Primary-key data paths, configuration, ACL, rebalance, cancellation, and cleanup. Validate the same paths separately before using a Fluss or fluss-go combination outside this baseline in production.

Dashboard And Metrics

The default dashboard provides 27 panels across Essential and Detail subpages.

The Essential subpage provides:

• Cluster overview: active CoordinatorServers and TabletServers, offline buckets, tables, and buckets
• Throughput: messages, input bytes, and output bytes per second
• Replication reliability: buckets under minimum ISR, ISR shrink rate, failed ISR updates, and request errors
• JVM resources: CoordinatorServer and TabletServer CPU and heap utilization

The Detail subpage provides:

• Replication margin: under-replicated and at-minimum-ISR buckets
• Request processing: request rate and queues, delayed writes and fetches, and expired writes
• Storage: logical log/KV and physical local/remote log sizes

Every logical metric in the default catalog therefore has one default dashboard panel. Use the Essential view to identify a problem, then correlate replication margin, request backlog, and capacity growth in Detail.

The default mapping pack targets Apache Fluss 0.9-style Prometheus metric names. For example, message input uses fluss_tabletserver_messagesInPerSecond, and buckets under minimum ISR use fluss_tabletserver_underMinIsr.

Request metrics are exported under request-specific scopes such as request_produceLog, request_putKv, and request_fetchLogClient. The plugin therefore aggregates request-type names instead of expecting one nonexistent generic request metric.

Diagnostics

The diagnostic summary presents short status rollups for five categories:

• Cluster: bootstrap TCP reachability, actual Fluss protocol negotiation, Coordinator metadata, active CoordinatorServers and TabletServers, catalog readability, and offline buckets
• Replication: buckets under, at, or below the expected replica state and failed ISR updates
• Requests: request errors and queues, delayed writes and fetches, and expired writes
• Storage: logical, local, and remote storage plus delayed write evidence
• Runtime: per-server CPU and JVM heap pressure

The page is organized as Summary, Cluster, Replication, Requests, Storage, Runtime, and History. Summary is the current triage view; each category shows observed values, decision basis, impact, and next action. History reads repeated risk and trend evidence from the linked metric source for 1h, 6h, 24h, 7d, 14d, or 30d. Missing history never proves that the current Fluss state is healthy.

Detail views provide observed values, decision basis, impact, evidence freshness, and the next action. Summary rows intentionally avoid repeating all detail evidence.

The automatic live probe checks up to three bootstrap addresses within an overall eight-second limit. TCP, protocol/TLS/SASL negotiation, Coordinator metadata, and catalog reads remain separate findings. A mismatch between the Admin API TabletServer count and linked Prometheus active-server evidence is reported as monitoring coverage rather than silently treated as server loss. A Prometheus collection failure alone does not prove that Fluss is down.

Time-window metric diagnosis requires at least five samples and at least 50% of expected samples for the selected interval. Evidence below that boundary is reported as partial, not healthy. Metric-source and mapping failures remain unavailable.

Run bounded diagnosis is a manual read-only probe separate from automatic refresh. Within 15 seconds it samples at most two databases, four tables, two partitions per table, and eight buckets per physical table. It checks GetTableInfo/GetTable schema and metadata, ResolveTableBuckets routing, and independent bucket outcomes from ListOffsets and GetTableStats. Primary-key Tables add latest KV snapshot availability and offset lag. A rebalance is queried only when this plugin already tracks its ID. The probe never scans rows, tails logs, performs lookups, or mutates data.

The report excludes database/table names, endpoints, bucket IDs, credentials, row/key payloads, and raw server errors. Resource fingerprints and bounded status aggregates are retained in memory for at most 30 minutes and 32 resources. Restarting the plugin clears them; this is not a persistent history store.

The request detail finding named Konduo Fluss client path uses the fluss-go observer to aggregate a fixed 15-minute window of RPC latency, dial failures, retries, timeouts, throttles, queue signals, scanner lag, and decode failures. It uses fixed latency buckets and at most 32 resource fingerprints, and stores no address, table path, bucket, or payload. RPC, dial, lookup, Log write, KV write, Log scan, and remote-read evidence is also separated into fixed operation aggregates. It represents only requests made by Konduo, never cluster-wide Fluss traffic.

Diagnostic Interpretation

State or evidence	Meaning	Next check
Bootstrap healthy	At least one TCP discovery path is reachable	Active CoordinatorServers/TabletServers and bucket state
TCP healthy, protocol failed	The port is open but TLS/SASL or a Fluss request failed	Security mode, credentials, permission, and Coordinator logs
Bounded read path is partial	Only some metadata/routing/offset/statistics bucket samples failed	Failure category and affected TabletServer state, then rerun manually
Bootstrap partially failed	Discovery remains available but has less redundancy	Failed address, DNS, firewall, server, and failure domain
Metric source unavailable	No linked Prometheus source can provide time-window evidence	Link the Prometheus resource that scrapes every Fluss server
Partial evidence	Fewer than five samples or less than 50% expected coverage	Scrape interval, selected window, target outages, and missing labels
Only one role has no data	That role is not scraped or its scope/mapping differs	Per-role targets, raw metric names, and labels
Replication or request risk	Related evidence must be investigated together	ISR, server availability, request type, queue, and storage

Healthy, partial, and unavailable are distinct states. Insufficient evidence is never promoted to healthy, and a Prometheus collection failure is not converted into a Fluss outage.

Alert Support

The current CE Fluss plugin does not provide a managed alert rule pack. The dashboard and diagnostics expose risk evidence, but operators must not assume that an alerts/rules surface exists. Until a managed pack is added, select the required logical metrics explicitly in the linked metric source policy or Konduo's general alert configuration.

Example Workflows

Bucket availability degradation

1. Check active CoordinatorServer and TabletServer counts.
2. Read offline, under-minimum-ISR, and under-replicated bucket counts together.
3. Check whether ISR shrink and failed ISR update signals rose at the same time.
4. Compare CPU, heap, storage, and network health on affected TabletServers before recovery.

Increasing request errors

1. Compare request error rate with request queue depth.
2. Check delayed writes, delayed fetches, and expired writes.
3. Identify whether errors are concentrated in one request type.
4. Compare TabletServer resources, replication state, and storage latency.

Storage growth

1. Separate logical log, KV, local, and remote log sizes.
2. Compare growth skew and write throughput across TabletServers.
3. Check for simultaneous delayed writes or replication degradation.
4. Follow Fluss retention and remote-log movement procedures.

Management Boundary

• The CE plugin executes explicitly requested catalog, data, configuration, ACL, server-tag, and rebalance actions after permission and safety checks. It never creates or drops tables, changes server tags, or starts rebalance autonomously from metric or diagnostic results.
• It does not provide continuous remediation, policy-driven automatic rebalance, or an arbitrary SQL executor. The Coordinator calculates the actual rebalance move plan; the plugin displays the returned plan and state.
• It does not infer cluster-wide configuration or replica/ISR placement that public APIs do not expose. Bounded Admin diagnostics report only calls that were actually executed.
• Do not interpret bootstrap TCP success as healthy data-path operation.
• Evaluate metric collection failure separately from Fluss resource failure.

Troubleshooting

Symptom	Check in this order
Every bootstrap probe fails	Check host:port format, CoordinatorServer/TabletServer state, backend DNS/network path, and firewall
Only some bootstrap addresses fail	Check for duplicate or stale addresses, then restore the affected server and failure domain
Connection succeeds but cluster diagnosis is unavailable	Identify whether protocol session, Coordinator metadata, or catalog read failed after TCP succeeded
Bounded diagnosis is partial	Inspect per-sample metadata/routing/offset/statistics failure categories and TabletServer state, then rerun manually
Every metric panel is empty	Check reporters on all Fluss servers, Prometheus targets, the metric-source link, and the Fluss 0.9 default mapping
Only one role is empty	Check per-role scrape targets and actual metric scopes and labels
Diagnosis is partial	Check sample count, scrape interval, and target outage time for the selected window
Only request metrics are empty	Check request-specific scopes such as request_produceLog, request_putKv, and request_fetchLogClient, plus mapping regular expressions
Custom reporter scopes produce no data	Inspect raw Prometheus names and labels and apply an environment-specific mapping override
Wait for New Rows returns zero rows	This mode excludes existing rows; append a new row or use From Beginning, Recent Rows, Offset, or Timestamp
Read Columns does not find rows with a value	Read Columns is a returned-column projection, not a predicate; use the current-result text filter or an applicable key lookup
Validation says rows_json must be a JSON array	Replace {...} with [{...}] and include the selected schema's required columns and complete Primary Key in every row
A write is outcome-unknown or reconciliation-pending	Do not repeat it immediately; inspect point lookup, bounded scan, or catalog detail first
No rebalance ID is shown	Start rebalance from this resource page so its returned ID is tracked, or provide an explicit ID to the API
ACL inventory is unavailable	Check that the Fluss authorizer is enabled, then verify admin role, operations.security.read, and the connection principal's permission

---

Apache Fluss Enterprise Overlay

The Apache Fluss Enterprise overlay adds multi-signal anomaly rules and read-only MCP descriptors to the Community Fluss resource plugin. Community continues to own connectivity, health checks, protocol diagnostics, the metric catalog, mapping packs, dashboards, and all operational APIs.

Read the Community manual's Apache Fluss Plugin entry first for catalog and data reads and writes, cluster configuration, ACL, server-tag, and rebalance workflows. This guide covers only the Enterprise analysis and MCP surfaces added on top of those workflows.

Feature Scope

Feature	Owning edition	Description
Connectivity and protocol health	Community	CoordinatorServer negotiation, server discovery, and health through fluss-go
Monitoring and diagnostics	Community	Dashboards, logical metrics, current and historical diagnostics, and bounded read-path evidence
Operations workspace	Community	Catalog, data, configuration, ACL, server-tag, and rebalance actions with preflight and audit
Anomaly rules	Enterprise	The fluss-anomaly-rules-v1 pack that combines multiple logical metrics
MCP catalog	Enterprise	Read-only descriptors for Community query routes and Enterprise anomaly rules

Enterprise MCP neither duplicates nor bypasses Community operational APIs. MCP read access does not grant data-write or administrative permissions.

Before Use

1. Follow the Community manual to configure bootstrap servers, TLS, or SASL, and confirm that CoordinatorServer protocol health succeeds.
2. Link a Prometheus-compatible metric source to the Fluss resource through the Fluss mapping pack. The resource scope must include metrics from the CoordinatorServer and every TabletServer.
3. Confirm that the Enterprise license enables mcp.gateway and anomaly.engine.
4. Grant anomaly-rule readers at least the viewer role and fluss-plugin.anomaly.read.
5. Give MCP callers both an allowed MCP read scope and access to the target Fluss resource instance.
6. Confirm the current supported baseline: Apache Fluss 0.9.1-incubating with fluss-go v0.1.0-beta.10.

Enterprise analysis does not replace Fluss protocol connectivity or Prometheus collection. Protocol diagnostics exercise actual Coordinator and TabletServer read paths, while anomaly rules evaluate logical series from the linked metric source. When they disagree, compare collection scope and observation time before treating either side as healthy.

Permissions and Safety Boundaries

Operation	Required access
Read Enterprise anomaly rules	Target-resource access, at least viewer, and fluss-plugin.anomaly.read
Read MCP resources or invoke read tools	MCP read scope and target-resource-instance access
Write catalog or table data	The applicable Community Operations editor permission
Change configuration, ACL, server tags, rebalance, or delete	The applicable Community Operations admin permission and confirmation flow

MCP descriptors expose query routes only. Diagnostic refresh and every mutation continue to use the existing Konduo API boundaries for RBAC, preflight, confirmation, read-back, and audit delivery.

Anomaly Evaluation

fluss-anomaly-rules-v1 provides five rules. Any means that one listed condition is sufficient; all means every condition must match in the same evaluation context.

Rule key	Mode	Severity and score	Logical metrics and conditions	First checks
fluss.cluster_availability_risk	any	critical, 0.98	fluss.cluster.coordinators.active < 1, fluss.cluster.tablet_servers.active < 1, or fluss.cluster.buckets.offline > 0	Coordinator leadership, TabletServer registration, offline buckets, and storage evidence
fluss.replication_degradation	any	critical, 0.95	fluss.replication.buckets_under_min_isr > 0, fluss.replication.buckets_under_replicated > 0, or fluss.replication.failed_isr_updates_per_second > 0	affected buckets, replica placement, TabletServer health, and storage latency
fluss.request_path_pressure	all	warning, 0.84	fluss.requests.queue_size >= 100 and fluss.requests.errors_per_second > 0	queue and errors by TabletServer and request type, network, and disk pressure
fluss.write_expiration_cascade	all	critical, 0.91	fluss.requests.delayed_writes > 0 and fluss.requests.expired_writes_per_second > 0	affected tables, replica acknowledgement, storage, and producer retry state
fluss.tabletserver_resource_pressure	all	warning, 0.82	fluss.tablet_server.jvm.cpu.percent >= 90 and fluss.tablet_server.jvm.heap.utilization >= 85	per-server skew, GC, request queues, storage latency, and container limits

These rules provide incident-classification evidence; they do not replace single-series alerts. Missing or unresolved series are not interpreted as healthy or zero. A rule remains without sufficient evidence when its required signals cannot be collected.

The availability rule evaluates Prometheus series. Community protocol diagnostics can use a different observation time or target set for Coordinator and TabletServer evidence. If results differ, compare metric targets, labels, the latest collection time, and Coordinator API results.

MCP Resources and Tools

The MCP catalog provides seven read-only resources and six read-only tools.

Kind	Names	Purpose
Resources	monitoring_overview	Read dashboard and metric-linkage status
Resources	diagnostics_summary, diagnostics_history	Read current diagnostics and bounded historical risk evidence
Resources	metrics_catalog, mapping_pack_catalog	Read logical-metric and Prometheus mapping metadata
Resources	anomaly_rules	Read Enterprise multi-signal rule metadata
Resources	software_inventory	Read bounded Fluss software identity evidence
Tools	monitoring_overview, diagnostics_summary	Query monitoring context and current diagnostics
Tools	diagnostics_detail	Query bounded detail for one diagnostic category
Tools	metrics_query_resolve, mapping_pack_resolve	Resolve a logical key into a metric-source query and mapping pack
Tools	anomaly_rules	Read the Enterprise rule pack

Tools with inputs use the following contract.

Tool	Input	Required	Accepted values
diagnostics_detail	category	yes	cluster, replication, requests, storage, or runtime
metrics_query_resolve	logical_metric_key	yes	A logical key from the current Community Fluss metric catalog
metrics_query_resolve	query_mode	no	instant or range

diagnostics_history is bounded historical evidence delegated to the linked metric source. An empty result can be valid when no metric source is linked or the requested interval contains no samples. software_inventory returns only bounded version and identity data; it does not expose credentials or raw protocol frames.

The Community Fluss plugin does not provide a managed-alert-rule route, so MCP does not advertise an alerts/rules resource or tool. Diagnostic execution, catalog or data mutations, configuration, ACL, server-tag, and rebalance actions are also excluded from the read-only MCP surface.

Incident Workflows

Cluster availability or replication degradation

1. Verify metric linkage and the latest collection time in monitoring_overview.
2. Compare cluster and replication status in diagnostics_summary.
3. Open the relevant category with diagnostics_detail and inspect CoordinatorServer-, TabletServer-, and bucket-level evidence.
4. Resolve the anomaly's logical metrics as range queries over the same time window.
5. After restoring offline buckets or replicas, confirm that both metric and protocol evidence recover.

Request pressure or write expiration

1. Compare queue depth, error rate, delayed writes, and expired writes over the same time window.
2. Inspect skew by TabletServer and request type together with replication and storage diagnostics.
3. Distinguish an outcome-unknown accepted write from a confirmed failure before retrying producers.
4. Do not repeatedly increase concurrency or retry rate before identifying the bottleneck.

TabletServer resource pressure

1. Confirm that CPU and heap are simultaneously high on the same TabletServer.
2. Compare GC, request queues, error rate, storage latency, and per-server throughput.
3. Distinguish a hot server or placement imbalance from total capacity shortage.
4. After changing JVM or container limits or running a rebalance, use the Community Operations workspace preflight and read-back safeguards.

Troubleshooting

Symptom	Check in this order
EE anomaly or MCP entries are absent	Check the Enterprise license, anomaly.engine, mcp.gateway, plugin version, and contribution loading
Anomaly-rule query is denied	Check target-resource access, the viewer role, and fluss-plugin.anomaly.read
No anomaly rule can be evaluated	Check the Prometheus link, Coordinator and TabletServer targets, logical mappings, and latest collection time
Only some rules are unevaluated	Confirm that every logical key required by the rule exists in both the catalog and raw series
Protocol diagnostics and availability anomaly disagree	Compare observation times, metric target and label scope, and the Coordinator API server list
MCP diagnostics_detail is rejected	Use one of the five accepted category values and verify resource access
MCP metric resolution fails	Confirm that logical_metric_key exists in the current catalog and use instant or range for query_mode
Historical diagnostics are empty	Check the linked metric source, requested interval, and whether samples exist in that interval
Diagnostic execution or operational writes are absent from MCP	This is the intended read-only boundary; use the RBAC-protected Community Operations action

Data Protection and Limitations

• EE MCP descriptors do not expose credentials, certificate private keys, tokens, row or key payloads, raw protocol frames, or raw server errors.
• Anomaly rules do not infer whole-cluster health beyond the currently linked logical-metric evidence.
• An MCP result is not change approval or recovery execution. Perform every mutation through Community Operations and verify permission, dry run, confirmation, read-back, and audit results.
• For Fluss or fluss-go combinations outside the supported baseline, validate Community connectivity and diagnostic compatibility before relying on Enterprise analysis.

Edition Boundary

The anomaly rule pack, MCP descriptors, and Enterprise localization remain in the Enterprise overlay. The Community plugin remains independently installable and usable without these commercial contributions. EE does not duplicate CE operation routes; it adds analysis metadata through registered contribution points.