Kafka Plugin

The Kafka plugin registers an Apache Kafka cluster as one Konduo resource and exposes broker, controller, topic, partition, and consumer group state as operational evidence.

Highlights

• Uses bootstrap servers as seeds and validates broker listeners and cluster id from Kafka metadata.
• Exposes KRaft/controller roles, broker inventory, topic/partition state, consumer group lag, and offset evidence.
• Connects JMX/exporter-style metrics through a Prometheus mapping pack.
• In mixed collection mode, broker metrics stay external while Konduo collects consumer group lag through the Kafka Admin API.

Before Registration

• Prepare bootstrap servers, security protocol, SASL, TLS, and timeout settings.
• Confirm that advertised broker listeners are reachable from the Konduo backend.
• For managed consumer group lag collection, prepare Admin API access and a remote-write target.
• When Prometheus is the remote-write target, run Prometheus with --web.enable-remote-write-receiver.

Operations Tips

• A down bootstrap broker does not always mean the whole cluster is unavailable if discovered broker endpoints still work.
• Read consumer lag with total lag, max lag, member state, and offset movement together.
• Topic partition changes, reassignment, throttles, ACLs, and quotas run behind permission and confirmation boundaries.
• Message search is an operational helper. Even when scanning a whole topic, keep rate limiting and cancellation in mind; it is not a bulk export or ETL substitute.
• When inspecting transactional messages, distinguish read_committed from the full visibility mode. read_committed shows committed transactional records and non-transactional records only.

Operator Views

• The broker inventory shows node id, listener, KRaft role, controller state, partition and replica counts, data size, and cordon state together.
• The topic view brings partition count, replica factor, internal-topic status, config, ACLs, and message search/produce jobs into one operational surface.
• The reassignment view calculates broker drain, rebalance, and replica-change plans as dry runs before showing expected data movement and target brokers.
• The consumer group view distinguishes classic consumer groups, share groups, Connect, Schema Registry, and similar group kinds so lag and member state are easier to interpret.
• The security view exposes SCRAM users, ACLs, quotas, delegation tokens, and transactional id evidence within the operator permission boundary.
• Message view, streaming, and search expose a transaction visibility option. Selecting read_committed excludes aborted or still-uncommitted transactional records while non-transactional records remain visible.
• Topic detail exposes related ACLs and active producer evidence for the selected topic. Active producer evidence helps narrow transaction incidents by producer id/epoch, coordinator epoch, last sequence, and current transaction start offset.
• The Transactional ID view shows transactional id state, coordinator, and producer id evidence with row details, navigation to related ACLs, and active-transaction action workflows.
• SCRAM user rows can navigate to the ACL list for that principal, and creating ACLs from that context pre-fills the principal.

Message Inspection And Production

• Message view, streaming, and search configure key/value deserialization separately from transaction visibility.
• Messages from classic offset commit/group metadata records in __consumer_offsets and transaction metadata in __transaction_state are decoded automatically into group, topic/partition, committed offset, member subscription/assignment, transaction state, and producer details.
• Internal-topic tombstones are distinguished from empty byte values. Parsed keys and values render as complete JSON rather than forced binary strings, while unknown Kafka coordinator record types and malformed messages retain raw key/value hex alongside an explicit decode status.
• read_committed hides aborted or not-yet-committed transactional records. It is useful when validating why a producer reported writes that consumers cannot see.
• Custom single-message production lets the operator provide the key, value, and headers directly. Headers use one key=value entry per line and are produced as entered.
• Template production repeatedly renders built-in schemas or contribution-provided schemas. Operators configure JSON/YAML output, key strategy, message count, rate limit, and dry-run preview together.
• Completed message produce jobs can be cleared from the job queue. This only cleans Konduo job history; it does not delete Kafka topic data.
• Search result payloads expire independently from lightweight job status. A status with result_available=false and result_expired=true means the job is known but its in-memory matches have aged out or were evicted under the aggregate byte limit; it does not mean Kafka messages were deleted.
• Successful produce history has only a short visibility grace because the actual result is already in the Kafka topic. Failed/cancelled diagnostics are retained longer. All job history is process-local and disappears on plugin restart.

Diagnostics And Alerts

• Diagnostics separate broker/controller state, partition safety, under-replicated or offline partitions, consumer lag, reassignment state, and metric coverage.
• Cluster-wide defaults such as min.insync.replicas are evaluated once per Kafka cluster. If brokers report different values, diagnostics show one cluster consistency warning; topic-level overrides are still evaluated per topic.
• Kafka availability is based on metadata and advertised-listener reachability; JMX/exporter metrics are supporting evidence for load and capacity risk.
• Alert rules focus on Kafka-domain risks such as offline partitions, under-replicated partitions, controller anomalies, request-handler pressure, and consumer lag.
• If the metric source is missing or the mapping pack does not match the exporter, the plugin should report unavailable or partial evidence instead of assuming a healthy state.

Management Boundary

• Broker drain and rebalance plans must exclude offline and cordoned brokers from new replica placement.
• Traffic rebalance planning uses only metric evidence scoped to this Kafka resource. When scoped topic or broker evidence is unavailable, the plan is rejected instead of using global metric-source series.
• When Kafka 4.3 or later exposes cordoned.log.dirs=*, drain can cordon the target broker first; uncordon remains an explicit operator action after the broker is empty.
• On Kafka 4.3.1, manually submitted reassignment plans may still target a cordoned broker. Konduo drain/rebalance plans exclude cordoned brokers from new targets, but externally submitted Kafka plans still need operator review.
• If a broker is already empty, drain is no longer useful; if it is cordoned, restore cordon before using it as a placement target again.
• Broker settings whose defaults are null, such as cordoned.log.dirs, should be restored by deleting/resetting the dynamic config entry rather than writing an empty value.
• Message search and produce jobs are operational helpers for a topic, not a replacement for bulk data recovery or stream processing.
• ACL, quota, SCRAM, and delegation-token changes directly affect Kafka security and must stay within RBAC, confirmation, and audit boundaries.

Security And Transaction Analysis

• Topic, consumer group, and transactional id ACL lists appear only when Kafka authorizer APIs are available. Treat permission errors and disabled authorizers differently from a true empty ACL list.
• Read ACL resource name, pattern type, principal, host, operation, and permission together; prefixed or wildcard ACLs can affect resources that do not have a literal ACL row.
• Transactional ID detail changes the available action: active or open transactions can be action candidates, while completed commit/abort states are not abort targets.
• Producer fencing is a strong action that forces out the current producer generation for a transactional id. Use it after confirming application restart, duplicate producer, or transaction hang evidence.

Example Workflows

• Before replacing a broker, inspect the broker inventory for partition/replica counts and data size, then review a broker-drain dry run.
• For consumer delay, compare group state, member count, partition lag, max lag, and recent offset movement.
• Before expanding a topic, compare partition changes, replica factor, broker distribution, and traffic evidence to avoid creating a new imbalance.
• For suspected authentication or authorization issues, inspect principal ACLs, SCRAM state, quotas, and transactional id evidence in that order.
• For suspected stuck transactions, inspect Transactional ID state and producer id, compare topic active producer evidence and related ACLs, then decide whether abort or fencing is appropriate.

---

Kafka Enterprise Overlay

The Kafka Enterprise overlay adds MCP descriptors and anomaly rule metadata on top of the Community Kafka plugin.

Highlights

• Exposes monitoring overview, diagnostics, consumer groups, security operations, metrics mapping, and anomaly rules through the MCP catalog.
• Evaluates anomaly rules in declarative_metric mode by resolving Kafka logical metrics through metric-source mappings.
• Mutating workflows such as reassignment, broker drain, replica changes, topic changes, and message actions keep the existing Konduo API, RBAC, and audit boundaries.
• Treats bootstrap addresses as seeds and interprets broker inventory with advertised listener evidence.
• References Community-backed broker cordon, message inspection/production, transactional ID, active producer, and topic/group ACL evidence from MCP and incident-analysis workflows.

Common Uses

1. Inspect Kafka operational state through MCP resources and tools
2. Explore broker, partition, and consumer group state in read-oriented workflows
3. Review the Enterprise anomaly rule catalog
4. Connect dashboards and diagnostics to agent workflows

Before Registration

• Kafka connectivity and permissions follow the Community Kafka plugin settings.
• The MCP gateway and related authorization profile must be enabled before MCP descriptors are usable.
• Anomaly rules operate in environments where the Enterprise analysis engine is enabled.

Operations Tips

• Use MCP tools for operational lookup and analysis assistance; configuration changes should still follow Konduo UI and approved procedures.
• Anomaly rules do not replace alert rules; treat them as supporting signals for correlation and incident analysis.
• For consumer lag analysis, review total lag, max lag, member state, offset movement, and controller state together.
• In message view/search, read_committed excludes aborted or not-yet-committed transactional records, which helps explain producer/consumer visibility differences.
• Messages from classic offset commit/group metadata records in __consumer_offsets and transaction metadata in __transaction_state are decoded automatically into group, topic/partition, committed offset, member subscription/assignment, transaction state, and producer details.
• Internal-topic tombstones are distinguished from empty byte values. Parsed keys and values render as complete JSON rather than forced binary strings, while unknown Kafka coordinator record types and malformed messages retain raw key/value hex alongside an explicit decode status.
• Custom single-message production lets the operator provide the key, value, and headers directly. Headers use one key=value entry per line and are produced as entered.
• Template production repeatedly renders built-in schemas or contribution-provided schemas. Operators configure JSON/YAML output, key strategy, message count, rate limit, and dry-run preview together.
• Completed message produce jobs can be cleared from the job queue. This only cleans Konduo job history; it does not delete Kafka topic data.
• Search result payloads expire independently from lightweight job status. A status with result_available=false and result_expired=true means the job is known but its in-memory matches have aged out or were evicted under the aggregate byte limit; it does not mean Kafka messages were deleted.
• Successful produce history has only a short visibility grace because the actual result is already in the Kafka topic. Failed/cancelled diagnostics are retained longer. All job history is process-local and disappears on plugin restart.
• Kafka 4.3+ drain with cordoned.log.dirs=* improves placement safety, but uncordon remains an explicit recovery action after the broker is empty.

Diagnostics And Alerts

• Diagnostics separate broker/controller state, partition safety, under-replicated or offline partitions, consumer lag, reassignment state, and metric coverage.
• Cluster-wide defaults such as min.insync.replicas are evaluated once per Kafka cluster. If brokers report different values, diagnostics show one cluster consistency warning; topic-level overrides are still evaluated per topic.
• Kafka availability is based on metadata and advertised-listener reachability; JMX/exporter metrics are supporting evidence for load and capacity risk.
• Alert rules focus on Kafka-domain risks such as offline partitions, under-replicated partitions, controller anomalies, request-handler pressure, and consumer lag.
• If the metric source is missing or the mapping pack does not match the exporter, the plugin should report unavailable or partial evidence instead of assuming a healthy state.

Management Boundary

• Broker drain and rebalance plans must exclude offline and cordoned brokers from new replica placement.
• Traffic rebalance planning uses only metric evidence scoped to this Kafka resource. When scoped topic or broker evidence is unavailable, the plan is rejected instead of using global metric-source series.
• When Kafka 4.3 or later exposes cordoned.log.dirs=*, drain can cordon the target broker first; uncordon remains an explicit operator action after the broker is empty.
• On Kafka 4.3.1, manually submitted reassignment plans may still target a cordoned broker. Konduo drain/rebalance plans exclude cordoned brokers from new targets, but externally submitted Kafka plans still need operator review.
• If a broker is already empty, drain is no longer useful; if it is cordoned, restore cordon before using it as a placement target again.
• Broker settings whose defaults are null, such as cordoned.log.dirs, should be restored by deleting/resetting the dynamic config entry rather than writing an empty value.
• Message search and produce jobs are operational helpers for a topic, not a replacement for bulk data recovery or stream processing.
• ACL, quota, SCRAM, and delegation-token changes directly affect Kafka security and must stay within RBAC, confirmation, and audit boundaries.

Operational Scenarios

• Before and after broker drain or rebalance, compare broker inventory, controller state, partition distribution, and metric coverage through MCP or diagnostics routes.
• During drain/rebalance analysis, confirm that offline and cordoned brokers are excluded from new replica targets, and review externally submitted manual reassignment plans separately.
• For consumer-group incidents, distinguish classic groups, share groups, connect, and schema-registry style groups before interpreting lag and member state.
• When security failures are suspected, inspect SCRAM, ACLs, quotas, delegation tokens, and transactional ID evidence in order, then apply changes through audited action routes.
• For suspected stuck transactions, compare Transactional ID state, topic active producer evidence, and related ACLs before deciding whether abort or producer fencing is appropriate.
• If exporter label differences change metric keys at a customer site, use EE mapping pack import/export to adjust the site-specific mapping.

Boundaries

• Kafka topic, ACL, reassignment, and message-producing mutations must keep the existing Konduo API, RBAC, confirmation, and audit boundaries instead of bypassing them through MCP descriptors.
• Kafka 4.3+ cordon evidence and drain/rebalance checks are safety signals; plugin behavior should not hide limitations in Kafka's own reassignment behavior.
• ACL, SCRAM, quota, delegation-token, and transactional-id changes should run through approved action routes and audit logs, not through Enterprise analysis automation.