Kafka-UI with Avro - can't view messages due to "Fallback Serde was used"

[DavidBerryUK]

I am learning Kafka along with Kafka-UI, and while everything is mostly going well, I am encountering an issue with viewing messages encoded in AVRO format in Kafka-UI. Instead of seeing the correct message, I get a garbled one along with the message 'Fallback Serde was used'. Despite this, the schemas do seem to be present within Kafka-UI.

I have followed instructions from various sites and forums, but any tweak I make either breaks the schema connection or results in no improvement. Many of the connection configuration suggestions I have found seem to be invalid or meant for other drivers.

I suspect that the issue lies in registering the Avro converter with Kafka-UI, but I am not sure how to do this. Any help would be greatly appreciated.

Topic Message View within Kafka-UI

Schema view within Kafka-UI

docker-compose.yml

version: "3.8"
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.4.4
    container_name: demo1-zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 2181:2181
    volumes:
      - ./data/zookeeper_data:/var/lib/zookeeper/data
      - ./data/zookeeper_log:/var/lib/zookeeper/log
    networks:
      - Demo1Network

  kafka1:
    image: confluentinc/cp-kafka:7.4.4
    container_name: demo1-kafka1
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: demo1-zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://demo1-kafka1:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
    volumes:
      - ./data/kafka1_data:/var/lib/kafka/data
    networks:
      - Demo1Network

  kafka2:
    image: confluentinc/cp-kafka:7.4.4
    container_name: demo1-kafka2
    depends_on:
      - zookeeper
    ports:
      - 9093:9092
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: demo1-zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://demo1-kafka2:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
    volumes:
      - ./data/kafka2_data:/var/lib/kafka/data
    networks:
      - Demo1Network

  kafka3:
    image: confluentinc/cp-kafka:7.4.4
    container_name: demo1-kafka3
    depends_on:
      - zookeeper
    ports:
      - 9094:9092
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: demo1-zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://demo1-kafka3:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
    volumes:
      - ./data/kafka3_data:/var/lib/kafka/data
    networks:
      - Demo1Network

  kafka-ui:
    image: provectuslabs/kafka-ui
    container_name: demo1-kafka-u1
    ports:
      - "8095:8080"
    depends_on:
      - zookeeper
      - kafka1
      - kafka2
      - kafka3
      - schema-registry
    environment:
      DYNAMIC_CONFIG_ENABLED: "true"
      KAFKA_CLUSTERS_0_NAME: demo-kafka-cluster
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka1:9092
      KAFKA_CLUSTERS_0_ZOOKEEPER: zookeeper:2181
      KAFKA_CLUSTERS_0_KAFKASTORE: true
      KAFKA_CLUSTERS_0_SCHEMAREGISTRY: http://demo1-schema-registry:8080/apis/ccompat/v6
    networks:
      - Demo1Network

  schema-registry:
    container_name: demo1-schema-registry
    image: apicurio/apicurio-registry-mem:2.6.0.Final
    depends_on:
      - zookeeper
      - kafka1
      - kafka2
      - kafka3
    environment:
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zookeeper:2181
      SCHEMA_REGISTRY_HOST_NAME: demo1-schema-registry
      SCHEMA_REGISTRY_LISTENERS: http://demo1-schema-registry:8080
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://demo1-kafka1:9092,PLAINTEXT://demo1-kafka2:9093,PLAINTEXT://demo1-kafka3:9094
    ports:
      - 9090:8080
    networks:
      - Demo1Network

  connect:
    image: debezium/connect:latest
    container_name: demo1-connect
    depends_on:
      - zookeeper
      - kafka1
      - kafka2
      - kafka3
      - schema-registry
    ports:
      - "8084:8083"
    environment:
      BOOTSTRAP_SERVERS: demo1-kafka1:9092,demo1-kafka2:9093,demo1-kafka3:9094
      GROUP_ID: 1
      CONFIG_STORAGE_TOPIC: debezium_configs
      OFFSET_STORAGE_TOPIC: debezium_offsets
      STATUS_STORAGE_TOPIC: debezium_status
      KEY_CONVERTER_SCHEMAS_ENABLE: "false"
      VALUE_CONVERTER_SCHEMAS_ENABLE: "false"
      CONFIG_ACTION_RETRY_MAX_DURATION: "30"
      ENABLE_APICURIO_CONVERTERS: true
      KEY_CONVERTER: io.apicurio.registry.utils.converter.AvroConverter
      VALUE_CONVERTER: io.apicurio.registry.utils.converter.AvroConverter
      CONNECT_KEY_CONVERTER: io.apicurio.registry.utils.converter.AvroConverter
      CONNECT_KEY_CONVERTER_APICURIO_REGISTRY_AUTO-REGISTER: true
      CONNECT_KEY_CONVERTER_APICURIO_REGISTRY_FIND-LATEST: true
      CONNECT_VALUE_CONVERTER: io.apicurio.registry.utils.converter.AvroConverter
      CONNECT_KEY_CONVERTER_APICURIO_REGISTRY_URL: http://demo1-schema-registry:8080/apis/registry/v2
      CONNECT_VALUE_CONVERTER_APICURIO_REGISTRY_URL: http://demo1-schema-registry:8080/apis/registry/v2
      CONNECT_VALUE_CONVERTER_APICURIO_REGISTRY_AUTO-REGISTER: true
      CONNECT_VALUE_CONVERTER_APICURIO_REGISTRY_FIND-LATEST: true
      CONNECT_SCHEMA_NAME_ADJUSTMENT_MODE: avro
    volumes:
      - ./data/debezium_data:/var/lib/debezium
    networks:
      - Demo1Network
    extra_hosts:
      - "host.docker.internal:host-gateway"

networks:
  Demo1Network:
    driver: bridge

Connector configuration

{
  "name": "orders-connector",
  "config": {
    "connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
    "database.hostname": "host.docker.internal",
    "database.port": "1433",
    "database.user": "sa",
    "database.names": "KafkaDemoSource",
    "database.password": "YourStrong!Passw0rd",
    "database.encrypt": "false",
    "database.server.name": "DESKTOP-DELL",
    "topic.prefix": "orders",
    "table.include.list": "dbo.customers, dbo.depots",
    "database.history.kafka.bootstrap.servers": "demo1-kafka1:9092,demo1-kafka2:9093,demo1-kafka3:9094",
    "schema.history.internal.kafka.topic": "topic_history",
    "schema.history.internal.kafka.bootstrap.servers": "demo1-kafka1:9092,demo1-kafka2:9093,demo1-kafka3:9094",
    "key.converter.apicurio.registry.url": "http://demo1-schema-registry:8080/apis/registry/v2",
    "key.converter.apicurio.registry.auto-register": "true",
    "key.converter.apicurio.registry.find-latest": "true",
    "value.converter": "io.apicurio.registry.utils.converter.AvroConverter",
    "value.converter.apicurio.registry.url": "http://demo1-schema-registry:8080/apis/registry/v2",
    "value.converter.apicurio.registry.auto-register": "true",
    "value.converter.apicurio.registry.find-latest": "true",
    "schema.name.adjustment.mode": "avro"
  }
}

[Kamapcuc]

Hello, David!
Did you solve the problem?

[MiLabuda]

Hello,
I'm also interested if you've managed to solve the problem because i encounter something similar currently

[DavidBerryUK]

I gave up to be honest. I was following multiple blogs, configurations and settings and almost got it working. But it felt far too fragile for me to recommend put it in a production environment. Basically, I wouldn't have met my house on it and I certainly wouldn't want to try and solve a production problem at 4 am using this technology as powerful as it is I would not use the compressed format.

[Haarolean]

Please consider that this repo is not maintained (#4255), the project now lives at https://github.com/kafbat/kafka-ui. Happy to help with your avro issues there (discussions/issues/discord)