kafka-streams-outer-join-stream-stream

Kafka Streams Outer Join Stream Stream

This module demonstrates the following:

• The usage of the Kafka Streams DSL, including outerJoin() between KStream and KStream, selectKey() and peek().
• The usage of sliding time windows.
• Unit testing using Topology Test Driver.

In this module, records of type <String, KafkaPerson> are streamed from two topics named PERSON_TOPIC and PERSON_TOPIC_TWO. The following tasks are performed:

1. Join the records on the last name within a 5-minute join window and a 1-minute grace period for delayed records.
2. Build a new KafkaJoinPersons object that holds both persons. If no person is matched, a value holding the left or right person is still emitted as an outer join is performed.
3. Write the resulting KafkaJoinPersons objects to a new topic named PERSON_OUTER_JOIN_STREAM_STREAM_TOPIC.

Requirements

To compile and run this demo, you will need the following:

• Java 21
• Maven
• Docker

Running the Application

To run the application manually, please follow the steps below:

• Start a Confluent Platform in a Docker environment.
• Produce records of type <String, KafkaPerson> to topics named PERSON_TOPIC and PERSON_TOPIC_TWO. You can use the producer person to do this.
• Start the Kafka Streams.

To run the application in Docker, please use the following command:

docker-compose up -d

This command will start the following services in Docker:

• 1 Kafka broker KRaft
• 1 Schema registry
• 1 Control Center
• 1 producer Person
• 1 Kafka Streams Outer Join Stream Stream