Kafka Streams API

Our plan

Lecture 1. Kafka (brief reminder) and Data Streaming Application configuration. Stateless transformations Transformations with local state Lecture 2. Stream-table dualism and Table joins Time and window operations

Tables vs Streams

Tables vs Streams

Tables vs Streams

Table-table join

Table-Table join

Table-Table join

Table-Table join topology

Rewriting the totalling app using KTable

KTable<String, Long> totals = input.groupByKey().aggregate(
    () -> 0L,
    (k, v, a) -> a + Math.round(v.getAmount() * v.getOdds()),
    Materialized.with(Serdes.String(), Serdes.Long())
);

$kafka-topics --zookeeper localhost --describe

Topic:
table2-demo-KSTREAM-AGGREGATE-STATE-STORE-0000000001-changelog
PartitionCount:10
ReplicationFactor:1
Configs:cleanup.policy=compact

Get a table of match scores

KStream<String, Score> scores =
    eventScores.flatMap((k, v) ->
        Stream.of(Outcome.H, Outcome.A).map(o ->
            KeyValue.pair(String.format("%s:%s", k, o), v))
            .collect(Collectors.toList()))
    .mapValues(EventScore::getScore);

KTable<String, Score> tableScores =
    scores.groupByKey(Grouped.with(...). reduce((a, b) -> b);

$kafka-topics --zookeeper localhost --describe

table2-demo-KSTREAM-REDUCE-STATE-STORE-0000000006-repartition
table2-demo-KSTREAM-REDUCE-STATE-STORE-0000000006-changelog

Demo: Combining the amount of bets with the current account

KTable<String, String> joined =
    totals.join(tableScores,
            (total, eventScore) ->
                String.format("(%s)\t%d", eventScore, total));

Co-partitioning

Number of partitions mismatch

Participle algorithm mismatch

GlobalKTable

GlobalKTable<...> global = streamsBuilder.globalTable("global", ...);

Foreign Key Joins: join + ForeignKeyExtractor

Operations on Streams and Tables: summary

Types of joins: Table-Table

Types of joins: Table-Table

Types of joins: Table-Table

Types of joins: Stream-Table

Types of joins: Stream-Stream

Our plan

Lecture 1. Kafka (brief reminder) and Data Streaming Application configuration. Stateless transformations Transformations with local state Lecture 2. Stream-table dualism and table joins Time and window operations

Save Timestamped values to  RocksDB

static Bytes toStoreKeyBinary(byte[] serializedKey,
                              long timestamp,
                              int seqnum) {
    ByteBuffer buf = ByteBuffer.allocate(
                                serializedKey.length
                                + TIMESTAMP_SIZE
                                + SEQNUM_SIZE);
    buf.put(serializedKey);
    buf.putLong(timestamp);
    buf.putInt(seqnum);
    return Bytes.wrap(buf.array());
}

Quick retrieval of key values for a time range

Demo: Windowed Joins

Time, Forward!

KStream<String, Bet> bets = streamsBuilder.stream(BET_TOPIC,
    Consumed.with(
            Serdes...)
            .withTimestampExtractor(

                (record, previousTimestamp) ->
                    ((Bet) record.value()).getTimestamp()

            ));

Demo: Windowed Joins

Score current = Optional.ofNullable(stateStore.get(key))
                .orElse(new Score());
stateStore.put(key, value.getScore());

Outcome currenOutcome =
    value.getScore().getHome() > current.getHome()
    ?
    Outcome.H : Outcome.A;

Demo: Windowed Joins

KStream<String, String> join = bets.join(outcomes,
    (bet, sureBet) ->

    String.format("%s %dms before goal",
                bet.getBettor(),
                sureBet.getTimestamp() - bet.getTimestamp()),
                JoinWindows.of(Duration.ofSeconds(1)).before(Duration.ZERO),
                StreamJoined.with(Serdes....
    ));

Tumbling window

TimeWindowedKStream<..., ...> windowed =
    stream.groupByKey()
        .windowedBy(TimeWindows.of(Duration.ofSeconds(20)));

Tumbling window

TimeWindowedKStream<..., ...> windowed =
    stream.groupByKey()
        .windowedBy(TimeWindows.of(Duration.ofSeconds(20)));

KTable<Windowed<...>, Long> count = windowed.count();

/*
* Windowed<K> interface:
* - K key()
* - Window window()
* -- Instant startTime()
* -- Instant endTime()
*/

Hopping Window

TimeWindowedKStream<..., ...> windowed =
    stream.groupByKey()
        .windowedBy(TimeWindows.of(Duration.ofSeconds(20))
                        .advanceBy(Duration.ofSeconds(10)));

Session Window

SessionWindowedKStream<..., ...> windowed =
    stream.groupByKey()
        .windowedBy(SessionWindows.with(Duration.ofMinutes(5)));

Window Retention time vs. Grace Time

Sometimes you don’t need windows, but Punctuator

class MyTransformer implements Transformer<...> { @Override public void init(ProcessorContext context) { context.schedule( Duration.ofSeconds(10), PunctuationType.WALL_CLOCK_TIME, timestamp->{. . .}); }

Our plan

Lecture 1. Kafka (brief reminder) and Data Streaming Application configuration. Stateless transformations Transformations with local state Lecture 2. Stream-table dualism and table joins Time and window operations

Kafka Streams in Action

William Bejeck, “Kafka Streams in Action, Second Edition”, Spring 2023? The first edition is out of date!

Kafka: The Definitive Guide

Gwen Shapira, Todd Palino, Rajini Sivaram, Krit Petty November 2021

Other sources

Communities, conferences

Conclusions

That’s all!