Apache Kafka

Our plan

What is a log and why is it important What is Kafka and what can it do What is the streaming architecture and what are stream processors capable of? Kafka and JavaScript How to Try Kafka Today

What is a log?

 — What is our life?

 — What is our life?

How to automate a warehouse?

All is well, until…​

What could it be?

What could it be?

Stolen (broken, thrown away, etc.) Lost: it’s somewhere nearby THIS IS YOUR DUMB PROGRAM NOT WORKING

What are we going to do?

How should we actually design the warehouse automation?

"A database is a warehouse"

How should we actually design the warehouse automation?

	"A database is a warehouse"
	The database reflects the processes. Current state is the result of the execution of processes

Warehouse Ledger

Warehouse Ledger

Warehouse Ledger

What if we made a mistake with the accounting?

What questions can already be answered?

Investigating the incident

What requirements can be easily implemented?

Solution architecture: log does not work alone

Intermediate conclusions

Our plan

What is a log and why is it important What is Kafka and what can it do What is streaming architecture and what are stream processors capable of? Kafka and JavaScript How to try Kafka today

Our plan

What is a log and why is it important What is Kafka and what can it do General information How a cluster works How writing works How reading works Retention and compactification What is streaming architecture and what are stream processors capable of? Kafka and JavaScript How to try Kafka today

Kafka is

distributed scalable fault-tolerant log millions of records per second

In Kafka you can

Write something to a named log (topic) Read entries from the topic in FIFO order (within the partition) Commit the offset of the latest processed entry

You can’t do it in Kafka

Erase a record Edit a record Extract a random record from the log except by offset

Our plan

What is a log and why is it important What is Kafka and what can it do General information How a cluster works How writing works How reading works Retention and compactification What is streaming architecture and what are stream processors capable of? Kafka and JavaScript How to try Kafka today

Kafka Cluster: Brokers and Zookeeper

Topics, partitions and messages

Topics, partitions and messages

Topics, partitions and messages

Our plan

What is a log and why is it important What is Kafka and what can it do General information How a cluster works How writing works How reading works Retention and compactification What is streaming architecture and what are stream processors capable of? Kafka and JavaScript How to try Kafka today

Anatomy of a message

Anatomy of a message

// hash the keyBytes to choose a partition
return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;

Throughput vs Latency

Writing to Kafka

Writing to Kafka

Writing to Kafka

Our plan

What is a log and why is it important What is Kafka and what can it do General information How a cluster works How writing works How reading works Retention and compactification What is streaming architecture and what are stream processors capable of? Kafka and JavaScript How to try Kafka today

Reading from Kafka

Reading from Kafka

Reading from Kafka

Reading from Kafka

Offset Commit

Offset Commit

Offset Commit

Offset Commit

Offset Commit

Offset Commit

Offset Commit

Our plan

What is a log and why is it important What is Kafka and what can it do General information How a cluster works How writing works How reading works Retention and compactification What is streaming architecture and what are stream processors capable of? Kafka and JavaScript How to try Kafka today

How Retention works

How Retention works

How Retention works

How Retention works

How Retention works

How Retention works

Compactification of topics

Our plan

What is a log and why is it important What is Kafka and what can it do What is streaming architecture and what are stream processors capable of? Kafka and JavaScript How to try Kafka today

Stream data processing architecture

Existing stream processing frameworks

When I am asked which streaming framework to use

When I am asked which streaming framework to use

Stateless Transformation

KStream<String, String> stream =  streamsBuilder.stream(
     SRC_TOPIC, Consumed.with(Serdes.String(), Serdes.String());

Stateless Transformation

KStream<String, String> upperCasedStream =
    stream.mapValues(String::toUpperCase);

Stateless Transformation

upperCasedStream.to(SINK_TOPIC,
     Produced.with(Serdes.String(), Serdes.String());

Three lines of code, and what’s the big deal?

Adding nodes

Limited only by the number of partitions

Magic of Stateful Transformation

Changes are replicated to a topic!

Partitioning and local state

Partitioning and local state

Partitioning and local state

Partitioning and local state

Партиционирование и local state

Partitioning and local state

Partitioning and local state

Partitioning and local state

What else can streams do?

Aggregate data in time windows

Aggregate data related to the work session

KSQL

CREATE STREAM pageviews_enriched AS
  SELECT pv.viewtime,
         pv.userid AS userid,
         pv.pageid,
         pv.timestring,
         u.gender,
         u.regionid,
         u.interests,
         u.contactinfo
  FROM pageviews_transformed pv
  LEFT JOIN users_5part u ON pv.userid = u.userid
  EMIT CHANGES;

KSQL

CREATE TABLE pageviews_per_region_per_30secs AS
  SELECT regionid,
         count(*)
  FROM pageviews_enriched
  WINDOW TUMBLING (SIZE 30 SECONDS)
  WHERE UCASE(gender)='FEMALE' AND LCASE(regionid)='region_6'
  GROUP BY regionid
  EMIT CHANGES;

Where are streaming systems needed?

Things to keep in mind

If you need to store data all the time

Nikita Salnikov-Tarnovsky //https://2019.jokerconf.com/2019/talks/2qw2ljhlfoeiipjf0gfzzb/[Streaming application is not only code, but also 3-4 years of support in the product]

Our plan

What is a log and why is it important What is Kafka and what can it do What is streaming architecture and what are stream processors capable of? Kafka and JavaScript How to try Kafka today

Kafka and Javascript

Kafka and Javascript

kafka-node

node-rdkafka

Converting/output stage on node.js?

Converting/output stage on node.js?

Nodefluent node-sinek — another client kafka-streams — KStreams re-implementation for node.js kafka-connect — re-implementation of kafka-connect

KSQL + Serverless

Our plan

What is a log and why is it important What is Kafka and what can it do What is streaming architecture and what are stream processors capable of? Kafka and JavaScript How to try Kafka today

You decided to try Kafka. Where to start?

It’s hard to run Kafka in production

Grigory Koshelev When everything went according to Kafka How to cook Kafka so it doesn’t burn

Kafka: The Definitive Guide

Gwen Shapira, Todd Palino, Rajini Sivaram, Krit Petty November 2021

Сообщества, конференции

Telegram

Meetup in Moscow

That’s all!