SUMMARY.adoc

Summary

1. Introduction

2. Kafka Streams — Stream Processing Library on Apache Kafka

3. Why Kafka Streams

4. Multi-Instance Kafka Streams Applications

5. Exactly-Once Support (EOS)

6. KafkaStreams, StreamThreads, StreamTasks and StandbyTasks

Demos

1. Creating Topology with State Store with Logging Enabled

Stateful Stream Processing

1. Stateful Stream Processing

2. StateStore

   1. KeyValueStore

   2. SessionStore

   3. WindowStore

3. Stores

4. Materialized

5. StoreSupplier

   1. KeyValueBytesStoreSupplier

   2. SessionBytesStoreSupplier

   3. WindowBytesStoreSupplier

6. StoreBuilder

7. TimestampedKeyValueStore

8. ReadOnlyKeyValueStore

Stream Processing with Kafka Streams

1. KafkaStreams

2. Topology

3. TopologyDescription

   1. Node

   2. Processor

   3. Sink

   4. Source

4. StreamsConfig

5. Configuration Properties

6. Logging

7. KafkaClientSupplier

High-Level Streams DSL

1. High-Level Streams DSL

2. StreamsBuilder

3. KStream

4. KTable

5. GlobalKTable

6. KGroupedStream

   1. SessionWindowedKStream

   2. TimeWindowedKStream

7. KGroupedTable

8. Consumed

9. Produced

10. Grouped

11. Joined

12. Printed

13. KeyValueMapper

14. ValueJoiner

15. ValueTransformer

    1. ValueTransformerSupplier

16. ValueTransformerWithKey

17. Transformer

18. TransformerSupplier

19. Windows

    1. JoinWindows

    2. TimeWindows

    3. UnlimitedWindows

20. Window

21. WindowedSerdes

    1. Windowed

    2. TimeWindowedSerializer

22. Scala API for Kafka Streams

    1. ImplicitConversions

    2. Serdes

    3. Consumed

    4. Produced

    5. Grouped

    6. Materialized

Low-Level Processor API

1. Low-Level Processor API

2. Processor Contract — Stream Processing Node

   1. AbstractProcessor — Base for Stream Processors

3. ProcessorContext

   1. TaskId

   2. To

4. Punctuator Contract — Scheduled Periodic Actions

5. Cancellable

6. ProcessorSupplier Contract

7. StreamPartitioner

8. TopicNameExtractor — Dynamic Routing of Output Records

   1. RecordContext — Record Metadata

9. TimestampExtractor Contract

   1. WallclockTimestampExtractor

   2. ExtractRecordMetadataTimestamp

   3. FailOnInvalidTimestamp

10. PartitionGrouper Contract

    1. DefaultPartitionGrouper

11. StateRestoreCallback

    1. BatchingStateRestoreCallback

12. StateRestoreListener

    1. AbstractNotifyingBatchingRestoreCallback

    2. AbstractNotifyingRestoreCallback

Monitoring Kafka Streams Applications

1. StateListener — KafkaStreams State Listener

2. CacheFlushListener

3. StreamsMetrics

   1. StreamsMetricsImpl

   2. StreamsMetricsThreadImpl

4. TaskMetrics

5. StoreChangeLogger

6. DelegatingStateRestoreListener

Testing

1. TopologyTestDriver

Internals of Kafka Streams

Low-Level Stream Processing Graph

1. InternalTopologyBuilder

2. AbstractNode

   1. Processor

   2. Sink

   3. Source

3. NodeFactory

   1. ProcessorNodeFactory

   2. SinkNodeFactory

   3. SourceNodeFactory

4. InternalTopologyBuilder.TopologyDescription

5. GlobalStore

6. StateStoreFactory

High-Level Stream Processing Graph

1. InternalStreamsBuilder

2. StreamsGraphNode

   1. BaseJoinProcessorNode

   2. BaseRepartitionNode

3. ProcessorParameters

4. GlobalStoreNode

5. GroupedTableOperationRepartitionNode

   1. GroupedTableOperationRepartitionNodeBuilder

6. KTableKTableJoinNode

7. OptimizableRepartitionNode

8. ProcessorGraphNode

9. StatefulProcessorNode

10. StateStoreNode

11. StreamSinkNode

12. StreamSourceNode

13. StreamStreamJoinNode

14. StreamTableJoinNode

15. TableProcessorNode

16. TableSourceNode

Processors and ProcessorSuppliers

1. KStreamAggProcessorSupplier

2. KStreamBranch

3. KStreamFilterProcessor

   1. KStreamFilter

4. KStreamJoinWindowProcessor

5. KStreamPeek

6. KStreamPassThrough

7. KStreamSessionWindowAggregateProcessor

   1. KStreamSessionWindowAggregate

8. KStreamTransformProcessor

   1. KStreamTransform

9. KStreamTransformValuesProcessor

   1. KStreamTransformValues

10. KStreamWindowAggregateProcessor

    1. KStreamWindowAggregate

11. KTableSourceProcessor

    1. KTableSource

12. KTableSuppressProcessor

13. KTableValueGetter

14. KTableValueGetterSupplier

    1. KTableMaterializedValueGetterSupplier

    2. KTableSourceValueGetterSupplier

    3. KTableKTableAbstractJoinValueGetterSupplier

Internals of State Stores

1. WrappedStateStore

   1. CachingKeyValueStore

   2. CachingSessionStore

   3. CachingWindowStore

   4. ChangeLoggingKeyValueBytesStore

   5. ChangeLoggingSessionBytesStore

   6. ChangeLoggingWindowBytesStore

   7. MeteredKeyValueStore

   8. MeteredSessionStore

   9. MeteredWindowStore

   10. RocksDBSessionStore

   11. RocksDBWindowStore

   12. InMemoryKeyValueStore

   13. InMemorySessionStore

   14. InMemoryWindowStore

   15. MemoryLRUCache

   16. MeteredKeyValueBytesStore

   17. RocksDBStore

   18. RocksDBTimestampedStore

2. AbstractStoreBuilder

   1. KeyValueStoreBuilder

   2. SessionStoreBuilder

   3. TimestampedKeyValueStoreBuilder

   4. TimestampedWindowStoreBuilder

   5. WindowStoreBuilder

3. InMemorySessionBytesStoreSupplier

4. RocksDbKeyValueBytesStoreSupplier

5. RocksDbSessionBytesStoreSupplier

6. RocksDbWindowBytesStoreSupplier

7. KeyValueToTimestampedKeyValueByteStoreAdapter

8. WindowToTimestampedWindowByteStoreAdapter

9. SegmentedBytesStore

   1. AbstractRocksDBSegmentedBytesStore

   2. RocksDBSegmentedBytesStore

10. TimeOrderedKeyValueBuffer

    1. InMemoryTimeOrderedKeyValueBuffer

11. CachedStateStore

Logical Plan of Stream Processing Execution

1. ProcessorNode

   1. SourceNode

   2. SinkNode

2. NodeMetrics

3. InternalTopicConfig

   1. WindowedChangelogTopicConfig

   2. UnwindowedChangelogTopicConfig

4. WindowedStreamPartitioner

5. WindowedSerializer

6. DefaultKafkaClientSupplier

7. SessionWindow

8. TimeWindow

9. UnlimitedWindow

10. AbstractStream

    1. GlobalKTableImpl

    2. KGroupedStreamImpl

    3. KGroupedTableImpl

    4. KStreamAggregate

    5. KStreamImpl

    6. KTableImpl

    7. SessionWindowedKStreamImpl — Default SessionWindowedKStream

    8. TimeWindowedKStreamImpl

11. MaterializedInternal

12. KeyValueStoreMaterializer

13. InternalNameProvider Contract

14. GroupedStreamAggregateBuilder

15. KStreamImplJoin

16. StaticTopicNameExtractor

17. ConsumedInternal

18. ProducedInternal

19. QuickUnion

20. TopicsInfo

Physical Plan of Stream Processing Execution

1. ProcessorTopology

2. Task

   1. AbstractTask

   2. StandbyTask

   3. StreamTask

3. ProcessorContextImpl

4. ProducerSupplier

5. AssignedTasks

   1. AssignedStandbyTasks

   2. AssignedStreamsTasks

6. ProcessorNodePunctuator

Kafka Streams Execution Engine

1. TaskManager

   1. TaskCreator

   2. StandbyTaskCreator

   3. AbstractTaskCreator

2. StreamThread

3. RebalanceListener

4. StateListener

5. StreamsMetadataState

6. StreamsPartitionAssignor

   1. InternalTopicManager

   2. AssignmentInfo

   3. SubscriptionInfo

   4. ClientMetadata

   5. TaskAssignor Contract

      1. StickyTaskAssignor

7. InternalProcessorContext Contract

   1. AbstractProcessorContext

   2. GlobalProcessorContextImpl

   3. StandbyContextImpl

8. ForwardingDisabledProcessorContext

9. GlobalStreamThread

   1. StateConsumer

10. GlobalStateMaintainer

    1. GlobalStateUpdateTask

11. RecordCollector

    1. RecordCollectorImpl

12. ThreadCache

    1. NamedCache

13. Stamped

14. PartitionGroup

    1. RecordInfo

15. RecordQueue

    1. StampedRecord

16. PunctuationQueue

    1. PunctuationSchedule

17. QueryableStoreProvider

18. StateStoreProvider

    1. StreamThreadStateStoreProvider

    2. GlobalStateStoreProvider

    3. WrappingStoreProvider

19. RecordDeserializer

20. StateDirectory

21. ProcessorRecordContext

22. CopartitionedTopicsValidator

State (Store) Management

1. StateManager

   1. AbstractStateManager

2. ProcessorStateManager

3. GlobalStateManager

   1. GlobalStateManagerImpl

4. Checkpointable

5. OffsetCheckpoint

6. ChangelogReader

   1. StoreChangelogReader

7. StateRestorer

8. RestoringTasks

9. RecordBatchingStateRestoreCallback

   1. CompositeRestoreListener

10. StateRestoreCallbackAdapter