RocksDB
Internally, KafkaStreams relies on an embedded key-value store so-called RocksDB to provided persistent storage. Depending of the throughput of your application, you may want to tune internal RocksDB instances.
Kafka Streams allows you to customize the RocksDB settings for a given Store by implementing the interface org.apache.kafka.streams.state.RocksDBConfigSetter.
The custom implementation must then be configured using :
streamsConfig.put(StreamsConfig.ROCKSDB_CONFIG_SETTER_CLASS_CONFIG, CustomRocksDBConfig.class)
Azkarra Streams provides a built-in DefaultRocksDBConfigSetter which is automatically configured if no one is already provided.
DefaultRocksDBConfigSetter allows you to override not only some default RocksDB options but also to enable log statistics for performance debugging.
Available properties are :
| Property | Type | Description |
|---|---|---|
rocksdb.stats.enable |
boolean | Enable RocksDB statistics |
rocksdb.stats.dump.period.sec |
integer | The RocksDB statistics dump period in seconds. |
rocksdb.log.dir |
string | The RocksDB log directory |
rocksdb.log.level |
string | The RocksDB log level (see org.rocksdb.InfoLogLevel). |
rocksdb.log.max.file.size |
integer | The RocksDB maximum log file size. |
rocksdb.max.write.buffer.number |
integer | The maximum number of memtables build up in memory, before they flush to SST files. |
rocksdb.write.buffer.size |
long | The size of a single memtable. |
Note that all properties described above are optional.
RocksDB properties can be passed either using default configuration :
context {
streams {
rocksdb {
stats.enable = false
stats.dumpPeriodSec = 30
log {
dir = "/var/log/kafka-streams/rocksdb"
file.size = 104857600
}
}
}
}
or programmatically through the method StreamsExecutionEnvironment#setRocksDBConfig:
context.defaultExecutionEnvironment()
.setRocksDBSettings(
RocksDBConfig.withStatsEnable()
.withLogDir("/tmp/rocksdb-logs")
);
Please read official documentation for more information: RocksDB Tuning Guide