Spark Structured Streaming

Enhancements

• SPAR-3747: Memory leak issues in RocksDB based state store are fixed, and the rocksdb configuration is better tuned to improve read performance. The previous rocksDB state-store provider class is deprecated. Users should use the following Spark configuration to enable RocksDB based state store:

  spark.sql.streaming.stateStore.providerClass = org.apache.spark.sql.execution.streaming.state.RocksDbStateStoreProvider
  

  This is supported on Spark 2.4.0 and later versions.