Kirin: Efficient In-Storage Learned Compaction for LSM-Trees via System-Algorithm Co-Design

Abstract: The log-structured merge-trees (LSM-trees) are widely used in modern Key-Value (KV) stores, offering strong write performance but facing significant inefficiencies in compaction and indexing. While recent researches have integrated learned indexes with LSM-trees to address these inefficiencies, their integration remains hindered by excessive cold data movement, limited parallelism in model training, and the decoupled nature of compaction and training. In this paper, we present Kirin, a hybrid KV store that synergistically integrates LSM-tree and learned index, and leverages computational storage devices (CSDs) to offload data-intensive tasks. Kirin introduces a novel learned compaction approach that embeds model training directly into the compaction process to conceal training latency and enable timely model updates. Kirin also employs a collaborative approach between the host and CSD to parallelize compaction and minimize storage access during indexing. Our experiments with DaisyPlus OpenSSD demonstrate that Kirin outperforms existing solutions in both read and write throughput by a large margin, while maintaining low read latency under heavy write workloads.