Designing efficient and scalable cache management systems

Zoom link for remote attendees: password 123123

Abstract: Software-managed caches have been ubiquitously deployed in today’s system infrastructure. From personal devices to servers on the edge and the cloud, these caches speed up data access, reduce data movement, and avoid repeated computation. However, they consume a huge amount of resources, i.e., DRAM and CPUs.

In this talk, I will discuss how to design efficient and scalable cache systems. In the first part, I will demonstrate that the efficiency of a key-value cache is not only determined by the eviction algorithm but also by other components, e.g., storage layout and expiration design. I will then describe how I designed Segcache to reduce memory footprint by up to 60% and increase throughput by 8x compared to state-of-the-art systems. Segcache has been adopted for production at Twitter and Momento.

In the second part, I will introduce a surprising new finding from our largest-scale eviction algorithm study: FIFO queues are all we need for cache eviction. I will then describe S3-FIFO, a new cache eviction algorithm that is simpler, more scalable, and more efficient than state-of-the-art algorithms. S3-FIFO has been adopted for production at Google, VMWare, Redpanda, and several others.

Finally, I will describe my future work on building efficient, performant, and robust data systems.

Bio: Juncheng Yang (https://junchengyang.com) is a 6th-year Ph.D. student in the Computer Science Department at Carnegie Mellon University. His research interests broadly cover the efficiency, performance, reliability, and sustainability of large-scale data systems.

Juncheng’s works have received best paper awards at NSDI’21, SOSP’21, and SYSTOR’16. His OSDI’20 paper was recognized as one of the best storage papers at the conference and invited to ACM TOS’21. Juncheng received a Facebook Ph.D. Fellowship in 2020, was recognized as a Rising Star in machine learning and systems in 2023, and a Google Cloud Research Innovator in 2023.

His work, Segcache, has been adopted for production at Twitter and Momento. The two eviction algorithms he designed (S3-FIFO, SIEVE) have been adopted for production at Google, VMware, Redpanda, and several others with over 20 open-source libraries available on GitHub.

Moreover, the open-source cache simulation library libCacheSim he created has been used by almost 100 research institutes and companies.