An AI stack: from cloud orchestration to LLM evaluation

A reception will follow the talk.

Abstract: With the release of ChatGPT, just over one year ago, large language models (LLMs) have taken the world by storm: they have enabled new applications, have exacerbated GPU shortage, and raised new questions about their answers’ veracity. In this talk, I will present several projects I have been working on over the past three years, which are now part of an open-source stack for training, fine tuning, serving and evaluating LLMs. In this talk, I will focus on three projects: (i) SkyPilot, a broker architecture that makes it easy to run AI workloads on a variety of clouds to improve availability, cost and performance, (ii) vLLM, a high-throughput inference engine for LLMs, and (iii) Chatbot Arena, a system to accurately benchmark LLMs.

Bio: Ion Stoica is a Professor in the EECS Department at the University of California at Berkeley, and the Director of Sky Computing Lab (https://sky.cs.berkeley.edu/). He is currently doing research on cloud computing and AI systems. Past work includes Ray, Apache Spark, Apache Mesos, Tachyon, Chord DHT, and Dynamic Packet State (DPS). He is a Member of NAE, an Honorary Member of the Romanian Academy, an ACM Fellow and has received numerous awards, including the Mark Weiser Award (2019), SIGOPS Hall of Fame Award (2015), and several “Test of Time” awards. He also co-founded three companies, Anyscale (2019), Databricks (2013) and Conviva (2006).

Bio: Ion Stoica is a Professor in the EECS Department at the University of California at Berkeley, and the Director of SkyLab (https://sky.cs.berkeley.edu/). He is currently doing research on cloud computing and AI systems. Past work includes Ray, Apache Spark, Apache Mesos, Tachyon, Chord DHT, and Dynamic Packet State (DPS). He is an Honorary Member of the Romanian Academy, an ACM Fellow and has received numerous awards, including the Mark Weiser Award (2019), SIGOPS Hall of Fame Award (2015), and several “Test of Time” awards, and was elected to the National Academy of Engineering in 2024. He also co-founded three companies, Anyscale (2019), Databricks (2013), and Conviva (2006).