Sasha: Introducing LLMs for smart spaces

Building on my weekend project to control some smart lights with ChatGPT, this wide-ranging paper fully introduces LLM-based reasoning to multi-device smart home environments. We introduce methods and benchmarks for measuring model performance at reasoning in smart homes, propose methods for engineering immediate and scheduled responses to user goals, propose a multi-step reasoning system for improving system performance, and conduct the first user study of a real LLM-controlled smart home.

Something that often gets lost in a research paper is the truly fun and challenging experiences you can have with the work. Between touring a trailer on the UT Austin JJ Pickle Campus to see if I could turn it into a smart home (I could not – it was outfitted with extremely sensitive equipment for conducting ventilation studies) to eventually hauling pegboard and furniture from my illegally-parked RAV4 up to an unused lab on the 7th floor of the EER building, this project was truly the highlight of my PhD.

What we accomplished was building and studying the first LLM-controlled smart home. As of writing, the work has been cited in Amazon patents for future Alexa iterations and has received an ACM Distinguished Paper Award at the following year’s ACM IMWUT conference.

This work would of course not be possible without contributions from my fellow graduate students, Haoxiang Yu, Sangsu Lee, and my advisor, Christine Julien.

Read the full text at the ACM Digital Library.