Casey Kennington , Computer Science, Boise State, 09-Nov 2023
ABSTRACT: Large Language Models (LLMs) are p rimarily trained using large amounts of text, but there have also been noteworthy advancements in incorporating vision and other sensory information into LLMs. Does that mean LLMs are ready for embodied agents such as robots? While there have been important advancements, technical and theoretical challenges remain including use of closed language models like ChatGPT, model size requirements, data size requirements, speed requirements, representing the physical world, and updating the model with information about the world in real time. In this talk, I explain recent advance on incorporating LLMs into robot platforms, challenges, and opportunities for future work.
Casey Kennington is associate professor in the Department of Computer Science at Boise State University where he does research on spoken dialogue systems on embodied platforms. His long-term research goal is to understand what it means for humans to understand, represent, and produce language. His National Science Foundation CAREER award focuses on enriching small language models with multimodal information such as vision and emotion for interactive learning on robotic platforms. Kennington obtained his PhD in Linguistics from Bielefeld University, Germany.
Josue Torres-Foncesca, Catherine Henry, Casey Kennington. Symbol and Communicative Grounding through Object Permanence with a Mobile Robot. In Proceedings of SigDial, 2022.
Clayton Fields and Casey Kennington. Vision Language Transformers: A Survey. arXiv, 2023.
Casey Kennington. Enriching Language Models with Visually-grounded Word Vectors and the Lancaster Sensorimotor Norms. In Proceedings of CoNLL, 2021
Casey Kennington. On the Computational Modeling of Meaning: Embodied Cognition Intertwined with Emotion. arXiv, 2023.
Understanding LLM Understanding 