Autotelic Agents that Use and Ground Large Language Models (Oudeyer)

Pierre-Yves Oudeyer, Inria, Bordeaux, 21 mar 2024

ABSTRACT: Developmental AI aims to design and study artificial agents that are capable of open-ended learning. I will discuss two fundamental ingredients: (1) curiosity-driven exploration mechanisms, especially mechanisms enabling agents to invent and sample their own goals (such agents are called ‘autotelic’; (2) language and culture enabling enabling agents to learn from others’ discoveries, through the internalization of cognitive tools. I will discuss the main challenges in designing autotelic agents (e.g., how can they be creative in choosing their own goals?) and how some of them require language and culture to be addressed. I will also discuss using LLMs as proxies for human culture in autotelic agents, and how autotelic agents can leverage LLMs to learn faster, but also to align and ground them on the dynamics of the environment they interact with. I will also address some of the current main limitations of LLMs.

Pierre-Yves Oudeyer and his team at INRIA Bordeaux study open lifelong learning and the self-organization of behavioral, cognitive and language structures, at the frontiers of AI and cognitive science. In the field of developmental AI, we use machines as tools to better understand how children learn, and to study how machines could learn autonomously as children do and could integrate into human cultures. We study models of curiosity-driven autotelic learning, enabling humans and machines to set their own goals and self-organize their learning program. We also work on applications in education and assisted scientific discovery, using AI techniques to serve humans, and encourage learning, curiosity, exploration and creativity.

Colas, C; T Karch, C Moulin-Frier, PY Oudeyer (2022) Language and Culture Internalisation for Human-Like Autotelic AI Nature Machine Intelligence 4 (12), 1068-1076 https://arxiv.org/abs/2206.01134

Carta, T., Romac, C., Wolf, T., Lamprier, S., Sigaud, O., & Oudeyer, P. Y. (2023). Grounding large language models in interactive environments with online reinforcement learning. ICML https://arxiv.org/abs/2302.02662

Colas, C., Teodorescu, L., Oudeyer, P. Y., Yuan, X., & Côté, M. A. (2023). Augmenting Autotelic Agents with Large Language Models. arXiv preprint arXiv:2305.12487. https://arxiv.org/abs/2305.12487

Language Writ Large: LLMs, ChatGPT, Meaning and Understanding (Harnad)

Stevan Harnad. UQÀM, McGill , 25 January 2004

VIDEO

ABSTRACT: Apart from what (little) OpenAI may be concealing from us, we all know (roughly) how ChatGPT works (its huge text database, its statistics, its vector representations, and their huge number of parameters, its next-word training, etc.). But none of us can say (hand on heart) that we are not surprised by what ChatGPT has proved to be able to do with these resources. It has even driven some of us to conclude that it actually understands. It’s not true that it understands. But it is also not true that we understand how it can do what it can do. I will suggest some hunches about benign “biases” — convergent constraints that emerge at LLM-scale that may be helping ChatGPT do so much better than we would have expected. These biases are inherent in the nature of language itself, at LLM-scale, and they are closely linked to what it is that ChatGPT lacks, which is direct sensorimotor grounding to integrate its words with their referents in the real world and to integrate its propositions with their meanings. These benign biases are related to (1) the parasitism of indirect verbal grounding on direct sensorimotor grounding, (2) the circularity of verbal definition, (3) the “mirroring” of language production and comprehension, (4) iconicity in propositions at LLM-scale, (5) computational counterparts of human “categorical perception” in category learning by neural nets, and perhaps also (6) a conjecture by Chomsky about the laws of thought.

Stevan Harnad is Professor of psychology and cognitive science at UQÀM. His research is on category-learning, symbol-grounding, language-evolution, and Turing-Testing

Bonnasse-Gahot, L., & Nadal, J. P. (2022). Categorical perception: a groundwork for deep learning. Neural Computation, 34(2), 437-475.

Harnad, S. (2024). Language Writ Large: LLMs, ChatGPT, Grounding, Meaning and Understanding. arXiv preprint arXiv:2402.02243.

Harnad, S. (2012). From sensorimotor categories and pantomime to grounded symbols and propositions In: Gibson, KR & Tallerman, M (eds.) The Oxford Handbook of Language Evolution 387-392.

Harnad, S. (2008) The Annotation Game: On Turing (1950) on Computing, Machinery, and Intelligence. In: Epstein, R, Roberts, Gary & Beber, G. (eds.) Parsing the Turing Test: Philosophical and Methodological Issues in the Quest for the Thinking Computer. Springer, pp. 23-66.

Thériault, C., Pérez-Gay, F., Rivas, D., & Harnad, S. (2018). Learning-induced categorical perception in a neural network model. arXiv preprint arXiv:1805.04567.

Vincent‐Lamarre, P; Blondin-Massé, A; Lopes, M; Lord, M; Marcotte, O; & Harnad, S (2016). The latent structure of dictionaries. Topics in Cognitive Science 8(3): 625-659.

Pérez-Gay Juárez, F., Sicotte, T., Thériault, C., & Harnad, S. (2019). Category learning can alter perception and its neural correlates. PloS one, 14(12), e0226000.

Mechanistic Explanation in Deep Learning (Millière)

Raphaël Millière, Philosophy, Macquarie University, 14 September, 2024

VIDEO

Abstract: Deep neural networks such as large language models (LLMs) have achieved impressive performance across almost every domain of natural language processing, but there remains substantial debate about which cognitive capabilities can be ascribed to these models. Drawing inspiration from mechanistic explanations in life sciences, the nascent field of “mechanistic interpretability” seeks to reverse-engineer human-interpretable features to explain how LLMs process information. This raises some questions: (1) Are causal claims about neural network components, based on coarse intervention methods (such as “activation patching”), genuine mechanistic explanations? (2) Does the focus on human-interpretable features risk imposing anthropomorphic assumptions? My answer will be “yes” to (1) and “no” to (2), closing with a discussion of some ongoing challenges.

Raphael Millière is Lecturer in Philosophy of Artificial Intelligence at Macquarie University in Sydney, Australia. His interests are in the philosophy of artificial intelligence, cognitive science, and mind, particularly in understanding artificial neural networks based on deep learning architectures such as Large Language Models. He has investigated syntactic knowledge, semantic competence, compositionality, variable binding, and grounding.

Elhage, N., et al. (2021). A mathematical framework for transformer circuits. Transformer Circuits Thread

Machamer, P., Darden, L., & Craver, C. F. (2000). Thinking about Mechanisms. Philosophy of Science, 67(1), 1–25.

Millière, R. (2023). The Alignment Problem in Context. arXiv preprint arXiv:2311.02147

Mollo, D. C., & Millière, R. (2023). The vector grounding problem. arXiv preprint arXiv:2304.01481

Yousefi, S., et al. (2023). In-Context Learning in Large Language Models: A Neuroscience-inspired Analysis of Representations. arXiv preprint arXiv:2310.00313.

LLMs, Patterns, and Understanding (Durt)

Christof Durt , Philosophy, U. Heidelberg, 30-Nov 2023

VIDEO

ABSTRACT: It is widely known that the performance of LLMs is contingent on their being trained with very large text corpora. But what in the text corpora allows LLMs to extract the parameters that enable them to produce text that sounds as if it had been written by an understanding being? In my presentation, I argue that the text corpora reflect not just “language” but language use. Language use is permeated with patterns, and the statistical contours of the patterns of written language use are modelled by LLMs. LLMs do not model understanding directly, but statistical patterns that correlate with patterns of language use. Although the recombination of statistical patterns does not require understanding, it enables the production of novel text that continues a prompt and conforms to patterns of language use, and thus can make sense to humans.

Christoph Durt is a philosophical and interdisciplinary researcher at Heidelberg university. He investigates the human mind and its relation to technology, especially AI. Going beyond the usual side-to-side comparison of artificial and human intelligence, he studies the multidimensional interplay between the two. This involves the study of human experience and language, as well as the relation between them. If you would like to join an international online exchange on these issues, please check the “courses and lectures” section on his website.

Durt, Christoph, Tom Froese, and Thomas Fuchs. preprint. “Against AI Understanding and Sentience: Large Language Models, Meaning, and the Patterns of Human Language Use.”

Durt, Christoph. 2023. “The Digital Transformation of Human Orientation: An Inquiry into the Dawn of a New Era” Winner of the $10.000 HFPO Essay Prize.

Durt, Christoph. 2022. “Artificial Intelligence and Its Integration into the Human Lifeworld.” In The Cambridge Handbook of Responsible Artificial Intelligence, Cambridge University Press.

Durt, Christoph. 2020. “The Computation of Bodily, Embodied, and Virtual Reality” Winner of the Essay Prize “What Can Corporality as a Constitutive Condition of Experience (Still) Mean in the Digital Age?”Phänomenologische Forschungen, no. 2: 25–39.

LLMs: Indication or Representation? (Søgaard)

Anders Søgaard , Computer Science & Philosophy, U. Copenhagen, 23-Nov 2023

VIDEO

ABSTRACT: People talk to LLMs – their new assistants, tutors, or partners – about the world they live in, but are LLMs parroting, or do they (also) have internal representations of the world? There are five popular views, it seems:

LLMs are all syntax, no semantics.
LLMs have inferential semantics, no referential semantics.
LLMs (also) have referential semantics through picturing
LLMs (also) have referential semantics through causal chains.
Only chatbots have referential semantics (through causal chains)

I present three sets of experiments to suggest LLMs induce inferential and referential semantics and do so by inducing human-like representations, lending some support to view (iii). I briefly compare the representations that seem to fall out of these experiments to the representations to which others have appealed in the past.

Anders Søgaard is University Professor of Computer Science and Philosophy and leads the newly established Center for Philosophy of Artificial Intelligence at the University of Copenhagen. Known primarily for work on multilingual NLP, multi-task learning, and using cognitive and behavioral data to bias NLP models, Søgaard is an ERC Starting Grant and Google Focused Research Award recipient and the author of Semi-Supervised Learning and Domain Adaptation for NLP (2013), Cross-Lingual Word Embeddings (2019), and Explainable Natural Language Processing (2021).

Søgaard, A. (2023). Grounding the Vector Space of an Octopus. Minds and Machines 33, 33-54.

Li, J.; et al. (2023) Large Language Models Converge on Brain-Like Representations. arXiv preprint arXiv:2306.01930

Abdou, M.; et al. (2021) Can Language Models Encode Perceptual Structure Without Grounding? CoNLL

Garneau, N.; et al. (2021) Analogy Training Multilingual Encoders. AAAI

Robotic Grounding and LLMs: Advancements and Challenges (Kennington)

Casey Kennington , Computer Science, Boise State, 09-Nov 2023

VIDEO

ABSTRACT: Large Language Models (LLMs) are p rimarily trained using large amounts of text, but there have also been noteworthy advancements in incorporating vision and other sensory information into LLMs. Does that mean LLMs are ready for embodied agents such as robots? While there have been important advancements, technical and theoretical challenges remain including use of closed language models like ChatGPT, model size requirements, data size requirements, speed requirements, representing the physical world, and updating the model with information about the world in real time. In this talk, I explain recent advance on incorporating LLMs into robot platforms, challenges, and opportunities for future work.

Casey Kennington is associate professor in the Department of Computer Science at Boise State University where he does research on spoken dialogue systems on embodied platforms. His long-term research goal is to understand what it means for humans to understand, represent, and produce language. His National Science Foundation CAREER award focuses on enriching small language models with multimodal information such as vision and emotion for interactive learning on robotic platforms. Kennington obtained his PhD in Linguistics from Bielefeld University, Germany.

Josue Torres-Foncesca, Catherine Henry, Casey Kennington. Symbol and Communicative Grounding through Object Permanence with a Mobile Robot. In Proceedings of SigDial, 2022.

Clayton Fields and Casey Kennington. Vision Language Transformers: A Survey. arXiv, 2023.

Casey Kennington. Enriching Language Models with Visually-grounded Word Vectors and the Lancaster Sensorimotor Norms. In Proceedings of CoNLL, 2021

Casey Kennington. On the Computational Modeling of Meaning: Embodied Cognition Intertwined with Emotion. arXiv, 2023.

Machine Psychology (Schulz)

Eric Schulz , MPI Tuebingen, 02-Nov 2023

VIDEO

ABSTRACT: Large language models are on the cusp of transforming society while they permeate into many applications. Understanding how they work is, therefore, of great value. We propose to use insights and tools from psychology to study and better understand these models. Psychology can add to our understanding of LLMs and provide a new toolkit for explaining LLMs by providing theoretical concepts, experimental designs, and computational analysis approaches. This can lead to a machine psychology for foundation models that focuses on computational insights and precise experimental comparisons instead of performance measures alone. I will showcase the utility of this approach by showing how current LLMs behave across a variety of cognitive tasks, as well as how one can make them more human-like by fine-tuning on psychological data directly.

Eric Schulz, Max-Planck Research Group Leader, Tuebingen University works on the building blocks of intelligence using a mixture of computational, cognitive, and neuroscientific methods. He has worked with Maarten Speekenbrink on generalization as function learning and Sam Gershman and Josh Tenenbaum.

Binz, M., & Schulz, E. (2023). Using cognitive psychology to understand GPT-3. Proceedings of the National Academy of Sciences, 120(6), e2218523120

Akata, E., Schulz, L., Coda-Forno, J., Oh, S. J., Bethge, M., & Schulz, E. (2023). Playing repeated games with Large Language Models. arXiv preprint arXiv:2305.16867.

Allen, K. R., Brändle, F., Botvinick, M., Fan, J., Gershman, S. J., Griffiths, T. L., … & Schulz, E. (2023). Using Games to Understand the Mind

Binz, M., & Schulz, E. (2023). Turning large language models into cognitive models. arXiv preprint.

The Debate Over “Understanding” in AI’s Large Language Models (Mitchell)

Melanie Mitchell , Santa Fe Institute, 19-Oct

VIDEO

ABSTRACT: I will survey a current, heated debate in the AI research community on whether large pre-trained language models can be said — in any important sense — to “understand” language and the physical and social situations language encodes. I will describe arguments that have been made for and against such understanding, and, more generally, will discuss what methods can be used to fairly evaluate understanding and intelligence in AI systems. I will conclude with key questions for the broader sciences of intelligence that have arisen in light of these discussions.

Melanie Mitchell is Professor at the Santa Fe Institute. Her current research focuses on conceptual abstraction and analogy-making in artificial intelligence systems. Melanie is the author or editor of six books and numerous scholarly papers in the fields of artificial intelligence, cognitive science, and complex systems. Her 2009 book Complexity: A Guided Tour (Oxford University Press) won the 2010 Phi Beta Kappa Science Book Award, and her 2019 book Artificial Intelligence: A Guide for Thinking Humans (Farrar, Straus, and Giroux) is a finalist for the 2023 Cosmos Prize for Scientific Writing.

Mitchell, M. (2023). How do we know how smart AI systems are? Science, 381(6654), adj5957.

Mitchell, M., & Krakauer, D. C. (2023). The debate over understanding in AI’s large language models. Proceedings of the National Academy of Sciences, 120(13), e2215907120.

Millhouse, T., Moses, M., & Mitchell, M. (2022). Embodied, Situated, and Grounded Intelligence: Implications for AI. arXiv preprint arXiv:2210.13589.

Symbols and Grounding in LLMs (Pavlick)

Ellie Pavlick , Computer Science, Brown, 05-Oct 2023

VIDEO

ABSTRACT: Large language models (LLMs) appear to exhibit human-level abilities on a range of tasks, yet they are notoriously considered to be “black boxes”, and little is known about the internal representations and mechanisms that underlie their behavior. This talk will discuss recent work which seeks to illuminate the processing that takes place under the hood. I will focus in particular on questions related to LLM’s ability to represent abstract, compositional, and content-independent operations of the type assumed to be necessary for advanced cognitive functioning in humans.

Ellie Pavlick is an Assistant Professor of Computer Science at Brown University. She received her PhD from University of Pennsylvania in 2017, where her focus was on paraphrasing and lexical semantics. Ellie’s research is on cognitively-inspired approaches to language acquisition, focusing on grounded language learning and on the emergence of structure (or lack thereof) in neural language models. Ellie leads the language understanding and representation (LUNAR) lab, which collaborates with Brown’s Robotics and Visual Computing labs and with the Department of Cognitive, Linguistic, and Psychological Sciences.

Tenney, Ian, Dipanjan Das, and Ellie Pavlick. “BERT Rediscovers the Classical NLP Pipeline.” Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019. https://arxiv.org/pdf/1905.05950.pdf

Pavlick, Ellie. “Symbols and grounding in large language models.” Philosophical Transactions of the Royal Society A 381.2251 (2023): 20220041. https://royalsocietypublishing.org/doi/pdf/10.1098/rsta.2022.0041

Lepori, Michael A., Thomas Serre, and Ellie Pavlick. “Break it down: evidence for structural compositionality in neural networks.” arXiv preprint arXiv:2301.10884 (2023). https://arxiv.org/pdf/2301.10884.pdf

Merullo, Jack, Carsten Eickhoff, and Ellie Pavlick. “Language Models Implement Simple Word2Vec-style Vector Arithmetic.” arXiv preprint arXiv:2305.16130 (2023). https://arxiv.org/pdf/2305.16130.pdf

Grounded Language Learning in Virtual Environments (Clark)

Stephen Clark, U Cambridge and Quantinuum, 19 November, 2020

VIDEO

Abstract: Natural Language Processing is currently dominated by the application of text-based language models such as GPT. One feature of these models is that they rely entirely on the statistics of text, without making any connection to the world, which raises the interesting question of whether such models could ever properly “understand” the language. One these models can be grounded is to connect them to images or videos, for example by conditioning the language models on visual input and using them for captioning. In this talk I extend the grounding idea to a simulated virtual world: an environment which an agent can perceive, explore and interact with. A neural-network-based agent is trained — using distributed deep reinforcement learning — to associate words and phrases with things that it learns to see and do in the virtual world.The world is 3D, built in Unity, and contains recognisable objects, including some from the ShapeNet repository of assets. One of the difficulties in training such networks is that they have a tendency to overfit to their training data.We demonstrate how the interactive, first-person perspective of an agent helps it to generalize to out-of-distribution settings. Training the agents typically requires a huge number of training examples. We show how meta-learning can be used to teach the agents to bind words to objects in a one-shot setting. The agent is able to combine its knowledge of words obtained one-shot with its stable knowledge of word meanings learned over many episodes, providing a form of grounded language learning which is both “fast and slow”. Joint work with Felix Hill.

Clark, S., Lerchner, A., von Glehn, T., Tieleman, O., Tanburn, R., Dashevskiy, M., & Bosnjak, M. (2021). Formalising Concepts as Grounded Abstractions. arXiv preprint arXiv:2101.05125.

Tull, S., Shaikh, R. A., Zemljic, S. S., & Clark, S. (2023). From Conceptual Spaces to Quantum Concepts: Formalising and Learning Structured Conceptual Models. arXiv preprint arXiv:2401.08585.