What’s Next?
Let’s journey through a forward-thinking framework for AI agents inspired by human cognition. With a rich trove of research already on the table, much is to be said. Taking a page from Daniel Kahneman’s book “Thinking, Fast, and Slow,” we can imagine AI functions as a two-part system. System 1, mirroring our automatic cognitive processes, handles tasks on ‘autopilot’ – think of riding a bike or driving. This is similar to prompt engineering routines and tool usage in agents. System 2, conversely, reflects our deliberative and analytical thoughts, involving critical thinking, planning, and informed action. Thanks to recent advances, AI agents are beginning to show hints of System 2-like mechanisms, tackling complex tasks, engaging in reasoning, and even developing System 1 routines for support.
Yann LaCun’s influential 2022 paper suggests a generalized architecture for autonomous agents. Many of the suggested concepts resonate with existing AI agent designs, albeit with some gaps. Our conversation will revolve around these gaps and the possible pathways for bridging them.
World Model: Envisioning future possibilities based on imagined action sequences is a remarkable human ability, enabling us to learn from few or no demonstrations, particularly when actions carry high stakes. Take driving as an example—our understanding of physics, vehicle control, and traffic rules helps us anticipate the results of sudden acceleration or abrupt braking. AI agents of today embed their world model within the LLM. While this works fine for general tasks, it could fall short on specialized ones. Consider an agent using GPT4 to play Minecraft [1]. It learns much about the game from the vast amount of relevant data used to train the model, like wikis and Reddit discussions. But what happens when it’s thrown into a completely new game environment? The extent to which LLMs reason within the language space is still a hot research topic. A pathway for more robust world models involving more efficient self-supervised learning methods in multimodal domains could greatly enhance an agent’s causal understanding and reasoning skills within and beyond the language domain. Self-supervision training methods, prioritizing an overarching understanding of data rather than a specific focus on every little detail, are starting to reveal their potential [2].
Actor: The “Actor” in AI agents is the planner, coming up with the best game plan for the task at hand. A reliable world model would help here, initially judging the actor’s proposed plans. It can rule out steps unlikely to work, forecasting their outcomes. Today’s AI systems already use an LLM to reflect on plans and actions, signaling progress in this area. More advanced methods, like [3], treat the possible actions as a tree structure navigating through the branches. AI systems also train policy networks to handle simpler, immediate-response actions, similar to how humans form and use tools [4, 5]. Here an interesting concept that emerged recently is the use of humans as tools, mainly for feedback and guidance within the AI agent journey of actions. Looking to the future, we expect significant advancements in creating a trainable actor module. The work being done on process supervision seems especially promising [6].
Memory: AI agents currently manage memory by using simple key-value pairs of embeddings and associated text data. Techniques such as MIPS (Maximum Inner Product Search) are employed to retrieve relevant memories. While this works for simple tasks, it struggles with complex ones, such as writing an in-depth README for a code repository, necessitating a conceptual understanding of the code flows and more advanced memory processes. Hierarchical memory, organizing data at varying detail levels, and associative memory, linking related data, are promising solutions (GraphDBs are an example of this.) Existing agents have started storing their reflections in memory, aiding in abstract knowledge accumulation [7]. Nowadays, AI agents use a combination of memory and tools for ongoing learning and adaptation. Because most leading LLMs aren’t fine-tunable (fine-tuning is complex and costly) it seems that memory usage takes precedence over fine-tuning. However, the pure memory-based approach leads to performance bottlenecks in retrieval of relevant data into context. Our reliable ally, gradient descent, could be the key here again. Instead of relying on heuristics and static memory embeddings, training memory-specific models that will naturally display human-like characteristics are a promising direction. Future innovations may take a hybrid approach, directly integrating part of the learning process into the language model’s weights while supporting a dedicated memory module.
Efficient Training: Propelling all other advancements in the AI sphere, the field is rapidly progressing towards more effective AI training methodologies, involving strategies such as distilled datasets, the QLORA training method for rapid model fine-tuning, knowledge transfer, prompt embedding editing, collaboration of multiple expert models, and usage of consumer-grade CPUs/GPUs. Alongside these, there’s a trend towards scaling up training to create increasingly advanced models, whose impacts remain to be fully seen for future models. If scaling laws persist as they did between GPT3 & 4, we might be closer to AGI than we think.
