(2023-05-05) The Socratic Method For Selfdiscovery In Large Language Models

Runzhe Yang and Karthik Narasimhan: The Socratic Method for Self-Discovery in Large Language Models. In this blog post, we explore three key aspects that hold immense potential in unleashing the capabilities of Language Model models (LLMs):

Multi-Agent Collaborative Problem Solving (with Human in the Loop)

The Power of the Socratic Method
We examine the Socratic method and its ability to robustly elicit analytical and critical thinking capabilities in LLMs. While approaches like CoT/ReAct have made strides in this area, they often rely on a fixed form of meticulously crafted prompting

Rethinking ‘Prompting’ for Knowledge and Reasoning
We generalize the concept of “prompt engineering” to a more comprehensive approach.

Based on these insights, we propose SocraticAI, a new method for facilitating self-discovery and creative problem solving using LLMs.

Platonic epistemology: all learning is recollection.

Socrates unveils a bold assertion, “All learning is recollection.” This provocative proposition, also known as the theory of anamnesis, posits that our souls are imbued with knowledge from past lives and that learning is simply the process of unearthing this dormant wisdom.

Large Language Models: all you need is prompting.

vaguely reminiscent of Meno’s servant boy, who possesses vast amounts of “innate” knowledge and problem-solving abilities, but only needs the right prompts to reveal them.

The pre-training process of LLMs is akin to the “past lives” in Socrates’ theory.

To fully unlock the potential of LLMs, “prompt engineering” becomes essential. Various techniques, including “Chain of Thought” (CoT) (Huang et al., 2022; Wei et al., 2022), “Describe, Explain, Plan, and Select” (DEPS) (Wang et al., 2023), “Reason+Act” (ReAct) (Yao et al., 2023) and self-reflection (Reflexon) (Shinn et al., 2023) have been developed to augment LLMs’ problem-solving capabilities through multi-step planning and articulated reasoning processe

As remarkable as these prompt engineering techniques are, they 1) require the creation of task-specific prompt templates, and 2) are limited to a single train of thought within the LLM which precludes the use of multiple attack angles for creative problem solving (as we demonstrate in examples below).

one cannot help but wonder if the Socratic dialogue method could be adapted to involve multiple LLMs in conversation, with the AI system also playing the role of Socrates and guiding the other agents to ask the “right questions”

In a recent study, researchers placed several LLM-based AI agents in a virtual environment similar to the game The Sims to simulate human-like interactions and behaviors

To validate this idea, we implemented a framework called “Socratic AI

This framework employs three independent LLM-based agents, who role-play Socrates (an analyst), Theaetetus (an analyst), and Plato (a proofreader) to solve problems in a collaborative fashion. The agents are provided access to WolframAlpha and a Python code interpreter.

At the beginning of each dialogue, all agents are provided with the following meta-level system prompt:
Socrates and Theaetetus are two AI assistants for Tony to solve challenging problems. The problem statement is as follows: "{question}."

They are encouraged to write and execute Python scripts

To aid them in their calculations and fact-checking, they are also allowed to consult WolframAlpha

Case study: the twenty-four game

More demos:

Estimate the connection desity in a fly brain

Calculate the sum of a prime indexed row

Future & Limitations: envisioning a collaborative AI society

Although we only experimented with homogeneous LLM-based agents (identical base models and expertise) so far, one can envision a future where a tapestry of AI agents, each boasting unique specializations, collaborate to address multifaceted challenges and spark a productivity renaissance.

This vision paves the way for an “AI Society,” where AI operates in unison and with self-censorship to engage in productive activities

However, we must also recognize the limitations of the Socratic approach.