Reasoning Alias: CoT In-Context Learning

Chain-of-Thought Prompting

Eliciting multi-step reasoning in large language models by prompting them to generate a sequence of intermediate thoughts before producing a final answer.

In One Sentence

"Rather than mapping inputs directly to outputs, Chain-of-Thought prompting structures the model's generation to output a step-by-step reasoning path, dramatically improving performance on complex arithmetic, symbolic, and commonsense tasks."

Intuition

Standard sequence-to-sequence generation forces an autoregressive model to produce a final answer immediately after reading the prompt. For complex tasks (like multi-step math word problems or symbolic logic), this requires the model to perform massive implicit computation within a single forward pass over a few output tokens.

Chain-of-Thought (CoT) solves this by introducing a "scratchpad" directly into the output sequence. By generating intermediate reasoning steps, the model:

Decomposes the problem: Breaks down a complex multi-step problem into simpler, sequential sub-problems.
Allocates more computation: Spends more decoding steps (and thus more forward passes/FLOPs) processing the problem before committing to a final answer.
Maintains state: Uses the context window to store intermediate facts, reducing the memory load on internal representations.

Formalization

Let $x$ represent the input prompt (e.g., a math question) and $y$ represent the final answer.

In Standard Prompting, we directly model the conditional probability of the output $y$ given input $x$:

P(y \mid x)

In Chain-of-Thought Prompting, we introduce a sequence of intermediate reasoning tokens (or "thoughts") $z = (z_1, z_2, \dots, z_N)$ where each $z_i$ represents a reasoning step. The joint probability of generating the reasoning chain and the final answer is modeled as:

$$P(z, y \mid x) = \left( \prod_{i=1}^N P(z_i \mid x, z_{ During inference, the model samples from $P(z, y \mid x)$ using greedy decoding or temperature sampling. The final answer $y$ is extracted from the end of the generated sequence.

Interactive Walkthrough

Compare standard prompting against Chain-of-Thought prompting on a multi-step word problem.

Problem Statement

"A toy store has 30 blocks. They sell 2 groups of 5 blocks, then buy 8 more. How many blocks do they have?"

Q: A toy store has 30 blocks. They sell 2 groups of 5 blocks, then buy 8 more. How many blocks do they have?

18 Incorrect

Why it fails: The model attempts to map the entire question directly to a single scalar token, skipping the necessary arithmetic steps ($30 - (2 \times 5) + 8$).

When to Use

Multi-step reasoning: Arithmetic, symbolic manipulation, and complex logical puzzles.
Large Models: Typically emerges as a highly effective capability in sufficiently large models (e.g., 100B+ parameters or high-quality instruction-tuned models).
Explainability: When you need human-readable justification for the model's final output.

When NOT to Use

Latency-critical tasks: Generating thoughts dramatically increases token count, leading to slower response times and higher inference costs.
Simple retrieval: Single-hop queries (e.g., "What is the capital of France?") do not benefit from CoT.
Small models: Models with low capacity often hallucinate incorrect reasoning steps, leading to worse performance than standard prompting.

History & Evolution

JANUARY 2022

Eliciting Reasoning via CoT

Wei et al. (arXiv:2201.11903) formally introduce Chain-of-Thought Prompting. They demonstrate that reasoning abilities emerge naturally in sufficiently large language models. Prompting a 540B-parameter model with just 8 exemplars achieves state-of-the-art accuracy on the GSM8K math benchmark, outperforming fine-tuned GPT-3 with a verifier.

MAY 2023

Tree of Thoughts (ToT)

Yao et al. (arXiv:2305.10601) generalize CoT into the Tree of Thoughts (ToT) framework. ToT enables deliberate decision-making by exploring multiple reasoning paths (thoughts), evaluating choices, and backtracking when necessary to solve complex planning tasks like the Game of 24.

MAY 2023

Step-by-Step Verification

Lightman et al. (arXiv:2305.20050) introduce process supervision ("Let's Verify Step by Step"), showing that providing feedback for each intermediate reasoning step (rather than just the final outcome) significantly improves model reliability on the MATH dataset.

Implementation

A clean Python example demonstrating both few-shot and zero-shot Chain-of-Thought prompting strategies.

# Chain-of-Thought Prompting Strategies

def get_zero_shot_cot_prompt(question: str) -> str:
    """
    Appends the classic trigger phrase to elicit zero-shot reasoning.
    """
    return f"Q: {question}\nA: Let's think step by step."


def get_few_shot_cot_prompt(question: str) -> str:
    """
    Constructs a prompt with a few-shot exemplar demonstrating intermediate reasoning.
    """
    exemplars = (
        "Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. "
        "Each can has 3 tennis balls. How many tennis balls does he have now?\n"
        "A: Roger started with 5 balls. 2 cans of 3 balls each is 6 balls. "
        "5 + 6 = 11. The answer is 11.\n\n"
    )
    return f"{exemplars}Q: {question}\nA:"


# Example Usage:
question = "If a bakery bakes 40 loaves of bread and sells 35, then bakes 20 more, how many are left?"
prompt = get_few_shot_cot_prompt(question)
# Output prompt is fed directly to the LLM's generation endpoint.

Common Variants

Zero-Shot CoT

Simply appending "Let's think step by step" to the prompt triggers reasoning paths without needing explicit exemplars.

Tree of Thoughts (ToT)

Generates a tree structure of thoughts, allowing the model to search, evaluate, and backtrack across multiple reasoning paths.

Process Supervision

Training reward models (PRMs) to evaluate and score each individual reasoning step, rather than just the final answer.

Representative Flash Papers

Language Models are Few-Shot Learners

ryhchkpe82

Introduces the foundation of in-context learning and few-shot prompting upon which Chain-of-Thought prompting is built.

Chain-of-Thought Prompting

In One Sentence

Intuition

Formalization

Interactive Walkthrough

When to Use

When NOT to Use

History & Evolution

Eliciting Reasoning via CoT

Tree of Thoughts (ToT)

Step-by-Step Verification

Implementation

Common Variants

Zero-Shot CoT

Tree of Thoughts (ToT)

Process Supervision

Representative Flash Papers

Language Models are Few-Shot Learners

Further Reading