Universal Verifier

The Core Concept of a Universal Verifier

%% Refined Diagram: Criteria as an Explicit Input to the Verifier
graph TD
    subgraph "The LLM System"
        LLM[("Large Language Model")]
    end

    subgraph "Knowledge Base"
        Criteria["Evaluation Criteria 
 (e.g., Rubrics, Principles)"]
    end

    subgraph "Generation & Evaluation"
        Output["LLM Output 
 (Code, Text, etc.)"]
        Verifier(Universal Verifier)
    end

    subgraph "Learning"
        Reward{Comprehensive Reward Signal 
 & Interpretable Critique}
    end

    LLM -- Generates --> Output
    Output --> Verifier
    Criteria -- Guides --> Verifier
    Verifier -- Produces --> Reward
    Reward -- Reinforcement Learning --> LLM

    style Verifier fill:#f9f,stroke:#333,stroke-width:4px
    style LLM fill:#9cf,stroke:#333,stroke-width:2px
    style Reward fill:#9f9,stroke:#333,stroke-width:2px
    style Criteria fill:#e9e,stroke:#333,stroke-width:2px

Current Paths to Building a Universal Verifier

%% Diagram 2: Current Research Paths to Building a Universal Verifier
graph LR
    A(Start: The Need for
Better Evaluation) --> B

    subgraph "Path 1: Generative Verifiers (GenRM)"
        B[LLM Output] --> B1(Generative Reward Model)
        B1 --> B2["Generates Natural Language Critique 
 'The reasoning is sound but
the tone is too formal.'"]
        B2 --> B3(Critique is converted
to Reward)
    end

    A --> C
    subgraph "Path 2: Rubric-Based Systems (RaR)"
        C[LLM Output] --> C1(Decompose 'Quality' into
Rubrics)
        C1 --> C2["- ✅ Clarity 
 - ✅ Empathy 
 - ❌ Conciseness"]
        C2 --> C3(Evaluate against Rubrics for
Multi-Dimensional Reward)
    end

    A --> E
    subgraph "Path 3: Pairwise & Bootstrapped RL"
        E["Generate Multiple Outputs 
 (A, B, C)"] --> E1(Randomly Select 'B'
as Reference)
        E1 --> E2(Pairwise Comparison 
 Is A better than B? 
 Is C better than B?)
        E2 --> E3(Generate Relative
Reward Signal)
    end

Impact on LLM Evolution: The Self-Improvement Loop

%% Diagram with minor refinement for RLHF clarity
graph TD
    subgraph "Future: Autonomous Loop 
(Fast & Scalable)"
        direction TB
        F_A[LLM Generates Output] --> F_B(Universal Verifier)
        F_B -- Immediate Feedback --> F_C{Generates Perfect
Reward Signal}
        F_C --> F_D[Instantly Fine-tunes
the LLM]
        F_D --> F_A
        style F_B fill:#f9f,stroke:#333,stroke-width:4px
    end

    subgraph "Current Method: RLHF 
(Slow, Human in the Loop)"
        direction TB
        C_A[LLM Generates Outputs] --> C_B{Human Annotator}
        C_B --> C_C[Creates Preference Data]
        C_C --> C_D[Trains a separate
Reward Model]
        C_D -- Provides Reward Signal --> C_E[Fine-tunes the LLM]
        C_E -.-> C_A
        style C_B fill:#ff9,stroke:#333,stroke-width:2px
    end

The Core Challenge: Who Verifies the Verifier?

%% Diagram 4: The Recursive Challenge of Alignment
graph TD
    A[Humans build Initial
Verifier V1] --> B

    subgraph "Autonomous Improvement Cycle"
        B("Verifier V(n) evaluates LLM") --> C("LLM(n) is improved via RL")
        C --> D("Improved LLM(n+1) helps
build a better verifier")
        D --> E("Verifier V(n+1) is created")
        E --> B
    end

    E --> F(("Verifier V(n+1) surpasses
human evaluation ability"))
    F --> G{How can humans ensure
the Verifier remains
aligned with our
best interests?}

    style G fill:#c33,stroke:#333,stroke-width:2px,color:#fff

Sources

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• Generative Verifiers: Reward Modeling as Next-Token Prediction
• Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
• Reinforcement Learning with Rubric Anchors
• Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards
• Learning to Reason without External Rewards