Privileged World Supervision:  Training Agents to Know What They Don't Know

Most reinforcement learning approaches to information retrieval struggle with the same problem: rewards are sparse, noisy, and hard to define when the task is open-ended. Privileged World Supervision (PWS) takes a different route. Instead of trying to evaluate agents in the real world, we build synthetic worlds where the ground truth is fully known, and use that access to compute exact reward signals.

The idea is simple but powerful. An agentic pipeline generates coherent synthetic corpora and knowledge graphs, complete with realistic noise, contradictions, and missing information. Because we control the environment, we know exactly which claims are true, which sources are reliable, and what the correct graph structure looks like. This lets us train RL agents (using PPO without a critic network) with precise per-action advantage signals, avoiding the instability that comes with standard value approximation.

To bootstrap performance before RL optimization, I fine-tuned Qwen3-30B-A3B using LoRA adapters on expert trajectories generated within these environments. The synthetic world generation itself relies on Claude 4.5's API to produce diverse, internally consistent document ecosystems across multiple domains.

This work is still exploratory and actively evolving. The long-term goal is to show that training in privileged environments can transfer to real-world information retrieval, where ground truth is never fully available but the reasoning skills learned in simulation still apply.