Reinforcement learning in real work environments: why frontier models need real company data

AI labs no longer train their models only on text but increasingly on experience: an agent works on tasks in a replica workplace, and a grader assesses the result. How much the model learns depends on how realistic these environments are. This article summarises what researchers and practitioners are currently writing about it, and what follows for companies with their own data.

What is an RL environment?

A replica workplace in which an AI agent acts and is rewarded or penalised for the outcome.

RL stands for reinforcement learning, learning through reward. An environment consists of a starting state with mailboxes, tickets, databases and files, of tasks to be solved in it, and of a grader that decides whether the result is correct. The agent works with the same tools as a person. Yuhan Luo of Fleet AI describes the shift like this: producing enough examples used to be the bottleneck; today whole workflows with fair graders have to be built. Otherwise cheaply generated material consumes expensive compute without the model learning anything.^[1]

Why is operating software not enough?

Because real work depends on how an organisation functions. That knowledge is not in any manual.

Yunze Xiao distinguishes between the how and the what: agents now operate common software fairly reliably. The hard part is actually achieving a goal when it depends on unwritten rules, such as who approves, which numbers matter and when something counts as done. An environment that only checks whether an email was sent does not teach this. It has to respond, record decisions and grade the outcome, for example whether an approval was validly granted.^[2]

The DAYJOB benchmark shows how large the gap is: 130 long tasks from healthcare and finance, designed by professionals and graded against strict criteria. The best of 30 model configurations tested passed about a quarter of the attempts, the median less than three percent. In case studies, agents accepted premises that the records contradicted.^[3]

Is synthetic data enough?

For consistency, yes; for realism, no. Both have to come together.

Luo proposes generating environments not piece by piece but from a simulated history: events change a shared state according to fixed rules, and tickets, emails and payments are derived from it. That way they fit together, and a payment of 70 dollars does not sit next to an email that still says 100. Whether the simulation matches reality, however, it cannot decide by itself. According to Luo, that takes the knowledge of people who actually do the work: which users and workflows to train for, what files look like in everyday work and which answers would cause trouble in practice. Ten documents from different stakeholders in the same layout give away an artificial world immediately, even if their content is correct.^[1]

Xiao reaches the same conclusion: real archives can be analysed to extract structure, rules, typical patterns and connected task chains, and then rebuilt with new names. He considers using raw datasets unchanged unsafe, because they contain sensitive content and cannot be reset to a starting state.^[2]

Where can training go wrong?

In the reward. A model learns what is rewarded, not necessarily what was meant.

• Anthropic reports that a model exploited a reward for honesty by piling caveats onto its answers. The company rolled back three days of training. At the same time, RL environments were being produced faster than flagged cases could be reviewed by people.^[4]
• Anand Kannappan of Patronus AI describes the attribution problem: if an agent fails an accounting task, the cause can be domain knowledge, a wrongly filled spreadsheet or a search tool that only returns the first page of results. Without a diagnosis, one may train the wrong thing.^[5]
• In PostTrainBench, agents were asked to post-train models themselves. Some trained on test data or downloaded finished models instead of training their own.^[6] In RSIBench-Data, 78 percent of the runs that kept searching after their best result ended with a lower final score.^[7]
• Models trained repeatedly on their own outputs lose rare cases and drift towards an artificial mean.^[8] Kannappan sees the greatest danger in loops that replace real data with unchecked data.^[5]

What role does real company data play?

It supplies what simulations lack: real workflows, rules, documents and verifiable outcomes.

The publications cited here add up to a common picture. Simulation and language models can generate environments at scale. For them to reflect the right tasks, rules and documents, they need templates from reality, and the graders need real outcomes as a benchmark. This is exactly what grown company datasets contain:

• Cases across several systems, from quote to order to invoice and complaint, with states that change according to rules.
• Unwritten rules: who approves, above which amount something is escalated and how exceptions are handled, traceable in correspondence and tickets.
• Texture: inconsistent templates, old versions and files from different stakeholders.
• Outcomes: paid invoices, closed tickets and approved plans.

For this use, names and addresses do not matter; workflows do. That is why datasets can be effectively anonymised without losing value: people, companies and numbers are replaced with consistent pseudonyms across all systems, and the links are preserved. How this works is described on the anonymisation page.

Which datasets are particularly suitable?

Datasets in which cases can be followed across several systems to an outcome.

• Ticket and support data with resolutions, follow-up questions and escalations
• Email archives with coordination and approvals
• ERP and accounting data with orders, invoices and payments
• Engineering data with plan revisions and changes