Dataset, task collection, benchmark and environment
A dataset is a set of records, such as documents, tables and files, used to train or analyze. A task collection is a set of problems cut from those records, each with a starting state and a defined outcome. A benchmark, or evaluation set, is a task collection kept apart from training so that capability can be measured. An executable environment adds tools, working state and a programmatic check, so a model can attempt a task many times and be scored on every attempt.
A document archive is a dataset. It becomes a task collection when someone decides what the question was and what a good answer looks like, and it becomes an environment when an attempt can be run and scored without a person in the loop.
What a record has to carry to become a task
Four things have to be recoverable: the request as it arrived, the material the people involved had at the time, the decision they made, and what happened as a result. If one is missing, the record can still be useful as text, but it cannot be graded as a task.
A task is cut at the moment of the decision. Anything written later, such as a revised drawing, a closing email or a corrective action report, can reveal the answer.
An illustrative workflow
This example was invented for this page and is not drawn from any company’s records or from any dataset on offer. A customer sends a request for quote for a bracket on a packaging line, with a specification at revision A. A quote goes out. The customer’s review comes back with a change, and the specification moves to revision B, which alters a hole pattern. Engineering updates the CAD model and the drawing and files a change order that lists the assemblies using the bracket. Fourteen months later a service call is closed with a failure code.
One chain supports several tasks, each cut at a different moment with different information. At the quote, the task is to price the job and the check is the recorded outcome. At the change order, the task is to find the affected assemblies and the check is the list engineering approved. At the service call, the task is to find the likely cause and the check is the closed work order.
Isolated CAD files and connected engineering workflows
A CAD model shows what was designed. The revision history, the reviewer comments, the change order and the supplier correspondence show how and why it changed, which is a process signal that finished files cannot give. A model trained or tested on isolated files can learn geometry.
Isolated files still suit tasks that need only geometry, such as checking that a model rebuilds or that an assembly fits. Tasks about change impact, supplier selection or root cause need the surrounding records, linked to each other and to the part.
A check can be exact, judged or mixed
Some checks are exact: a model rebuilds or it fails, a heat and material balance closes, a schematic and an I/O list agree tag for tag. Some compare an attempt with a requirement written before the work began, such as a specification line or an acceptance test. Others compare it with a recorded human decision, which involves judgment.
A verifier is deterministic only when the same attempt always receives the same score and nothing in the check depends on a model or on chance. A noisy check trains noisy behavior, so every task should say what its check measures and where it can be gamed.
Questions to put to any provider
What is the source of the material, and could it already sit in a model’s training data? What does each task contain: the starting state, the tools, the information cutoff? What does the check measure, is it exact or judged, and how was it tested against attempts that look right and are wrong? Which tasks are held out for evaluation, and who can see them?
Who authorized the use, which uses are permitted, and who may receive the material? How is personal data handled, and what is left to the contract rather than to anonymization? Which tasks proved too easy or unsolvable in a sample, and what happened to them?
Where real data helps and where it does not
Synthetic tasks are cheaper to produce and easier to verify. Which one trains a given capability better is an empirical question, and real data does not win every comparison.
RL environments for industrial work →