Pilot-testing a data extraction form

Pick pilot studies that test the form

A pilot only helps if its studies can expose problems. Include a study with more than two arms, one with several follow-up times, and one whose results appear only in a table or figure. The Cochrane Handbook recommends that several people extract data from at least a few articles when testing a form.

In eviStreams, a pilot runs the prepared form on papers that have finished processing. You can pick the papers or take a random sample, up to ten per pilot; pick them yourself when you know where the difficult cases are.

Check each answer against the paper

Start with whether each question is clear and complete, then check accuracy. A number can be copied correctly and still come from the wrong arm, time point, or analysis population.

1. Run the prepared form on the pilot papers.
2. Open the source passage behind each answer and confirm the arm, time point, unit, and denominator as well as the value.
3. Mark each answer correct or wrong, and enter the correct answer where it is wrong.
4. Where an error comes from an unclear definition, rewrite that field’s instructions and save the form.
5. Run another round on the same papers and check whether the revised fields now give consistent answers.

Fix the definition, not only the answer

If two careful readers could defend different answers, the field definition is the problem. State the denominator, arm, time point, and unit to capture, and what to record when the paper is silent. Agree the change as a team before editing the form.

In eviStreams, marking an answer wrong opens that field’s instructions: its description, where to look, rules, and examples. You can record the correct answer and why it went wrong, edit the instructions yourself, or ask for an AI-suggested revision, which changes nothing until you apply it.

Collect results as the paper reports them, such as a median and range, and convert them later in the analysis with the method noted.

Decide when the pilot is finished

Stop when a round gives consistent answers and no field needs rewording. Another round in eviStreams re-runs the same papers with your saved changes. If the form changed substantially, the Cochrane Handbook suggests repeating the pilot on a new set of reports.

Ratings and corrections sent with a new round are kept as worked examples for the form’s later AI extraction, so check your corrections as carefully as the extraction itself. A pilot calibrates the form; it does not replace independent review of the full extraction.