AI screening: how to check it against your own judgement

To use AI for title and abstract screening responsibly, test it against your own judgements on a sample before you rely on it, look first at how many of your includes it missed, review every disagreement, and write down what you did.

The AI is one more classifier

Screening is classification: each record is put into include, exclude or, often, maybe. A person does this, a keyword rule does it, and a language model does it too. Treating the AI as one more classifier makes the question clear: how often does it agree with a careful person applying the same criteria, and where does it differ?

The AI can also fail outright, for example when the provider returns an error or an answer that does not fit the options. Such records should be marked for a person to look at, never quietly counted as excluded.

Build a reference sample

Take a random sample of the records you will screen and decide each one yourself, with the final criteria, before you look at the AI's answers. If two people screen the sample and resolve their disagreements, the reference is stronger.

Make sure the sample contains enough includes. In many searches only a small share of records is relevant, so a sample of 100 may hold only a handful of includes, and every figure about includes rests on that handful.

Compare: the two-by-two table

Put your decisions against the AI's in a table. Here is an example with 100 records, of which you included 10.

Agreement and Cohen's kappa, simply

Raw agreement is the share of records where you and the AI gave the same answer: (8 + 83) / 100 = 91%. That sounds good, but much of it is easy agreement on obvious excludes.

Cohen's kappa (Cohen, 1960) asks how much better the agreement is than chance. Chance agreement comes from how often each side says include. You included 10%, the AI 15%, so by chance you would agree on include 0.10 × 0.15 = 1.5% of the time and on exclude 0.90 × 0.85 = 76.5%, together 78%. Kappa is (observed − chance) / (1 − chance) = (0.91 − 0.78) / 0.22 ≈ 0.59.

Kappa runs from below 0 (worse than chance) to 1 (perfect). Verbal labels such as "moderate" or "substantial" come from conventions proposed by Landis and Koch in 1977; they are not quality thresholds. Kappa also falls when includes are rare, even with few errors, so read it next to the table, not instead of it.

Why recall comes first

The two kinds of error do not cost the same. An extra include costs a person a minute at the next stage. A missed include is gone: nobody will look at it again. That is why screening tools are usually judged first on recall, also called sensitivity: of the records you included, how many did the AI also include? In the example, 8 of 10, or 80%.

With only 10 includes, that 80% is uncertain: a 95% Wilson interval runs from about 49% to 94%. To say more, you need more includes in the sample. Decide what recall you need before you look at the results, and decide what happens to records the AI is unsure about.

Ask first how many of your includes the AI missed. Everything else comes second.

Review every disagreement

Read each record where you and the AI differ. Sometimes the AI is wrong. Sometimes you were, and sometimes the criteria are ambiguous and both readings are defensible. The last case is the most useful: it tells you which criterion to rewrite.

If you change the criteria or the instructions, test again on a fresh sample. Tuning on the same sample until it agrees tells you little about the records you have not seen.

Why people stay in control

In 2025, Cochrane, the Campbell Collaboration, JBI and the Collaboration for Environmental Evidence published a joint position statement on AI in evidence synthesis. It says authors remain accountable for their synthesis. They may use AI and automation if they can show it does not compromise methodological rigour, human oversight is required, and any AI use that makes or suggests judgements must be reported in full. The statement points to the RAISE recommendations on responsible AI use.

In practice this leaves several designs: the AI as a second reviewer next to a person, with disagreements resolved by a person; the AI as an advisor whose suggestion a person sees; or AI exclusions that a person spot-checks. Which one fits depends on the stakes of the review and on the measured recall.

Document it

• the model, its version or date, and the instructions or criteria it received;
• how the sample was drawn, its size and how many includes it held;
• recall, specificity and kappa, each with an interval;
• how disagreements were resolved and which decisions the AI made alone.

PRISMA 2020 asks reviews to report how many reviewers screened each record, whether they worked independently, and how any automation tools were used in the selection process.

How Examine Evidence supports this

In the Studio workspace, "Compare with people" runs a ruleset, workflow or tag set on a sample of 100 studies and compares it with people's answers to a screening question. It shows sensitivity, specificity, precision, the missed includes and Cohen's kappa, with 95% intervals, and a table whose cells open the studies behind them. Nothing is applied by a comparison. In a blind screening plan you are compared only with your own answers.

This check is part of Studio, not of the free quick map. Examine Evidence is built for evidence maps, which are a way to screen information and not systematic reviews. If you are doing a systematic review, follow its own standards for screening and reporting, and report any AI use as above.

Sources

• Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 1960;20(1):37-46.
• Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics, 1977;33(1):159-174.
• Position statement on artificial intelligence (AI) use in evidence synthesis across Cochrane, the Campbell Collaboration, JBI and the Collaboration for Environmental Evidence 2025. Environmental Evidence, 14:20.
• Page MJ, et al. The PRISMA 2020 statement. BMJ, 2021;372:n71. Item 8, selection process.