This page covers two things: the research question that motivated VeriScite before any code was written, and the concrete findings that came out of testing it against real papers and real citations.
This project sits inside a broader interest in natural language inference, transfer learning, and the interpretability and reliability of machine learning models used for scientific claim and multimodal verification. The starting question was narrow: can a fine-tuned NLI model tell whether a piece of evidence supports a scientific claim, and can it explain that answer, beyond a bare label.
That question produced clAIm, a two-stage NLI system trained first on MultiNLI and then on SciFact, paired with token-level attribution through Integrated Gradients and retrieval through Semantic Scholar. Building clAIm surfaced a limitation directly in its own write-up: the system checks whether a supplied sentence textually entails a claim. It does not check whether a citation, once resolved from a bibliography entry, still says what it is cited for. Judging that kind of faithfulness means resolving a real citation and retrieving the paper it points to first. It is a harder problem than judging entailment on a hand-picked sentence.
VeriScite answers that gap directly. It reuses clAIm's fine-tuned model as one of two independent verifiers, and adds the parts clAIm does not have: citation resolution from a bibliography entry, a second independent verification method, and an agent that resolves disagreement between the two on its own. The Agentic Workings page describes what was built. This page covers what testing it revealed.
An early version of the disagreement-resolution step followed a fixed script: retry retrieval once, and if the two verifiers still disagreed, ask a third model to pick a side. That is a workflow. The control flow was decided in code before any claim was ever checked, and a model's output only ever fed into branches already written. A common distinction in agent design literature separates this from a true agent, where the model itself chooses its next action and decides when it is done. On that distinction, the fixed-script version did not meet the bar.
The escalation step was rebuilt around a ReAct-style loop, short for reason, act, observe, where a planner model gets tools and decides for itself which to call, in what order, and how many times, up to a set limit. Two principles from the surrounding literature shaped the rebuild directly. First, a critic model tends to favor reasoning that resembles its own, so the planner always uses a third model, distinct from both original verifiers. Second, when evidence is genuinely inconsistent, a well-designed system should say so rather than force a confident answer. A calibrated "not enough information" is a complete, legitimate result on its own.
Concrete results from testing the system against real papers and real citations.
On two test cases, both the fine-tuned NLI model and the zero-shot language model verifier independently agreed on a support verdict that wasn't actually correct. The claim and evidence simply shared topical vocabulary, terms like "transformer" or "attention" or "scientific claim verification," without the evidence making the specific assertion in the claim. Two independent methods failing the same way suggests the problem may sit in the task itself: matching a short claim against abstract-level evidence is hard on its own terms, whichever model attempts it.
"Transformer models require substantially more labeled training than architectures to reach comparable accuracy."
"The architecture trains faster than models and generalizes well under limited , using attention in place of recurrence."
Every highlighted term appears on both sides. None of them establish the specific comparison the claim makes.
The clearest evidence that the ReAct rebuild was worth doing came from re-running the same claim that had exposed the shared-bias problem above.
Drag to compare, or use ← → when focused.
| Version | Verdict | Confidence |
|---|---|---|
| Fixed-script escalation (before rebuild) | SUPPORT (incorrect) | 0.9931 |
| ReAct agent (after rebuild) | NOT ENOUGH INFO (correct) | 0.78 |
The claim concerned how much labeled training data transformer models need compared to recurrent architectures. The cited abstract discusses training speed and generalization to limited data, but never directly compares data volume requirements. The old fixed script forced a confident answer regardless. Given the freedom to reason, the new agent correctly identified that the evidence does not settle the question, and said so.
Running that same claim several times produced three different final verdicts across separate runs: not enough information after three retrieval attempts, contradict after a single reasoning step, and contradict again with higher confidence on a later run. Each is a defensible single-pass reading of evidence that genuinely does not settle the question either way, and each is a real improvement over the old script's confident, incorrect support verdict. The variation itself is worth stating plainly: the agent's conclusions on genuinely ambiguous claims are not fully deterministic, even with model temperature set to zero. That is one more reason to treat "not enough information" as a real, first-class outcome.
Semantic Scholar can return a real, correctly matched paper for a bibliography entry while providing no abstract text at all, most often for older or non-open-access papers. VeriScite treats this the same as a failed search rather than proceeding with empty evidence, and reports the claim as unresolved rather than guessing.
Wadden et al. (2020), "Fact or Fiction: Verifying Scientific Claims", introduced the SciFact dataset and task that clAIm's model is fine-tuned on. Panickssery et al. (2024) document that language model critics tend to favor outputs resembling their own reasoning, which is why VeriScite's planner and second-opinion checks run deliberately on models distinct from the verifier they are checking. A concurrent line of work, including systems built specifically around claim-citation alignment, addresses closely related ground and is worth reading alongside this project.