A system that checks citations for a living should be honest about where it breaks. These come directly from testing.
seven documented edges — click one to jump to it
VeriScite checks whatever claim, inline marker, and bibliography entry are pasted into the demo. It does not currently accept a full paper, a PDF, or an arXiv link and extract every citation automatically. Auditing a full thesis today means pasting one citation at a time.
Every test case in the development log was audited one claim and citation at a time, following the pairs listed in the testing methodology section. Batch or full-document upload has never actually been exercised.
Retrieved evidence comes from Semantic Scholar's indexed abstract for the cited paper. A citation can be entirely faithful to a claim made in the paper's methods or results section while the abstract never mentions it. VeriScite is likely to report "not enough information" in that case, a safe outcome but not a full-text check.
Not directly observed in testing yet, but it follows from how retrieval works: verify_dual only ever sees the abstract text, so a claim that's faithful to a paper's methods or results but absent from its abstract would be invisible to it regardless.
Citation resolution works by turning a bibliography entry into a keyword search. There is no direct lookup by identifier. Common author names, similar titles, or sparse bibliography entries can cause the wrong paper to be retrieved with high confidence.
Not yet observed in testing. The query-building fix in the development log (extracting first author, year, and title words rather than searching the raw reference string) was built specifically to reduce this risk, but a genuine mismatch on a common name hasn't happened in testing so far.
Semantic Scholar can return a correct match for a citation with no abstract text indexed at all, most often for older or non-open-access papers. VeriScite marks this as an unresolved citation and declines to guess. Some real, valid citations cannot be checked at all right now.
Test case: a citation to Antoniou et al. (2003), on BRCA1 mutation and breast cancer risk, resolved to the correct paper with a real paper ID, but Semantic Scholar had no abstract indexed for it at all. Treated as unresolved rather than guessed.
Testing found cases where the fine-tuned model and the independent language model verifier agreed on a support verdict that was not actually correct, both keyed off shared topical vocabulary and missing the claim's specific factual content. Agreement between the two methods raises confidence, but it doesn't guarantee correctness.
Test case: this is the same failure documented on the Research & Findings page. Both verifiers agreed on SUPPORT for a paraphrased motivational sentence checked against the SciFact paper, and again for a data-efficiency claim checked against Vaswani et al. (2017) — wrong both times.
On genuinely ambiguous claims, repeated runs of the same input have produced different final verdicts across separate sessions, even with model temperature set to zero. Each individual answer has been a defensible reading of the evidence. The system should not be treated as giving one fixed, repeatable answer to a borderline case.
Test case: the same ambiguous claim, run three separate times, returned three different final verdicts: NOT ENOUGH INFO at 0.78, CONTRADICT at 0.92, and CONTRADICT at 0.99. All three are defensible reads of evidence that never fully settles the question.
Semantic Scholar's authenticated tier is limited to roughly one request per second and is throttled further during periods of heavy platform-wide use, independent of how the requests are paced. Groq's free tier for the models used here carries its own daily token limits. Both are handled with retries and graceful fallbacks, but a citation check can occasionally take longer than expected, or fail to resolve, because of limits outside this system's control.
Test case: testing repeatedly hit real 429 responses from Semantic Scholar despite correct 1.1-second request spacing, confirmed not to be a misconfigured key via a dedicated diagnostic script. Retrieval now retries up to three times with increasing backoff (2s, 4s, 8s) before giving up.
VeriScite is well suited to checking a specific, isolated claim against its cited source when both are available in text. It is not yet a drop-in tool for auditing an entire document unattended, and its verdicts on genuinely borderline claims should be read as one careful pass, useful but provisional. Further detail on each of these points, including the actual test runs that surfaced them, is available in the project repository's development log.
View the development log on GitHub