What Google's Deep Review can check in a technical document

A technical review is useful when it changes a decision for a reason someone can check. A fluent paragraph saying that a proof seems incomplete is only a starting point. The reviewer needs to identify the statement, the missing step and the consequence before treating the critique as a finding.

Google launched Deep Review publicly on 14 October, expanding an earlier Paper Assistant Tool programme into broader technical-document feedback. The announcement describes a tool for authors and document workflows, not an automatic certificate of correctness. Google’s launch account is the dated source for its current product claims.

The strongest way to use such a system is to turn its output into a queue of testable claims. Some claims will reveal real errors; some will ask for clarification; some will be mistaken. The workflow succeeds when people can resolve those categories efficiently, rather than when the tool produces the longest or most confident review.

The reviewer’s output is another document to verify

A manuscript makes claims about definitions, reasoning, experiments and prior work. An automated review makes claims about that manuscript. It therefore creates a second verification task: does the critique accurately describe the original, and would its proposed correction improve the work?

This is not a reason to reject automated feedback. Human reviews also require interpretation and can contain mistakes. The practical implication is to preserve an evidence chain. A useful finding identifies a passage, states a problem and supplies a reason the problem matters. “The methodology is weak” may be a reaction; “the reported denominator excludes the failed trials described in the methods” can become an auditable issue.

The Paper Assistant Tool paper describes segmenting manuscripts, assigning specialised review work and synthesising the results. It also distinguishes its selected mathematical-error evaluation from the original benchmark’s broader protocol. That orchestration and measurement history should not be reduced to the claim that a model simply reads a paper and knows whether it is correct.

For an author, the first step is to stabilise the document version. Page numbers and theorem labels can move between revisions. A critique that refers to an earlier lemma may appear nonsensical after an edit. Preserve the reviewed file and associate findings with it before applying changes.

Next, separate factual claims from preferences. “The proof uses an assumption not stated in the theorem” is a claim about validity. “The theorem should appear earlier” concerns presentation. “The paper is insufficiently novel” may depend on literature coverage and judgement. These categories deserve different resolution procedures and should not share one undifferentiated severity label.

A practical review ledger can be small. Record the passage, proposed issue, supporting evidence, owner and outcome. The ledger is useful because it prevents a corrected typo, a rejected criticism and an unresolved mathematical gap from all becoming “feedback addressed”. The status should say what happened.

Our model-card guide uses a related principle: evidence must match the claim a reader is making from it. For technical review, the evidence needs to match both the manuscript’s assertion and the reviewer’s criticism. A convincing explanation of the wrong passage does not resolve the actual question.

The conference pilots had a particular role

Google’s STOC account describes a programme aimed at theoretical computer science authors. The task was to provide feedback before submission. That role is different from a programme committee deciding which papers to accept, and it should remain different when the pilot is cited as evidence of usefulness.

The ICML retrospective reports approximately 4,500 papers receiving feedback and a survey with 869 responses. It gives distinct figures for reuse intention, helpfulness and changes prompted. These are author-reported outcomes from a particular programme, rather than independently adjudicated accuracy for every review comment.

NeurIPS’s announcement explicitly separates optional private author feedback from the official review process. It also describes changes such as stronger search and related-work support. That means different programme iterations should not automatically be treated as one frozen system with one performance number.

The later ICLR programme likewise makes the feedback private to authors and outside official review. Its dates refer to a specific access window. A later public launch can build on that experience without making every earlier programme condition part of the current service.

The correct inference is that authors found useful opportunities for improvement in these programmes. A stronger claim that the system reliably identifies all important flaws would require a different evaluation. A claim that it can rank papers for acceptance would require another evaluation again, including a clear definition of the decision being assessed.

This distinction makes the pilot evidence more useful. An author choosing whether to request an additional critique can reasonably care about whether other authors found it worthwhile. A conference considering delegation of consequential decisions needs evidence about errors, fairness, confidentiality and process integrity. Those are different purchasing and governance questions.

Work a mathematical criticism down to a counterexample

Consider an invented manuscript statement: for every positive real number x, a single positive number c exists such that c is smaller than x. The wording is ambiguous unless the order of the quantifiers is made explicit. Is c allowed to depend on x, or must one c work for all positive x?

If the claim means “for each x there is a c”, choosing c equal to half x works. If it means “there is one positive c for every positive x”, choose x equal to half c and the claim fails. The arithmetic is elementary, but the logical distinction can become easy to miss in dense notation.

An automated reviewer might flag the passage as a quantifier problem. That is useful only if it identifies which interpretation the theorem actually states. If a definition elsewhere already allows c to depend on x, the criticism could be mistaken. The author should inspect the full dependency structure before rewriting the theorem to satisfy the review.

The resolution should preserve the argument. If the proof needs one universal constant, merely clarifying that c depends on x does not repair the result. It changes the property being established. The team must revisit any later step that relies on uniformity rather than treating an improved sentence as a completed mathematical fix.

This example is original explanatory material, not a finding produced by Deep Review. It illustrates an evidence ladder: a suspicion about wording, a precise logical claim, a candidate counterexample, confirmation that the counterexample satisfies the assumptions and a determination of what must change.

The same structure works for a claimed missing assumption. A reviewer might propose a case where division by zero occurs. The author checks whether the domain already excludes zero. If it does, the criticism fails. If it does not, the author must determine whether adding the exclusion preserves the intended theorem and application.

A particularly important habit is to search for a minimal counterexample. A long narrative can obscure the essential issue. A short construction satisfying the premises and violating the conclusion gives both author and reviewer something concrete to inspect. Conversely, failure to produce a counterexample does not prove the theorem true; a different proof obligation may remain.

Clarification and correction should not be confused

Sometimes a review reveals that the manuscript is correct but difficult to interpret. The right response may be a definition, an explanatory sentence or a reordered proof. That is a real improvement, but it should not be reported as the discovery of a fatal error.

Sometimes a notation change exposes a genuine gap. In that case, presentation alone cannot resolve it. The author needs a repaired argument, a narrower claim or withdrawal of the result. A useful review process allows each outcome without rewarding dramatic labels for their own sake.

Record which category applies. That record helps assess the tool later: did it find validity problems, improve readability or create unnecessary work? Combining all three into a satisfaction score would hide the difference between valuable scrutiny and expensive misunderstanding.

Experimental critique needs matched quantities

An invented experimental report might say that a new system is twice as efficient because it processes twice as many examples. That statement is incomplete until the resource and workload are specified. Twice the examples in twice the time does not demonstrate twice the throughput. Twice the examples on twice the hardware introduces another denominator.

A useful reviewer should identify the missing quantity. Suppose the manuscript reports completed examples and elapsed time but omits failed attempts. The critic can ask whether the efficiency claim is per attempted example, per successful example or per usable result. The answer may change the calculation materially without changing any raw measurement.

The author should reconstruct the calculation from recorded values. If failures consume resources, omitting their cost can make a method look more efficient than the deployed workflow. If an experiment intentionally measures only a narrow kernel, it can still be valid, but the claim should name that boundary rather than imply an end-to-end saving.

A second invented case involves a baseline. The new method receives a carefully tuned prompt while the baseline uses an older default. An automated critique might call the comparison unfair. To resolve it, the author needs to inspect the research question: is the paper comparing complete practical recipes or isolating one component? The same experimental design can support one claim and fail to support another.

A matched comparison would keep relevant conditions consistent or explain why they differ. This includes data splits, sampling budgets, stopping rules and model selection. A reviewer who asks for matching should identify which difference could explain the reported result, rather than list every conceivable configuration detail indiscriminately.

A third case concerns a percentage. A rise from forty to fifty is ten percentage points and a relative increase of one quarter. These descriptions answer different questions. The reviewer can verify the arithmetic, but the author still needs to choose wording that fits the scientific claim and report the underlying values.

These examples show why automated feedback can be valuable even without a final verdict. It can surface a denominator, an omitted baseline condition or an ambiguous calculation. The author then performs the decisive work: checking the record and revising the claim to match what was measured.

A missing experiment is a proposal, not an observed failure

A review may suggest an additional ablation or dataset. That suggestion can be useful without establishing that the present result is wrong. The author should distinguish a required test of a central alternative explanation from a speculative extension outside the paper’s scope.

The decision depends on the claim. If a paper attributes a gain to one component while several changed together, an ablation can be necessary to justify that attribution. If the paper reports a result in one domain without claiming universality, testing every domain may be future work rather than a prerequisite for the stated conclusion.

A good response explains the boundary. Run the experiment when it resolves a material uncertainty, narrow the claim when the evidence is insufficient or explain why the suggestion concerns a different question. Treating every automated suggestion as mandatory can inflate a paper without making its argument stronger.

Error benchmarks and satisfaction surveys answer different questions

SPOT assembles papers with known errors associated with errata or retractions. Its evaluation examines whether systems rediscover those errors and whether proposed findings are correct. It is relevant because error detection requires both finding true issues and avoiding false ones, rather than merely producing plausible criticism.

The selected mathematical subset used in the PAT work differs from that full benchmark, including its grading approach. Consequently, a number from one protocol should not be placed beside another as if only the model changed. A reader needs to know the task population, scoring rule and what the denominator contains before interpreting an apparent improvement.

Recall asks how many known issues were found. Precision asks how many proposed issues were valid. A system can improve recall by producing more accusations, while increasing the time authors spend rejecting false ones. The useful balance depends on how expensive each finding is to check and how consequential a missed issue would be.

An author survey measures another outcome. Someone might find a report useful because one suggestion improved clarity, despite rejecting several criticisms. Another author might receive no useful issue because their paper had none in the tool’s detectable range. Satisfaction is meaningful evidence about experience, but it does not determine error-detection sensitivity or completeness.

A practical evaluation should retain these separate measures. Count verified issues and rejected findings. Record the time needed for resolution. Identify which issue classes were tested. Report whether the system’s output changed the manuscript in a way independent reviewers judge beneficial, rather than equating more edits with better science.

The ICML policy experiment studies yet another question: how different rules about LLM use relate to reviewer behaviour and outcomes. It reports limited effects on decisions in its randomised setting alongside self-reported noncompliance. It is not a benchmark of Deep Review, but it shows why policy assignment, actual use and review quality must not be treated as identical variables.

Even first-party summaries need reconciliation. The ICML retrospective and the PAT manuscript present survey figures with different sample descriptions and some different percentages. This article uses the retrospective’s own programme and response counts when describing that event, rather than combining figures across the two sources. A reader should follow each figure back to its stated question and population before making a comparison.

This is itself a useful review task. If a document quotes a survey statistic, ask whether it refers to all respondents, authors with theoretical results or authors with experimental results. A percentage about a subset cannot be relabelled as the share of every participant. The denominator and wording belong with the result. An automated reviewer that notices the discrepancy should preserve the alternatives and request clarification, rather than silently choose the larger value or average incompatible figures.

For an organisation buying document-review assistance, the appropriate study could be much smaller and more specific. Use representative documents with independently established issues and clean controls. Ask specialists to adjudicate candidate findings without being told which system produced them. Include ordinary documents, rather than only famous flawed papers that may be easy to recognise.

A literature claim needs the paper behind the citation

ScholarPeer, whose components the NeurIPS programme describes incorporating, separates literature contextualisation, baseline discovery and technical scrutiny into distinct roles. Its paper describes a historian, a baseline scout and a question-and-answer auditing engine. That separation is useful because locating a related title and establishing that it supports a claim are different tasks.

A reviewer might say that a manuscript omits a decisive prior result. Before the author accepts the criticism, identify the result precisely. Does the cited work solve the same problem, use the same assumptions and provide the comparison the reviewer requests? Two papers can use similar terminology while addressing different tasks. A title-level resemblance cannot establish that the newer manuscript lacks novelty.

The reverse mistake is also common: a reference exists, but it does not support the sentence attached to it. The paper may report a narrower population, a different model version or a conditional result. A useful automated review should make that mismatch inspectable by linking the relevant passage, rather than producing a citation that merely looks authoritative.

Crossref’s REST API documentation explains how bibliographic metadata can be retrieved from member deposits and trusted sources. A known DOI can be resolved to a record, and work records can include titles, identifiers, publication details and update information. This can help establish that a reference points to a real scholarly object. It does not establish that the object’s findings support a particular argument.

Consider an invented manuscript claiming that a training method reduces total deployment cost. Its reference may be a real paper reporting reduced training memory on one model. Metadata checking confirms the citation’s identity. Reading the methods and results reveals the narrower scope. The author must either find evidence for the deployment-cost claim or rewrite the sentence to describe the measured memory result.

The distinction produces a practical sequence. First resolve identity: title, authors, identifier and version. Then inspect relevance: task, assumptions, data and outcome. Finally inspect support: the exact relation between the manuscript’s sentence and the cited result. A system that skips the final stage can create technically correct bibliographies attached to unsupported prose.

Version and date require special care. Crossref’s query guidance distinguishes publication, creation, update and indexing dates. An indexing change is not necessarily a newly published scientific result. A review that treats every recent metadata timestamp as a new paper can manufacture an apparent missing-literature problem.

A preprint and a later conference version can also differ in theorem statements or experiments. The author should cite the version that supplies the claimed evidence and check whether a later correction changes it. If the automated report links a different version, resolve the mismatch before accepting its critique. Bibliographic precision is part of scientific precision.

Search coverage should be visible in a novelty claim

A claim that no previous work exists is harder to support than a claim that one relevant paper was found. Search can fail through terminology, indexing gaps or access restrictions. An automated reviewer should not turn an unsuccessful query into proof of novelty, and an author should not treat a list of search results as an exhaustive history of a field.

A useful report can state the concepts and neighbouring terms searched, identify the most relevant work found and explain what comparison it supports. This gives a domain expert something to extend. If the expert knows a separate research tradition using different language, that knowledge can improve the search rather than compete with a confident but incomplete generated verdict.

In our invented cost example, related work might discuss training efficiency, inference efficiency or systems utilisation under different names. A baseline scout should identify which category is relevant to the manuscript’s claim. Requesting a comparison with every superficially related system can waste time; omitting a directly comparable result can undermine the argument. The distinction needs task-level understanding.

The author can respond with a short evidence table: candidate paper, shared problem, important difference and required manuscript change. One row may lead to a new experiment. Another may require a citation and a narrower novelty sentence. A third may be out of scope with an explanation. This is a concrete way to turn broad literature feedback into a defensible revision.

An automated review can accelerate discovery, but it should leave the scientist with a traceable chain rather than a list of names. The decisive question is whether the newly inspected evidence changes the result, the comparison or the wording. If none changes, the report should explain why instead of retaining an unresolved accusation of missing work.

Confidentiality and decision authority remain outside a launch claim

Permission to use a document is not implied by a tool’s availability. An author reviewing their own work, a reviewer holding a confidential submission and an employee assessing a customer’s contract can have different obligations. The tool’s technical capability does not collapse those roles into one authorised use.

The ICML 2026 LLM policy provides a concrete example. It distinguishes a conservative policy from a permissive one, limits the permitted assistance under the latter and keeps responsibility with the reviewer. A public author-feedback programme does not override the policy applying to an assigned confidential review.

Before using any service, identify who owns the document, who can approve processing and which current terms apply. Retention, training use and access are different questions. A statement about one should not be translated into an assumption about all of them. Product conditions should be checked directly at the time a team intends to upload material.

Decision authority is equally important. A suggested weakness is not an acceptance decision. A useful automated report can support a human without becoming a hidden final reviewer. If an organisation wants to delegate more authority, it needs to evaluate that workflow explicitly and provide a process for challenging mistakes.

An author should be able to contest a criticism through evidence. The resolution process should ask whether the statement is true, not whether a sophisticated tool produced it. Otherwise, automation can give an incorrect critique institutional weight while leaving the author responsible for disproving an unspecified objection.

This is where our AI governance framework guide becomes practical. Name the decision, evidence and responsible person. A review tool is easier to use well when its output has a defined role and a route to resolution. Ambiguous authority makes both correct and incorrect findings harder to handle.

A useful review ends with a verified change or an explicit open question

A finished workflow should produce more than a long response. For each important issue, keep the original passage, the critique, the checking evidence and the resulting decision. Mark presentation improvements separately from validity repairs. Keep rejected findings so that recurring misunderstandings can inform later evaluation.

Recheck the revised document. A corrected formula can alter later results; a new experiment can change the description of the dataset; a narrowed theorem can invalidate an application claim. The resolution is complete only when those dependencies are addressed, not when the first criticised sentence disappears.

Leave unresolved questions visible. If the author cannot reproduce an experiment or verify a citation, label the gap and adjust the claim accordingly. A review process that always ends with “all issues resolved” can conceal uncertainty more efficiently than it removes it.

Deep Review’s interesting contribution is to make additional technical scrutiny easier to request. The valuable unit is still a checkable finding: a precise issue, relevant evidence and a defensible resolution. Use the tool to increase that scrutiny, and require every important correction to earn its place in the document.