Turned On the AI Reviewer for Scientific Reproducibility

The screenshots in this article are illustrative diagrams of a run for reference only and do not represent a validated research result.

TL;DR

An AI reviewer for scientific reproducibility is an independent review pass, not a language edit. I ran Kimi K3 through a bioinformatics reproduction again with the built-in AI reviewer in AIPOCH open-science enabled. This was not a human researcher checking the work or a journal peer reviewer commenting after submission. It was a separate AI review pass inside the workbench, reading the agent's preserved run record and sending corrections back into the workflow.

The reviewed run corrected the cohort definition, restored the paper's gene lists and DEG threshold, and ultimately changed which biomarkers the analysis supported. This was not a better-written version of the same result; it was a materially better scientific process.

The reviewer did not catch every problem, but that does not diminish what changed. It shows that an independent review layer is already useful before it becomes domain-complete, while giving us a concrete path for improving the scientific rules it applies next.

Why Does an Independent Review Layer Matter for Reproducibility?

An independent review layer matters because a scientific agent rewarded by momentum will finish an altered analysis rather than question it. In my previous two articles, I wrote about a Kimi K3 run inside AIPOCH open-science that reproduced the computational workflow of a published ulcerative-colitis biomarker paper, Sheng et al., 2026 in PLOS ONE. The model did a surprising amount of work. It downloaded and integrated two GEO datasets, recovered from a failed batch-correction step, ran differential-expression and machine-learning analyses, and left behind scripts, logs, tables, figures, and a final report.

The project looked complete, but the preserved record showed that several important scientific decisions had changed along the way. Samples from inflamed and uninflamed tissue had been placed into disease and control groups in a way that changed the biological comparison. The paper's mitophagy- and hypoxia-related gene lists had been replaced with much smaller lists from other sources. The differential-expression threshold had been lowered. A dataset used in feature discovery and model fitting was later described as an independent validation set.

That first run became the basis of A scientific agent should know when not to make the figure. My argument was that a scientific agent needs more than the ability to recover from errors and keep moving. It needs a way to distinguish an engineering repair from a scientific substitution, and a substitution from an evidence failure that should block the next artifact.

The obvious next question was whether a reviewer would actually change any of this. A reviewer can appear in an architecture diagram without telling us what it contributes in practice, and review is often reduced to a final language pass that adds caveats after the code, figures, and conclusions have already been produced. I wanted direct evidence that enabling the reviewer could change the scientific work itself.

Here, "reviewer" does not mean that a human bioinformatician inspected the project between runs. It refers to the AI reviewer built into open-science: an independent model role that can inspect the acting agent's transcript, execution activity, and generated artifacts, then return findings and trigger a bounded correction loop. The distinction matters because the question is not whether expert human review improves an analysis; we already know it does. The question is whether the workbench itself can introduce a useful second line of judgment before the researcher has to reconstruct the entire run manually.

The answer was clearer than I expected. The most important difference was not that the second report sounded more cautious. The reviewed run changed the samples entering the analysis, the gene sets defining the candidate space, the threshold determining which genes moved downstream, and eventually the biomarkers that survived the workflow. The reviewer mattered because it changed the inputs, not just the wording.

What Did the First Correction Change About the Research Question?

The first correction changed the research question itself, not a cosmetic detail. The first run used 22 inflamed samples from GSE179285 as UC and 31 uninflamed samples from people with UC as controls. Combined with GSE75214, this produced 96 UC and 42 control samples.

That grouping is not a harmless approximation. It mixes two biological comparisons. GSE75214 contributes active UC tissue and healthy controls, while the selected GSE179285 samples compare inflamed and uninflamed tissue within UC. Once those samples are combined under the same UC and Control labels, the model may learn inflammation status, disease status, platform differences, or some mixture of all three. A high AUC can still be calculated, but it no longer answers a clean question about diagnosing UC against healthy controls.

In the reviewed run, the sample selection was changed to 23 UC and 23 controls from GSE179285. Together with GSE75214, the combined cohort became 97 UC and 34 controls, matching the composition reported in Sheng et al., 2026.

This is the first place where the reviewer clearly earned its role. It did not merely notice that two sample counts were different. It surfaced a correction to the definition of the groups entering every downstream calculation. That decision affected batch correction, differential expression, feature selection, model fitting, ROC analysis, and the interpretation of the final biomarkers.

A scientific workflow can survive a misspelled filename without changing its question. It cannot silently survive a different control group. Once the control definition changes, every clean-looking figure downstream belongs to a different experiment.

Reviewer effect 1

The reviewed run moved from 138 samples with a mixed control definition to the paper's 131-sample composition. The correction changed what the analysis was comparing, not just how many rows it contained.

How Did the Reviewer Keep Unavailable Inputs From Becoming Different Inputs?

The reviewer stopped unavailable inputs from quietly becoming different inputs. The first run encountered a common reproduction problem: Sheng et al., 2026 described 1,680 mitophagy-related genes and 2,360 hypoxia-related genes, but the original GeneCards query was not directly available through the running environment. The agent responded by assembling replacement lists from GO, MSigDB, and curated sources. That gave it 143 mitophagy-related genes and 452 hypoxia-related genes.

From an engineering perspective, this was a reasonable way to keep moving. From a scientific perspective, it changed the candidate universe before the analysis had even started. Intersecting the paper's DEGs with a few hundred substitute genes is not equivalent to intersecting them with the paper's several thousand genes. The first run obtained only nine MHRDEGs and later built a two-gene model around PPARGC1A and BNIP3. Those may be interesting genes, but they were the output of a different input definition.

The reviewed run went back to the paper's Supporting Information and recovered the published MRG, HRG, MHRDEG, TF, and miRNA lists from the supplementary tables. Instead of treating unavailable database access as permission to redefine the analysis, it looked for another source of the same research objects.

That distinction is fundamental for scientific agents. A substitute package can sometimes execute the same statistical operation. A substitute gene list usually changes the scientific object being studied. Both may appear to the agent as missing-resource problems, but they require different review decisions.

The reviewer also changed how downstream uncertainty was represented. The reviewed run found 71 MHRDEGs, including 51 of the paper's 60 reported genes. Rather than forcing those 71 genes into a single narrative, it kept a paper track based on the published gene sets and a de novo track based on the newly calculated intersection. The final report explained that the paper's exact 20-gene logistic subset was not uniquely recoverable and that LASSO selection varied with the input matrix, solver, and lambda choice.

This is much better than allowing an alternative analysis to overwrite the original one. The paper track asks whether the published result can be reconstructed from the information the authors released. The de novo track asks what the current implementation finds from the available data. Both are useful, but they are not interchangeable.

Reviewer effect 2

The reviewed run replaced a silent change in the candidate-gene universe with two visible analysis tracks: one anchored to the paper's released objects and one derived from the current data.

How Did the Threshold Correction Affect Downstream Results?

The threshold correction changed every downstream result because it sat at the head of the analysis chain. The first run lowered the combined differential-expression threshold from the paper's abs(logFC) > 0.5 to abs(logFC) > 0.2. The stated reason was to compensate for cross-platform log-fold-change shrinkage.

There is a legitimate scientific discussion to have here. Cross-platform integration can compress effect sizes, and sensitivity analyses across thresholds can be useful. The problem is not that 0.2 can never be examined. The problem is that the threshold was changed inside the primary reproduction path and then the resulting workflow was described as faithfully reproduced.

The reviewed run restored the published abs(logFC) > 0.5 threshold. It obtained 2,306 combined DEGs, compared with 2,276 in Sheng et al., 2026, a difference of 1.3%. The separate datasets produced 3,592 and 2,629 DEGs, within roughly 4% and 2.3% of the reported counts.

These numbers do not prove that the preprocessing is uniquely correct. The reviewed run explored several log transformations, normalization choices, and ComBat variants, and the choice of a variant partly depended on agreement with the paper's DEG counts. That creates its own review question, which I will return to later. But the important improvement is that the primary threshold was no longer quietly altered to keep the workflow moving.

A reviewer should not forbid sensitivity analysis. It should prevent the sensitivity analysis from silently becoming the reproduction. The original specification, alternative specification, reason for the change, and downstream impact should remain connected.

Reviewer effect 3

The reviewed run restored the paper's threshold and made method sensitivity visible instead of allowing a lower threshold to become an undocumented replacement for the original analysis.

Did the Scientific Answer Itself Change?

Yes. The most convincing evidence for the reviewer's importance is that the final biological answer was no longer the same. In the first run, the substituted gene lists and altered threshold produced nine MHRDEGs. SVM-RFE ranked PPARGC1A, BNIP3, and CPT1A as the top three genes, while LASSO retained PPARGC1A and BNIP3. The final report described the resulting two-gene model as a successful implementation of the paper's workflow, even though the paper's main biomarkers were CD55 and CPT1A.

In the reviewed run, CPT1A was the most stable signal across the paper-based track, the de novo LASSO analysis, and the cohort-level AUC calculations. CD55 was treated more cautiously: its AUC was above 0.9 in the combined data but 0.802 in GSE179285, so the report described its support as partial rather than presenting it as uniformly reproduced. The LASSO solution was also no longer compressed into one apparently definitive gene list. At lambda.min, eight of the paper's ten genes were recovered; the more strongly regularized lambda.1se solution was sparser.

This is the strongest result of the comparison: upstream review changed which scientific claim was supportable. A reviewer that only checks grammar would leave the original two-gene model intact and add a paragraph about limitations. This reviewer changed the data and method choices that produced the model, producing a project that was substantially closer to the paper and much easier to inspect.

Analysis decision	Reviewer off	Reviewer on	Downstream effect
Combined samples	96 UC + 42 Control	97 UC + 34 Control	Restored the paper's cohort composition
MRG / HRG source	143 / 452 substitute genes	1,680 / 2,360 genes from Supporting Information	Rebuilt the candidate-gene universe
Combined DEG threshold	`abs(logFC) > 0.2`	`abs(logFC) > 0.5`	Restored the published primary analysis
MHRDEGs	9	71, including 51/60 paper genes	Changed the feature-selection input
Main model result	PPARGC1A + BNIP3	CPT1A most stable; CD55 partially supported	Changed the final biomarker interpretation

This is the argument for keeping review independent from generation. The acting agent is rewarded by momentum. Once it has repaired an environment, produced a matrix, trained a model, and generated a clean figure, every additional step makes the current trajectory feel more complete. A reviewer can return to an earlier decision without carrying the same investment in finishing that trajectory.

When Should an AI Reviewer Intervene in a Workflow?

An AI reviewer should intervene as early as possible, because scientific errors compound downstream. If an agent catches a typo in the final report, it saves a sentence. If it catches the wrong control definition after the ROC has been generated, it may require a full rerun. If it catches the same problem before batch correction, it can prevent the wrong comparison from entering differential expression, feature selection, model training, calibration, DCA, and the final conclusion.

This is why I do not think reviewer performance should be measured only by the number of findings. Ten comments about formatting are less valuable than one finding that corrects a dataset role before the downstream branch expands. A useful scientific reviewer changes decisions at the point where those decisions can still change the result.

There is also a direct resource consequence. Every unsupported branch consumes model tokens, execution time, and human attention. The first run did not stop after constructing a model on altered inputs. It continued into ROC curves, a nomogram, calibration, decision-curve analysis, immune analysis, and a final narrative. Once an upstream condition has failed, pushing ahead becomes expensive. The system is spending more tokens making an unsupported branch look complete and then spending additional reviewer and researcher time undoing that completeness.

The reviewed run demonstrates the positive version of this principle. Correcting sample definitions, input lists, and thresholds upstream is more valuable than asking the model to rewrite the conclusion after all artifacts have been generated. Reviewer cost should therefore be compared with the work it prevents, not only added to the cost of the acting agent.

What Does the Reviewer Still Need to Learn?

The reviewed run was materially better, but it left one important validation issue unresolved, which helps define the next capability the reviewer needs. The largest unresolved issue was validation independence. GSE75214 and GSE179285 were merged for differential expression and the combined matrix was used for candidate selection and machine-learning analysis. The workflow then processed GSE179285 separately and called it a validation cohort. But a dataset does not become unseen again because it is reloaded from a raw matrix. If it influenced feature discovery or model selection upstream, its later AUC is not independent external validation.

The reviewed report still presented a ten-gene model AUC of 0.994 on the combined data and 1.000 on GSE179285. It still generated ROC, calibration, and decision-curve artifacts. Its conclusion still referred to near-perfect validation performance and said that the final biological conclusions were supported by an independent reproduction.

This is an important domain rule rather than a reason to discount the reviewer's other corrections. The reviewer corrected the composition of GSE179285 but did not preserve its role across the full analysis graph. It understood which samples belonged in the cohort; the next step is to ensure that it can also enforce what that cohort is allowed to become later.

The reviewed run also selected among preprocessing variants partly by asking which one reproduced the paper's DEG counts most closely. Agreement with a published result is useful diagnostic evidence, but it should not be the sole rule for selecting a primary preprocessing pipeline. Otherwise the known result becomes a tuning target. A stronger review process would preserve the tested variants, require an independent methodological reason for the primary choice, and present count agreement as a consequence rather than the selection criterion.

Several substitutions also remained compressed into a strong overall conclusion. The original R workflow was reimplemented in Python. MCP-counter signatures for ten immune populations replaced the paper's 28-cell ssGSEA resource. Some nomogram, functional-similarity, and chromosome analyses were approximated or skipped. The report disclosed these points, which is a real improvement, but still opened by saying that all computational analyses were reproduced end to end and closed by calling the paper broadly reproducible.

A reviewer can materially improve a scientific run even before it has learned every domain rule.

This is why reviewer evaluation should be treated as seriously as model evaluation. The first question is whether the reviewer changes consequential decisions, and this run shows that it can. The next question is which unsupported claims still survive, so those failures can become explicit rules, tests, and product improvements rather than being rediscovered in another project.

Disclosure Is Not the Same as Intervention

Disclosure, interpretation, and intervention are three different levels of review, and the reviewer reached the first two more reliably than the third. The reviewed report contained a much better deviations-and-caveats section. It recorded the MHRDEG mismatch, the unrecoverable logistic subset, LASSO sensitivity, the immune-signature substitution, unspecified preprocessing details, approximated or skipped analyses, and the boundary between computational reproduction and wet-lab validation.

That record matters. A future researcher can see where the current implementation differs from the paper without reconstructing every decision from 50 code cells. This is already a substantial improvement over a report that describes only what the workflow intended to do.

But scientific review has at least three levels:

Disclosure: record the mismatch or substitution.
Interpretation: change the evidence state and claim strength.
Intervention: redirect or block downstream work that no longer has the required support.

The reviewed run did the first level well and reached the second level in several places. It described CD55 as partially supported, separated paper and de novo tracks, and excluded wet-lab validation from the computational claim. It did not reliably reach the third level. The invalid independent-validation label survived, and the clinical-facing artifacts remained in the final package.

This distinction gives us a better way to evaluate scientific reviewers. A reviewer should receive credit for making uncertainty inspectable, more credit when that uncertainty changes the final interpretation, and the most credit when it prevents an unsupported branch from consuming more work and acquiring rhetorical weight.

What Does This Change for AIPOCH open-science?

For AIPOCH open-science, this test turns an architecture choice into demonstrated evidence. The current open-science v0.7.2 release includes an opt-in, record-scoped AI reviewer with a bounded fix loop. It is part of the workbench rather than a human review service: a separate reviewer role can inspect a turn's transcript, execution record, and artifacts independently from the acting agent and send bounded corrections back into the workflow. That architecture is already important because asking the acting agent to check its own work is not equivalent to giving the preserved run to a separate evaluation pass.

This test provides concrete evidence for the value of that design. The reviewed run became substantially closer to the paper, and the final scientific interpretation changed. Without an independent review layer, the first run's substitute cohort, substitute gene lists, altered threshold, and alternative biomarkers could easily have remained packaged as one coherent reproduction. The reviewer surfaced consequential corrections rather than merely polishing the final prose.

The current reviewer is record-scoped, and the next stage is to make it increasingly domain-aware. The v0.7.2 release notes explicitly state that domain-specific citation, units, statistics, and scientific rubrics are not yet built in. A general reviewer can already compare claims with files and flag discrepancies in a preserved run record. Adding persistent dataset roles, validation rules, and method-specific checks would let it recognize that a cohort used in feature selection cannot later be relabeled as independent validation, that an apparent calibration curve should not be presented as evidence of generalization, and that choosing preprocessing by closeness to a reported DEG count requires a separate sensitivity record.

That is the layer we are now testing and building around open-science. The goal is not to turn the workbench into one fixed bioinformatics pipeline, because different papers and domains require models to interpret incomplete methods, explore alternatives, and make judgment calls. The goal is to give the reviewer durable scientific objects against which those judgments can be checked: dataset roles, analysis-node contracts, original and executed methods, substitutions, assumptions, evidence states, downstream dependencies, and stop conditions.

If GSE179285 has already been used for feature discovery, its dataset role should persist. A later ROC node should not be able to call it independent validation simply because the file was reloaded. If the immune-analysis method changes from 28-cell ssGSEA to ten-population MCP-counter, the resulting artifact should inherit a substituted state. If a threshold changes, the workflow should preserve both the original branch and the sensitivity branch. If an upstream validation node becomes unsupported, the final claim should inherit that weaker state rather than recovering to reproduced in the prose.

We are continuing to test the reviewer across controlled reviewer-on and reviewer-off runs, different scientific domains, failure types, artifacts, and task lengths. We are looking at what it catches, what it misses, when it intervenes, whether its proposed fix changes the downstream analysis, how many unnecessary tokens it prevents, and whether a researcher can understand and act on the finding without reading the entire execution history.

The purpose is not to create a reviewer that blocks everything uncertain. Scientific work contains incomplete methods, imperfect data, and reasonable substitutions. The reviewer has to distinguish a fatal evidence problem from an acceptable limitation, a claim-changing substitution from an engineering repair, and an exploratory result from a failed result. That judgment is harder than generating a warning, but it is what makes review useful inside the workflow rather than decorative at the end.

Conclusion

The first run showed why a capable scientific agent needs an independent reviewer. The second showed that the reviewer can change more than the report. It corrected the cohort, restored the study's gene universe, recovered the published threshold, separated paper-based and de novo analyses, and changed the biomarkers supported by the final project. Those are not cosmetic improvements. They are the difference between efficiently completing an altered analysis and returning to the scientific question the paper claimed to ask.

The remaining validation error gives us a better product target rather than reversing that result. The reviewer proved its value because the reviewed run was materially better, and the unresolved issue tells us exactly where general record review needs to become domain-aware scientific review. For scientific agents, that is a useful standard. We should ask whether the system makes consequential mistakes easier to detect, correct, preserve, and learn from before they become finished-looking results. You can follow the work on the AIPOCH open-science product page and GitHub repository. The next step is giving the reviewer enough domain structure to make those corrections earlier, more consistently, and across more kinds of research.

FAQ

What does an AI reviewer do in a scientific reproduction run?

An AI reviewer in a scientific reproduction run is an independent model role that inspects the acting agent's transcript, execution record, and artifacts, then sends bounded corrections back into the workflow. In this test with AIPOCH open-science, it changed the cohort composition from 96 UC + 42 control to 97 UC + 34 control and restored the paper's abs(logFC) > 0.5 threshold, producing 2,306 DEGs within 1.3% of the published count.

Is an AI reviewer the same as journal peer review?

No. Journal peer review happens after submission by human experts, while the AI reviewer described here runs inside the workbench during the analysis, before a researcher reconstructs the run manually. It is designed to support researchers by surfacing consequential errors early, not to replace expert human review or provide validated scientific conclusions.

Can an AI reviewer change the scientific result, not just the writing?

Yes. In this run, upstream review changed which biomarkers the analysis supported: the reviewer-off run built a two-gene PPARGC1A + BNIP3 model, while the reviewer-on run identified CPT1A as the most stable signal and described CD55 as partially supported (AUC above 0.9 combined, 0.802 in GSE179285). This shifted the candidate-gene universe from 143/452 substitute genes to the paper's 1,680/2,360 genes.

What can this AI reviewer not yet do?

The reviewer does not yet enforce persistent dataset roles across the full analysis graph. In this run it corrected the composition of GSE179285 but still allowed it to be labeled independent validation after it had influenced feature discovery, leaving a reported ten-gene AUC of 1.000 on that dataset. The v0.7.2 release notes state that domain-specific citation, units, statistics, and scientific rubrics are not yet built in.

Where can I try AIPOCH open-science?

AIPOCH open-science is available as an open-source, model-agnostic desktop workbench. You can find it on the AIPOCH open-science product page and the open-science GitHub repository, with the opt-in AI reviewer included from the v0.7.2 release onward.

Disclaimer

This article is intended for informational purposes only and does not constitute medical advice, clinical guidance, diagnostic recommendations, treatment decisions, or validated scientific conclusions. Sample data, workflow outputs, and parameter values shown are illustrative and do not represent any validated research finding or real-world result.

AIPOCH open-science is a research workflow tool. It does not replace researcher judgment, and researchers remain fully responsible for evaluating the accuracy, completeness, and appropriateness of any outputs generated. The AI reviewer described here surfaces and organizes possible discrepancies in a preserved run record for researcher review; it does not guarantee correctness, and all outputs require independent verification and expert interpretation before use in any research context.

References and external links in this article are provided for informational purposes. AIPOCH does not endorse and is not responsible for the content of third-party sources.