Abstract

Post-training large language models (LLMs) on scientific review texts helps us evaluate them on generated review text along with paper ratings. Peer-review serves more than one purpose for academia and verification of the scholarly output is just one facet. As LLMs are integrated into scientific review workflows, AI-assisted pre-review will stand as the most consequential feature that will disrupt review dynamics and address bottlenecks around productivity for verifying the underlying scientific output. While we don't have systems that fully automate peer reviews yet, it is vitally important to study alignment and safety risks from raw model weights to understand bias amplification, unsupported yet persuasive critiques, and optimization for reviewer agreement rather than scientific correctness.

Motivation and empirical foundation

Science is accelerating faster than verification can keep up. As AI-accelerated research drives an exponential increase in submissions to venues like NeurIPS, ICLR, and ICML, the verification infrastructure has not changed. Verification is not just a search problem, an execution problem, or a generation problem — it is all three in a loop. You have to read the paper, extract the claims, run the code, compare results, and judge whether the conclusions hold. Exponential human labor is not the answer.

Fine-tuning is often evaluated on the same preference or score signal used to guide training. Scientific review makes the limitation concrete: agreement with reviewers is observable, but a useful critique must also identify supported problems, distinguish missing evidence from negative evidence, and accurately report what was checked. A model optimized for agreement may improve these behaviors, leave them unchanged, or weaken them. We aim to measure which outcome occurs and under what conditions.

Our earlier study, Pre-review to Peer review, examines open-weight LLM reviews against human ratings and post-publication signals, reporting systematic rating overestimation and high confidence despite prediction errors [1]. Our next experiments ask how post-training changes these behaviors and whether optimization for reviewer agreement introduces failures that transfer to other tasks.

Betley et al. show that narrow insecure-code fine-tuning can induce behavioral changes beyond coding, with important dependence on training context [2]. Çağatan and Zhao study divergence between observed reward and hidden safety objectives in language-agent environments [3]. Hadida et al. report cross-task transfer of reasoning obfuscation, including after supervision on final actions [4]. We extend these questions to an ordinary downstream task in which the supervision is useful but incomplete: learning to assess scientific work from peer reviews. The focus is behavioral evidence, with causal interventions on data, reward, and oversight; this is not a claim to identify an internal mechanistic circuit.

Reviewer workspace

“The interface shows a multi-phase review process where the agent first analyzes the manuscript PDF, executes provided code to verify claims, and generates structured feedback.”

Akella, Siravuri & Rohatgi (2025), Pre-review to Peer review, Appendix G. The preprint describes a mock interface; the example below illustrates that proposed workflow.

resci · review · submission-4821.pdfIllustrative example

Manuscript §4.2

We ablate the retrieval module and observe that removing it reduces accuracy by 4.1 points on the held-out split. The effect is consistent across seeds.

Our method outperforms all prior work on this benchmark, which we attribute to the proposed alignment objective.

All experiments use the released code and the configuration in Appendix B.

Execution sandbox

$ git clone …/repo && pip install -r requirements.txt$ python train.py --config appB.yaml --ablate retrievalacc 71.3 reported 72.9 Δ −1.6 vs paper$ resci cite --claim "outperforms all prior work"3 related results found · 1 not compared in paperenv hash 9f3a…c21 · config matches Appendix B

Telemetry signals

Noveltyretrieval thin
Generalizability1 dataset
Reproducibilityrerun differs

Inputs to human judgment. Never emitted as a score without linked evidence.

Review 3 notes · linked

ReproducibilityAblation rerun with the Appendix B config gives 71.3 vs reported 72.9. Direction holds; magnitude differs. Ask authors for seeds.
↳ cmd train.py · out diff · env 9f3a…c21
Novelty“All prior work” is not supported: one related result is not compared. Weak claim, not a rejection.
↳ retrieval 3 refs
ArtifactsReleased code and config match Appendix B. Runnable is not the same as correct.
↳ env hash match

Ablation result: linked rerun output and reproducibility note highlighted.

Background

Peer review determines which research articles the scientific community treats as sound, novel, and significant, and it places a heavy burden on human reviewers. Researchers already send manuscripts to commercial LLMs such as ChatGPT, Claude, and Gemini for informal pre-reviews, yet we know little about how reliable those judgments are. As we discuss in [1], prior work on automated review has largely focused on building review corpora or predicting accept/reject decisions. Comparatively less attention has been paid to two questions: whether LLM reviews align with human reviewers, including whether a model's stated confidence tracks its error, and whether LLM scores relate to what happens to a paper after publication.

In Pre-review to Peer review [1], we examined both questions using DLMRSD, a dataset of 77,845 OpenReview reviews covering 28,033 papers from venues including ICLR, NeurIPS, CoRL, and MIDL. We linked each paper to SciSciNet [8] to attach five-year citation counts (C5), top-1%, 5%, and 10% hit-paper status, novelty and conventionality scores, and the disruption index. We then ran two zero-shot experiments. In the first, five dense models rated each paper's idea from its title, keywords, and abstract. In the second, the same five models and four reasoning models (including gpt-oss-120b, Qwen3-Next-80B-A3B-Thinking, and DeepSeek-R1-Distill-Llama-70B) reviewed the full manuscript.

We observed weak agreement between LLM and human scores. The best correlation in the abstract-only setting was ρ = 0.153 (Gemma-3-27B-IT), which explains less than 2.5% of the variance in human ratings. With full text, mean absolute error (MAE) ranged from 2.13 (gpt-oss-120b) to 3.38 (Llama-3.3-70B-Instruct) on a 10-point scale, and reasoning models did not consistently outperform dense models. The models inflated their scores: mean predictions fell between 7.56 and 8.98, while human medians fell between 3 and 7. Error was largest on weak papers. For papers that humans rated 0–3, MAE was roughly 4.5–6.2, compared with 0.55–1.02 for papers rated 7–10. In a screening setting, this pattern would let weak submissions pass.

The models were also poorly calibrated. Human reviewers in DLMRSD report lower confidence at the extremes of the rating scale. By contrast, once the LLMs saw the full text, their self-reported confidence clustered at 0.7–1.0 regardless of how far their scores deviated from human consensus; full-text access raised confidence more than it reduced error. Two reasoning models, gpt-oss-120b and Qwen3-Next-80B-A3B-Thinking, generated reviews almost deterministically (median perplexity 1.24 and 1.35), and their reviews reached a cosine similarity of about 0.88 with human reviews. These results suggest that the models have learned the form of a peer review, but that fluency does not carry over to accurate scores.

The relationship with post-publication outcomes was more mixed. Scores from gpt-oss-120b correlated with C5 at ρ = 0.21, compared with ρ = 0.14 for human scores, and the model identified top-1% hit papers slightly better than human reviewers (AUC 0.600 vs. 0.573). However, both humans and LLMs scored disruptive work lower, and the tendency was stronger for LLMs (ρ = -0.21 for gpt-oss-120b vs. -0.09 for humans). LLM scores also correlated positively with conventionality (ρ ≈ 0.05–0.15), whereas human scores correlated negatively (ρ = -0.13). These patterns suggest that LLMs may respond to textual signals associated with citation success, such as clear exposition, broad literature coverage, or alignment with active research topics, more than to the novelty of the contribution. Deployed without oversight, such models could reinforce the preference for conventional research documented in human peer review [12, 13].

Prompting only partly corrected these problems. Adding explicit scoring-calibration instructions and raising reasoning effort reduced MAE from 1.88 to 1.13 for gpt-oss-120b and from 2.78 to 1.85 for Qwen3-Next-80B-A3B-Thinking. The largest gains came on low-rated papers, but predicted scores stayed compressed toward the top of the scale.

Taken together, these findings suggest that current open-weight LLMs can serve as pre-review screening aids under human oversight but are not reliable as autonomous reviewers. All of these measurements are zero-shot. Score inflation, confidence that does not track error, and a stronger preference than human reviewers for conventional work are the baseline behaviors that post-training on reviewer agreement could correct or entrench.

Research Question

In this work, we examine how training on peer reviews changes a model's scientific judgment. Reviewer agreement is an ordinary training target: the reviews are real and agreeing with them is useful, but agreement cannot confirm that a critique is correct. The study asks three questions. Does supervised training on reviews change how a model handles evidence on tasks outside peer review, and how does any change depend on training tokens and model size? When reinforcement learning (RL) optimizes reviewer agreement, does reward separate from independently verified correctness as optimization pressure and model size increase, and can early signals predict the separation? When training penalizes a monitor's flags, do the underlying errors decrease, or only the flags? A fourth analysis asks which published papers a trained model would recommend, a consequence that agreement scores do not capture. Each comparison changes one element of training (the supervised objective, the reward, or the oversight penalty) and holds the base model, training manuscripts, and seeds fixed.

Downstream scientific selection

Reviewer-agreement scores do not show which science a trained model would favor. In our zero-shot study, LLM scores favored conventional work more than human scores did, and training on agreement could strengthen that preference even as agreement improves. We will measure this by linking model judgments to citation counts (C3 and C5), disruption, and the novelty and conventionality of each paper's cited references, using SciSciNet-v2 [7, 8] (Figures 6–10). These measures serve only as evaluation outcomes and never enter a training reward.

We will first audit training exposure for each paper in the current 295-paper cohort and exclude any paper used in SFT, RL, or prompt development. We will then seek a locked cohort of at least 1,000 linked papers across two or more venue-years, subject to manuscript access and verified observation windows, and publish linkage and exclusion rates by venue and year, comparing matched with unmatched papers on available metadata. If fewer papers qualify, we will report the eligible cohort with wider uncertainty rather than fill missing outcomes with zeros. Pretraining exposure to historical papers remains unknown; the newly collected evidence-checking cases provide a complementary test.

Every model receives the same manuscript-only input under two protocols. The first elicits a review and a rating. The second predicts fixed ordinal labels for each outcome, from the manuscript alone or from the manuscript plus the model's own review. Label thresholds are set on development papers, preserve ties, and stay fixed across checkpoints. Candidate-label likelihoods or constrained decoding prevent formatting failures from changing which papers enter the evaluation. The human-review-conditioned scores in Figures 2 and 3 remain a separate diagnostic.

Across checkpoints, we will compare Kendall τb, rating error, ordinal macro-F1, and the distribution of predicted scores. In a simulated screening policy, each checkpoint selects the top 20% of eligible papers, with ties broken by a fixed paper-ID rule. We then measure recall of papers in the top citation decile and shifts in the selected papers' disruption, tail novelty, and conventionality. Comparisons within venue-year and topic, and between papers with similar human ratings, test whether gains in reviewer agreement coincide with a stronger preference for conventional research. Lower tail z-scores indicate greater novelty and higher median z-scores greater conventionality [9, 8]. Disruption enters as a continuous rank because only three papers in the current cohort have positive values.

This analysis estimates how training changes a model's recommendations on a fixed historical set. The papers' outcomes already exist, so it cannot estimate a causal effect on future citations or on the scientific system. A model can favor highly cited work while overlooking unconventional work, and neither citation counts nor disruption establishes that a paper is correct; the evidence checks measure correctness directly.

Preliminary results

Building on the pilots below, we plan to compare review training, review training with an added penalty for rating errors [5], and training on summaries of the same manuscripts. Training on summaries helps isolate the effect of learning to judge research from the effects of scientific content and further training. We will test whether review training changes how models check evidence and respond to authority outside peer review.

For reinforcement learning, we will compare rewards for matching human reviews with rewards for independently verified correctness, and test whether penalties for flagged errors reduce mistakes or make them harder to detect. We will identify reward hacking only when reward rises, independently checked performance declines, and an audit identifies the behavior responsible for the gain.

Qwen3.5-2B: expected-rating Kendall tau-b versus estimated training FLOPs for SFT and TRACT-style runs at three token budgets.
Figure 2. Qwen3.5-2B trained with SFT or the TRACT-style objective at 5M, 10M, and 20M tokens (marker size); dotted lines connect runs of one method. Expected-rating Kendall τb compares the probability-weighted rating over candidate labels with the human rating on 909 held-out review rows from 241 ICLR 2025 papers. The model sees the manuscript and the human review, so this protocol measures how well it maps a review to a rating. The horizontal axis estimates dense-equivalent training FLOPs as 6ND. At 5M tokens, the TRACT-style model predicted a rating of 5 for every row.
Gemma-4-E2B-it: expected-rating Kendall tau-b versus estimated training FLOPs for SFT and TRACT-style runs at three token budgets.
Figure 3. Gemma-4-E2B-it under the same design and human-review-conditioned protocol, evaluated on 969 held-out review rows from 258 ICLR 2025 papers; tokenizer-dependent length filtering yields a different cohort from Qwen's. Methods should be compared within each model, since the two plots use different vertical ranges. At 20M tokens, the TRACT-style model predicted 5 for every row while its expected-rating τb (0.517) exceeded that of SFT (0.509).
Four panels compare held-out Kendall tau-b, valid completions, review similarity, and rating error before and after the Qwen3.5-4B RL pilot.
Figure 4. Held-out evaluation of the Qwen3.5-4B RL pilot before and after seven updates with the agreement reward. Both test passes use the same 769 papers and sampling seeds; the validation pass uses 778 different papers and has no before-training counterpart. Error bars are 95% intervals from 2,000 paper-level bootstrap resamples within each pass. Kendall τb compares per-paper mean valid model ratings with mean human ratings. Validity counts all completions; cosine similarity and rating MAE use valid completions only.
Human, baseline-model, and post-RL rating distributions for 769 test papers; model ratings remain concentrated between 7 and 9.
Figure 5. Rating distributions on the 769 test papers: 2,973 human reviewer ratings, 2,639 valid baseline completions, and 2,680 valid post-RL completions. Invalid outputs are excluded. Each series is normalized separately, and dashed lines mark its pooled mean (5.45 for humans; 7.70 before and 7.73 after RL). ICLR reviewers used a 1–10 scale in 2021 and {1, 3, 5, 6, 8, 10} in 2022 and 2023. These are rating-level distributions, whereas the τb panel in Figure 4 aggregates by paper.

Post-publication dataset and downstream evaluation

Our linked dataset, Dpost-pub, currently contains 295 unique ICLR 2021 papers, each matched to a SciSciNet paper ID. Figures 6–10 show the recorded outcomes that the downstream selection analysis will use. They describe the papers themselves; no SFT or RL model has yet been evaluated against them. SciSciNet-v2 provides the infrastructure for extending the cohort [7], and the metric definitions follow SciSciNet and the original studies of atypical combinations and disruption [8, 9, 10, 11].

The cohort is small and selectively linked. It contains 255 negative, 37 zero, and 3 positive disruption values, and 156 papers have negative tail-novelty z-scores. Both citation distributions have a median of 5 and a maximum of 858. The source documentation assumes a 2024 cutoff, under which every C3 window is flagged as expected complete and every C5 window as incomplete; C3 = C5 for 292 papers. We will verify the snapshot and observation horizons before any confirmatory comparison. These properties shape the plan: expand the cohort, analyze disruption as a continuous rank, and test directly whether training favors conventional work.

Disruption distribution for 295 papers: 255 negative, 37 zero, and 3 positive values.
Figure 6. Disruption in the 295-paper cohort, on a symmetric-log axis. Negative values indicate consolidating work and positive values disruptive work, as measured in the subsequent citation network [10, 11, 8]; the median is -8.05×10-5. Zeros are shown as stored and do not establish neutral impact. Only three values are positive, which limits sign-based comparisons.
Distribution of stored three-year citation counts for 295 papers, with a median of 5 and maximum of 858.
Figure 7. Stored three-year citation counts, C3 [8], with one entry per paper regardless of its number of reviews. Thirty of the 295 papers have zero recorded citations. The source flags every C3 window as expected complete under an assumed cutoff, which we will check against the source snapshot before outcome evaluation.
Distribution of stored five-year citation counts; all observation windows are incomplete.
Figure 8. Stored five-year citation counts, C5 [8]. Every five-year window is incomplete, so C5 equals C3 for 292 papers; this reflects the coverage of the source snapshot and says nothing about later citations. We show these values for the dataset audit and exclude them from confirmatory five-year analyses until the windows are verified complete.
Empirical cumulative distribution of tail-novelty scores, with 156 of 295 papers below zero.
Figure 9. Paper-level cumulative distribution of Atyp_10pct_Z, the 10th-percentile journal-pair z-score of each paper's references [9, 8]. Lower values indicate greater tail novelty; 156 papers (52.9%) fall below zero. The signed base-2 axis preserves the original scores, shading marks their sign, and dots mark empirical quantiles.
Empirical cumulative distribution of conventionality scores, with a cohort median of 80.97.
Figure 10. Paper-level cumulative distribution of Atyp_Median_Z, the median journal-pair z-score [9, 8]. Higher values indicate more conventional combinations of cited knowledge; the cohort median is 80.97. Because a paper can pair a conventional core with a few unusual combinations, we analyze conventionality and tail novelty separately.

References

  1. Akhil Pandey Akella, Harish Varma Siravuri, and Shaurya Rohatgi. Pre-review to Peer review: Pitfalls of Automating Reviews using Large Language Models. 2025. arXiv:2512.22145.
  2. Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, and Owain Evans. Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs. 2025; arXiv version 7, 2026. arXiv:2502.17424.
  3. Ömer Veysel Çağatan and Xuandong Zhao. Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds. 2026. arXiv:2606.15385.
  4. Nathaniel Mitrani Hadida, Sassan Bhanji, Cameron Tice, and Puria Radmard. Chain-of-thought obfuscation learned from output supervision can generalise to unseen tasks. 2026. arXiv:2601.23086.
  5. Cheng-Han Chiang, Hung-yi Lee, and Michal Lukasik. TRACT: Regression-Aware Fine-tuning Meets Chain-of-Thought Reasoning for LLM-as-a-Judge. ACL, 2025. arXiv:2503.04381.
  6. Thinking Machines Lab. Tinker: Models & Pricing. Accessed September 25, 2026. Official model catalog and token pricing.
  7. Akhil Pandey Akella, Zihang Lin, Yifan Qian, and Dashun Wang. SciSciNet-v2: A refreshed update to SciSciNet, a large-scale, integrated dataset designed to support research in the science of science domain. 2025. Dataset website. DOI: 10.57967/hf/5692.
  8. Zihang Lin, Yian Yin, Lu Liu, and Dashun Wang. SciSciNet: A large-scale open data lake for the science of science research. Scientific Data 10, 315, 2023. doi:10.1038/s41597-023-02198-9.
  9. Brian Uzzi, Satyam Mukherjee, Michael Stringer, and Benjamin Jones. Atypical Combinations and Scientific Impact. Science 342(6157), 468–472, 2013. doi:10.1126/science.1240474.
  10. Russell J. Funk and Jason Owen-Smith. A Dynamic Network Measure of Technological Change. Management Science 63(3), 791–817, 2017. doi:10.1287/mnsc.2015.2366.
  11. Lingfei Wu, Dashun Wang, and James A. Evans. Large teams develop and small teams disrupt science and technology. Nature 566, 378–382, 2019. doi:10.1038/s41586-019-0941-9.
  12. Carole J. Lee, Cassidy R. Sugimoto, Guo Zhang, and Blaise Cronin. Bias in peer review. Journal of the American Society for Information Science and Technology 64(1), 2–17, 2013. doi:10.1002/asi.22784.
  13. Kevin J. Boudreau, Eva C. Guinan, Karim R. Lakhani, and Christoph Riedl. Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance, Novelty, and Resource Allocation in Science. Management Science 62(10), 2765–2783, 2016. doi:10.1287/mnsc.2015.2285.

Cite this article

@misc{akella2026generalizationandrewardhacking,
  author = {Akella, Akhil Pandey},
  title = {{Narrowly fine-tuning models to act as scientific reviewers}},
  year = {2026},
  howpublished = {Resci},
  note = {September 25, 2026},
  url = {https://resci.co/blog/generalization-and-reward-hacking}
}
← All research