Assassins and zealots: Variations in peer review: Special report

Stanley S. Siegelman

Research output: Contribution to journalArticle

Abstract

Radiology reviewers are required to assign numerical grades of 1 to 9 (1 = accept, 9 = reject) in the rating of manuscripts. The mean ratings for the 660 referees who were assigned 10 or more reviews over a 41/2-year period were analyzed. The mean score was 4.8 ± 0.8, and 87.4% of reviewers (the mainstream) had ratings of mean ± 1.5 standard deviations. Categories of reviewers with greater deviation from the mean were identified: zealots and pushovers, whose ratings of manuscripts were more favorable, versus assassins and demoters, who supplied less favorable ratings. To exclude the possibility that the referees who were classified as more critical had actually been sent less meritorious papers, the scores and rejection rates of 859 papers co-reviewed by assassins, demoters, and mainstream referees were compared. Significant differences were confirmed. Deviant referees were widely distributed in the pool of reviewers, including 13 members of the Editorial Board and representatives in each of 19 subspecialty areas. Failure to recognize and control for reviewer variation may be unfair to authors. An Editor has the capacity to reduce unfairness by monitoring reviewer variation and by modulating the review process accordingly.

Original languageEnglish (US)
Pages (from-to)637-642
Number of pages6
JournalRadiology
Volume178
Issue number3
StatePublished - Mar 1991

Fingerprint

Peer Review
Manuscripts
Radiology
Rejection (Psychology)

Keywords

  • Radiology (journal)
  • Radiology and radiologists, research

ASJC Scopus subject areas

  • Radiological and Ultrasound Technology

Cite this

Assassins and zealots : Variations in peer review: Special report. / Siegelman, Stanley S.

In: Radiology, Vol. 178, No. 3, 03.1991, p. 637-642.

Research output: Contribution to journalArticle

Siegelman, SS 1991, 'Assassins and zealots: Variations in peer review: Special report', Radiology, vol. 178, no. 3, pp. 637-642.
Siegelman, Stanley S. / Assassins and zealots : Variations in peer review: Special report. In: Radiology. 1991 ; Vol. 178, No. 3. pp. 637-642.
@article{b890d1e18c254c33a339d01ae1ad4c99,
title = "Assassins and zealots: Variations in peer review: Special report",
abstract = "Radiology reviewers are required to assign numerical grades of 1 to 9 (1 = accept, 9 = reject) in the rating of manuscripts. The mean ratings for the 660 referees who were assigned 10 or more reviews over a 41/2-year period were analyzed. The mean score was 4.8 ± 0.8, and 87.4{\%} of reviewers (the mainstream) had ratings of mean ± 1.5 standard deviations. Categories of reviewers with greater deviation from the mean were identified: zealots and pushovers, whose ratings of manuscripts were more favorable, versus assassins and demoters, who supplied less favorable ratings. To exclude the possibility that the referees who were classified as more critical had actually been sent less meritorious papers, the scores and rejection rates of 859 papers co-reviewed by assassins, demoters, and mainstream referees were compared. Significant differences were confirmed. Deviant referees were widely distributed in the pool of reviewers, including 13 members of the Editorial Board and representatives in each of 19 subspecialty areas. Failure to recognize and control for reviewer variation may be unfair to authors. An Editor has the capacity to reduce unfairness by monitoring reviewer variation and by modulating the review process accordingly.",
keywords = "Radiology (journal), Radiology and radiologists, research",
author = "Siegelman, {Stanley S.}",
year = "1991",
month = "3",
language = "English (US)",
volume = "178",
pages = "637--642",
journal = "Radiology",
issn = "0033-8419",
publisher = "Radiological Society of North America Inc.",
number = "3",

}

TY - JOUR

T1 - Assassins and zealots

T2 - Variations in peer review: Special report

AU - Siegelman, Stanley S.

PY - 1991/3

Y1 - 1991/3

N2 - Radiology reviewers are required to assign numerical grades of 1 to 9 (1 = accept, 9 = reject) in the rating of manuscripts. The mean ratings for the 660 referees who were assigned 10 or more reviews over a 41/2-year period were analyzed. The mean score was 4.8 ± 0.8, and 87.4% of reviewers (the mainstream) had ratings of mean ± 1.5 standard deviations. Categories of reviewers with greater deviation from the mean were identified: zealots and pushovers, whose ratings of manuscripts were more favorable, versus assassins and demoters, who supplied less favorable ratings. To exclude the possibility that the referees who were classified as more critical had actually been sent less meritorious papers, the scores and rejection rates of 859 papers co-reviewed by assassins, demoters, and mainstream referees were compared. Significant differences were confirmed. Deviant referees were widely distributed in the pool of reviewers, including 13 members of the Editorial Board and representatives in each of 19 subspecialty areas. Failure to recognize and control for reviewer variation may be unfair to authors. An Editor has the capacity to reduce unfairness by monitoring reviewer variation and by modulating the review process accordingly.

AB - Radiology reviewers are required to assign numerical grades of 1 to 9 (1 = accept, 9 = reject) in the rating of manuscripts. The mean ratings for the 660 referees who were assigned 10 or more reviews over a 41/2-year period were analyzed. The mean score was 4.8 ± 0.8, and 87.4% of reviewers (the mainstream) had ratings of mean ± 1.5 standard deviations. Categories of reviewers with greater deviation from the mean were identified: zealots and pushovers, whose ratings of manuscripts were more favorable, versus assassins and demoters, who supplied less favorable ratings. To exclude the possibility that the referees who were classified as more critical had actually been sent less meritorious papers, the scores and rejection rates of 859 papers co-reviewed by assassins, demoters, and mainstream referees were compared. Significant differences were confirmed. Deviant referees were widely distributed in the pool of reviewers, including 13 members of the Editorial Board and representatives in each of 19 subspecialty areas. Failure to recognize and control for reviewer variation may be unfair to authors. An Editor has the capacity to reduce unfairness by monitoring reviewer variation and by modulating the review process accordingly.

KW - Radiology (journal)

KW - Radiology and radiologists, research

UR - http://www.scopus.com/inward/record.url?scp=0025965169&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0025965169&partnerID=8YFLogxK

M3 - Article

C2 - 1994394

AN - SCOPUS:0025965169

VL - 178

SP - 637

EP - 642

JO - Radiology

JF - Radiology

SN - 0033-8419

IS - 3

ER -