Rules for an agent-based peer review model

Last week I wrote about a set of AGU EOS articles from 2003 that focus on anonymity in peer review. A quote from one of the articles really stuck with me regarding the personal decision to sign reviews:

Okal (2003) states that, as an editor of GRL, ~40% of the reviews he sees are signed. As a reviewer, he signs 2/3 of his reviews. And as an author, 1/2 the reviews he receives are signed. His experience suggest that:
The above numbers — 40%;two-thirds; one- half — suggest that the community is divided, with no overwhelming majority in its attitude toward anonymous versus signed reviews. This diversity may indeed be precious and should be respected. Why not keep the system as it is now, leaving it to the individual reviewer to exercise a free decision regarding waiving anonymity?”

Over the course of the next few weeks I hope to build a fun little toy model of ‘peer reviewing’ agents to see if I can tease out something  — is diversity in peer review behavior (re: signed vs blind) in some way ‘precious’?

the rules of the model are:

Each agent (scientist) is set to either sign or blind their reviews.

For each time step:

  • Randomly pick the number of scientists (‘P’) out of ‘N’ total scientists who will publish a single paper
  • Randomly assign ‘R’ reviewers for each paper
    • Nobody can review their own paper
    • Writing Sceintists can review
    • Scientist can do multiple reviews
  • Each reviewer gives a random review score (good or bad)
  • Reviews are returned to each writer and writers ‘mood’ changes
    • signed + reviews result in + feelings toward the reviewer
    • signed – reviews result in – feelings toward the reviewer
    • unsigned + reviews result in + feelings toward a random scientist
    • unsigned – reviews result in – feelings toward a random scientist

And we see how the feelings of the community (toward one another) develop through time.

The beginning of the code is already up on Github. Feel free to contribute or give an opinion.

The AGU EOS ‘Anonymous Peer Review’ debate of 2003-2004

This summer I stumbled upon a cache of EOS newsletters from 2003. Among the pages was a series of comments and letters about anonymous review, specifically problems and possible solutions. It’s nice to know that we struggle with the same issues 14 years later.

The original article written on July 1, 2003 by Beck (2003) was focused on the rejection of a paper by 2 anonymous reviews and an anonymous AE. After listing and discussing potential reasons that a reviewer and/or AE would prefer to remain anonymous. Beck ends by writing:

“The only reviews I remember that left me perma­nently angry were anonymous. There is far too much unpleasantness in the world already to needlessly introduce even a little bit more. Anonymous reviews are unnecessary, unacceptable, and should not be permitted.” 

Strong statement! I have my own opinions about anonymity in peer review (I’m sure everyone does), but what is most interesting to me is that fact that this article produced such a large reaction — I can find 15 letters and comments published in EOS as a response to Beck (2003) — compared to the rare comment-reply pairs in JGR-ES.

On July 29th, 2003

  • Roninove (2003) writes in to support Beck (2003), having written a letter about problems with anonymous reviews (back in 1990).
  • Criss and Hofmeister (2003) suggest discounting anonymous reviews, and discuss the issues surrounding signed vs unsigned reviews for underrepresented groups.

On Sept 23rd, 2003

  • Geller (2003) writes in to suggest that AEs should always sign reviews because they often make the decision for the editor.
  • Goff (2003) writes in to suggest AEs should sign reviews and that AGU should encourage signed reviews and newer journals should require signed reviews
  • Walder (2003) writes in to suggest that AGU AEs should sign reviews and we should collect data — reviewers be asked ‘why’ they choose to remain anonymous.

Sept 30th, 2003

  • Forel (2003) is an ‘advocate’ for anonymous reviewing, but believes editors should not be anonymous.
  • Fisher (2003) writes in to suggest double blind reviewing
  • Savov (2003) writes that science should be “…discussed in the open air.” and suggests that the paper, reviews, and reviewer names should all be published together.
  • Okal (2003) writes that the current system should be preserved and personal preference (re: signed vs unsigned reviews) should be respected. Okal writes that this debate has been going on for decades with no clear solution:

“The debate on peer review has been going on for decades. It may be the worst possible system, but by and large it works. And to paraphrase Sir Winston Churchill, wait until you consider all the other ones….”

Dec 23rd, 2003

  • The editors of JGR-Atmopsheres respond in O’Dowd et al. (2003). They discuss the editorial process in the journal and highlight the role of anonymity for the AEs and reviewers.

Dec 30th, 2003

  • Kirwan Jr. (2003) writes that peer reviews should not be signed because it could be self serving. Furthermore authors should not speculate about the authors of their anonymous reviews because of possible negative and counterproductive consequences.
  • Wesolowski (2003) writes that finding reviewers is difficult enough without requiring the identification of reviewers, and forced signing of reviews may lead to overly positive reviews.

April 20th, 2004

  • Armstrong (2004) discusses the possibility of multiple review stages, some with or without anonymity.  
  • Sturrock (2004) presents a ‘Code of Ethics’ for peer review.

April 27th, 2004

  • Genereaux and Sen (2004) discuss the NSF proposal review process, specifically how proposers do not have an opportunity to respond to “Incorrect and Overly Negative Statements (IONS)”.

 

N.B. — There was an article on anonymous peer review in GSA Today by McBirney (2003) — here is a link to the issue — something must have been in the air. 

Age distribution of JGR-ES articles cited in a given year

In a previous post I looked at yearly cohorts of JGR-ES papers, and the mean number of citations that papers from each yearly cohort accrues through time as the papers age. In this post I will look at something slightly different — what is the age distribution of cited JGR-ES articles in a given year? Put another way, if we had all the citations that JGR-ES papers racked up in a single year, say 2007, what fraction of are citations to JGR-ES papers published in 2003? or 2004? or 2005? The idea for this plot came from Clark and Hanson (2017).

CHplot.jpg

Each line on this plot represents all citations to JGR-ES articles in a given year. For instance, the dark blue curve represents all citations to JGR-ES articles in the year 2010. The big spike in the dark blue curve means that, in 2010, 27% of all citations to JGR-ES articles were specifically focused on JGR-ES articles from 3 years prior (i.e., 2007). Notice that the orange curve (2011 citations) has a peak at 4 years prior (i.e., 2007 as well). As we march forward in time, that peak in citations moves back, and always represents the 2007 cohort of JGR-ES papers.

So what happened in 2007? Looking further in the citation data, there is a rotating group of ~10 papers from 2007 that are all highly cited. In additional a single paper (Schoof, 2007) is highly cited every year after its publication. This group of papers from 2007 accounted for 10% – 30% of all citations that JGR-ES received from 2010-2016.

Share of JGR-ES paper citations received after 2 and 5 years

Last week I looked at citation histories for JGR-ES articles — most articles tend to accrue citations steadily through time after publication, and mean citations/year do not tend to show a strong decrease with time.  So how does this relate to Journal Impact Factor (JIF), which is calculated by dividing the total number of citations received by a journal (for all articles) in the 2 previous years, by the total number of articles published in the previous two years. There is also a five year variant of the JIF. a time series of JIF for JGR-ES and other geomorphology journals can be seen here.

Specifically, what share of total citations to a JGR-ES paper are received within the first 2 years after publication; how about after the first 5 years? After downloading the JGR-ES data from the Web of Science, I grouped papers by year and investigated these yearly cohorts in the box and whisker plots below.

2yearfrac.jpg

 

The oldest JGR-ES articles (2003-2006) recieved between 0 – 40% of their total citations in the first 2 years of their life — the JIF time window.  The newest articles (2008-2010) receive a larger share of their total citations, but keep in mind that this share will continue to shrink through time as the paper accumulates more citations.

 

5yearfrac.jpg

The five year fraction is larger, and seems to represent 10-80% of total citations for the oldest JGR-ES papers (2003-2006).

These plots demonstrate some of the complications with JIFs for journals where citations might accrue more slowly that a 2 and 5 year window.

JGR-ES citation histories

This post is focused on citation histories for JGR-ES papers — how a paper and groups of papers accumulate citations through time. JGR-ES started in 2003, and in a previous post I outlined the growth in number of papers published per year.

From the Web of Science I downloaded all of the data for all JGR papers and their respective citations. I grouped papers by year and investigated yearly groups/cohorts in the plots below.

I’ll call this first figure a set of cumulative citation plots — the x-axis is the year since publication, and the y-axis is fraction of total citations for each paper. Each panel represents a specific cohort of JGR-ES papers (all of the papers published in a given year), and each black line represents a single paper as it accumulates citations through time from 0 to 1 (i.e., no citations to its own maximum number of citations).

cumulative.jpg

Keep in mind that each x-axis is a different length (depending on the age of the cohort). Notice how each paper tends to accrue citations in a generally smooth manner — a notable outlier is the curve in 2006, where a paper waits 7 years before its first citation(s).

We can also look at the citation history of these yearly cohorts of papers in a different way— given group of papers published in a single year, what is the mean number of citations/paper in a given year as a function of paper age. Below is a plot for mean citations/year as a function of paper age for 8 cohorts —(all papers from JGR-ES published each year from 2003 to 2010). (‘mean’ might not be the best metric, but let’s stick with it for now, without ‘min’ and ‘max bounds).

meanhistory.jpg

One key aspect of each curve is that most tend to trend upward even after the 2 and 5 year windows — keep in mind that these are the time periods used for calculating Journal Impact Factor (and 5 year Journal Impact Factor). In fact the mean citations/year do not show strong downward trends, suggesting that papers from JGR-ES tend to have a long ‘life’.

One other note — cohorts from more recent times (2007 onward) tend to have larger mean citations/year. I imagine this is related to the general growth in JIF through time for geomorphology journals.

Some context for future posts: Journal Impact Factor for Geomorphology journals

For the next few posts I will be digging into the citation data for JGR-ES. I want to look at how the papers from JGR-ES accumulates citations — from which journals, over what time frame, when citations peaks occur, etc.

My motivation is to look at how actual papers accrue citations relative to the 2 and 5 year windows over which the ‘Journal Impact Factor’ (JIF) is calculated. Therefore I want to show the history of ‘normal’ (2 year) and 5 year JIF for JGR-ES and the other three geomorphology journals that I have focused on previously.  All data was downloaded from the Journal Citation Report.

From the plots below, my general take away is that these journals are all very similar to each other, mirroring their similarity in citation distribution — see the 2014, 2015 and 2016 distributions.

JIF.jpg

5yr JIF.jpg

2 notes:

Citation distributions for Geomorphology journals (2016 impact factor)

The 2017 Journal Citation report was released last month, which includes the new journal impact factor calculations for 2016.

A journal impact factor is calculated by dividing the total number of citations received (for all articles) in a 2 year period by the total number of articles published in the previous two years. It is a poor metric for individual papers. To see the full range of citations garnered by the papers published in the previous two years, it is useful to see citation distributions (the number of papers with a given number of citations published in a single journal), a useful visulaization that I became aware of from a paper by Larivière et al. 2016.

Here is a link to my post last year with distributions re: the 2014 and 2015 Impact factor calculation. Below I recalculate the citation distributions for geomorphology journals with data for the 2016 impact factor calculation (downloaded from the Web of Science). I also report the median of these distributions.

2016 JIF.jpg

Just to clarify further — this plot reports the number of papers with a given number of citations for each journal.

Similar to the last 2 years, the distributions are skewed and similar to each other. Also similar to the past 2 years, structure-from-motion papers are often in the long tail (far to the right; highlighting the broad interest of the technique).

  • A recent blog post by AGU discussed the adoption of alternative journal metrics that the organization will soon report.
  • A recent editorial in WRR also discusses citation metrics in regards to the hydrology community.