All of the reviews for Earth Surface Dynamics are open, published, and citable. Today I do a bit of webscraping to determine the % of mix of signed and blind reviews for the 198 paper reviewed in EsurfD. Also, since reviews occur in sequence (i.e., R1 submits their review before R2), we can exame how R1’s decision to sign a review influences the decision of R2.
The code to do the webscraping is here. Note that R is not my best language, but I am using it because of all the cool packages written for R to interface with Crossref (rcrossref, for obtaining publication DOIs), and the easy webscraping (rvest).
The code works by:
- Pulling (from Crossref) details for all ESurf Discussion publications using the ISSN number.
- Going to every EsurfD page (following the DOI link)
- Scraping the webpage for author, editor, and reviewer comment (see this helpful tutorial on using rvest).
- Checking for descriptive words, for instance “Anonymous Comment #1”, to determine if Reviewer 1 and/or Reviewer 2 were anonymous.
- Check to see if a Reviewer 3 exists (to exclude the data… I only want to deal with papers with 2 reviewers for this initial study).
I imagine some specific pathological cases in review comments may have slipped through this code, but a cursory check shows it captures relevant information. After the code runs, I am left with 135 papers with 2 reviewers, for a total of 270 reviews. In total, 41% reviews are signed — this matches previous reports such as 40% reported by Okal (2003) and the 40% reported by PeerJ
- Reviewer 1 totals are 74 unsigned, 61 signed —55% unsigned, 45% signed
- For the 74 papers where Review 1 is unsigned,
- Reviewer 2 data is 59 unsigned, 15 signed — 80% unsigned, 20% signed
- For the 61 papers where Review 1 is signed,
- Reviewer 2 data is 27 unsigned, 34 signed — 44% unsigned, 54% signed.
There is one clear confounding factor here, which is how positive/negative reviews impact the likelyhood to sign a review (both for R1 and R2). I imagine referee suggestions to the editor (e.g., minor revisions, major revisions, reject) and/or text mining could provide some details. (I can think of a few other confounds beyond this one)…. Furthermore, I would assume that since many (all?) journals from Copernicus/EGU have open review, this analysis could be scaled…