Incentivizing Peer Reviewers

It?really annoys me! I review papers for scholarly journals and, if I agree to undertake the task, have never taken more than six weeks to get the job done.? But sometimes my own papers are held by scholarly journals for a year, as the journal waits on reviews by one or more delinquent reviewers.? David?Figlio of Northwestern University has proposed a clever?matching mechanism that seems to offer the right incentives on both sides of the market:? The editor of scholarly journals can base the author/reviewer match in part on the author’s reviewing speed when s/he has reviewed.? Obviously, topic-specific knowledge would remain the major match criterion; but matching along reviewing speed can within some range alter that match.?Publicize this policy widely.? Then reviewers who want to have their own research treated expeditiously will have an incentive to do the job more quickly than otherwise; and authors will know that their treatment will be better if they behave better.


Frederik Anseel

Brilliant! I'm on the editorial board of two journals and this policy would be pretty motivating both to authors and reviewers. Seems like a good application of equity theory. One of the journals I'm involved in also keeps track of performance of the reviewers in terms of speed, timeliness, constructive feedback, depth, etc. Each year, they acknowledge the 10 best reviewers on the basis of those scores (as rated by senior editors) - this is also pretty motivating!

Mantonat

In the corporate world, the solution is generally to include a deadline for review along with the message "please review by X date. If you do not respond by this date, I will assume that the document has your approval and will move forward with publication."

Eric H.

As a fellow academic I fully agree with this - it can be very frustrating to wait months on end for a single reviewer. However, I also worry this would cause reviewers to just give uninformed reviews or outright dismiss work because they don't have time to read it (but want to make the deadline). Waiting is frustrating, but not nearly as much as having months of work dismissed by a two sentence review.

Drill-Baby-Drill drill Team

A chemical reaction will only proceed at the rate of the slowest portion. A rate limiting process.

Like molecular rxns, so is the bureaucratic process. The slowpokes control the overall progression of any process.

Like students with Term Paper Deadlines, perhaps DEADLINES are necessary. With severe consequence for failures.

Ian Kemmish

A cunning plan with but one tiny flaw: suddenly it''s in everyone's interest to do a slapdash job of reviewing, rather than just in most people's interest. What happens if the only person who's actually doing a thorough job is the one taking a year over it, who is suddenly not even going to be asked any more?

Tony

Also, everyone will request their best buddies review their work... Independence will go out the window for (you approve me and I'll approve you).

EdM

The longest i've ever had a paper under review is 5 weeks so maybe journals I submit to are doing a better job. The the longest it's ever taken me to review a paper is 3 weeks. No matter what my original intentions are however, I rarely start reviewing the paper until a few days before the deadline. I only accept papers to review if I can get them down by the deadline (which is normally 2-4 weeks after I receive it).
I don't understand why there would be a terrible problem in economics. Shouldn't the reviewers not accept the paper if they can't complete it by the deadline. In my field it's completely acceptable to do that. I think the best bet though, is to set a hard deadline after which a new reviewer is found. If the deadline is 6 weeks, most reviews will be turned in at 6 weeks. If the deadline is 2 weeks, I'd be willing to bet most would be turned in at 2 weeks. No one is too busy to get a review done in 6 weeks, so it really shouldn't be longer than that.

Read more...

TG

It sounds like the problem here is that the process moves as fast as the slowest reviewer. Why not create a system whereby 2 potential reviewers are selected as qualified for every 1 review that is needed? So if 5 reviews are sufficient and you have received 4 already, you have 6 potential candidates to complete the one remaining review. Also, why not add some small $ to the equation? The laboratory or university associated with the scientist seeking review would pay a $500 administrative fee for their reviews requested, which would be split amongst the instituions of the scientists that completed the review. If you are reviewing as much as requesting reviews, its a zero sum game. The more selfish scientists/institutions will pay a price for that behavior.

AaronS

How about this:

"If you accept this assignment, you agree to complete your review by XX/XX/XX. If you cannot do this, please do not accept the assignment. NOTE: If you fail to complete by deadline, your own writing will be given last priority of review in the future."

It might make someone do a rush job? I'm thinking some of the year-long peer reviews are a matter of someone cleaning off their desk and saying, "Oh, my! This has been here for over six months--I better put something out ASAP!"

Joe

Have Journals compete for authors on the basis of promised review time.

Submit here and we promise a decision in 3 months for anyone who has reviewed for us. For new submitters, it's a six month promise. Reviewers have an incentive for quick turn around (in what other world is 3 months a quick turnaround?), and authors have an incentive to submit.

Publication times are silly in this day and age, where a paper can be across the world in 30 seconds and printing is so much easier/faster than it was even 10 years ago.

MBS

I think it's an idea that will incentivize good behavior, but there might be adverse consequences. Namely, repeat players will have the best of reasons to be on their good behavior, review papers quickly, so that they can be matched up with other reviewers with best practices. But then there are the not-so-frequent-repeat players, and the occasional one shot-players. The last two categories probably outnumber the top category of repeat players. The last to categories of players won't be all that moved by the policy, but the real problem is that it puts researchers into little cliques based on something other than area of research. The repeat players will interact only with other repeat players, reviewing each other's work, while the others will similarly only move among their circle of peers. My concern is that the policy would be a breeding ground for group-think and create a disconnect (or amplify it) between those who publish a lot and those that maybe have one or two publications to send out there.

Read more...

Doc

As editor of Journal of Real Estate Portfolio Management I (and others before and after me) followed most of the suggestions above and sent routine follow-up e-mails to reviewers after a month. We also give a Red Pen award each year for the best reviewer for each journal we publish. Despite the effort there are some colleagues who can be relied upon and some who can't. Over time the conscientious reviewers end up taking on a bigger burden and the laggards fail to do their share of the heavy lifting. Perhaps matching reviewing speed would be possible despite the other variables, but just saying that you will could be sufficient warning to the slothful. At the extreme, take them off the editorial board which might have some modest consequence for them.

Drill-Baby-Drill drill Team

Alternative Solution:
Put Academic Review Articles on an Open Freely Edited and Annotated Web Site Open to ALL Academics in a Particular Field. Have the Top Ten Articles of the Day for speed reviews and comments that will occur at lighting speed.

Call it WIKI- ACADEMIA.

David Chowes, New York City

Great idea, Dr. Hamermesh. Peer reviews are quite important. This idea should be attempted. If so, will it work?

We amy find out.

William

Yes, can be a serious problem. The worst I've ever had was 15 months, and I had to threaten to withdraw the paper to get any action. Truly awful journal and I'll never submit there again, even though the impact factor is the best in the field (how, I'll never know).

Maybe a better incentive is to just get journals to advertise their average review time.

John Smith

To Ian Kemmish:

The reviewers can already do a lousy job. The disincentive is already there. The threat that they are exposed as fools should the paper turn out to have serious errors and is noted by the media.

Richard simper

Easy - pay me to review the paper - I can not still believe we have to pay submission fees and then editors expect us to work for free - odd being an economist

MangoPunch

Or journals could hold publication of an author's piece until the mean review time that they have taken has elapsed.

Gary

The Climategate emails showed how prone the peer-review system is to gaming. A solution to "pal-review" and one that actually aids in ferreting out the facts/truth of a research question is to broaden the pool of reviewers. Let the wisdom of crowds give the matter a thorough examination. Sure, it's not without its own problems, but the blogs are beginning to do this now and the worth of work is being evaluated much sooner than with the old peer-review process. Online journals where authors interact with reviewers to fix flaws and shortcomings will advance knowledge. The worry is that they may not advance careers....

John

One problem with this scheme is that it further rewards established authors. Established authors (i.e., those that are well published and still productive) have an incentive to referee papers quickly as they will then receive faster reviews. Established authors who are no longer productive but are still asked to referee based on past work have no incentive to be quick.

Junior authors who are not established (and therefore not typically asked to referee) are more likely to get a slow referee than a fast referee. This would further penalize junior authors, and potentially make getting those first publications even more difficult.

As for whether this scheme would make the referee reports more slapdash, personally, I've waited 18 months only to get a cursory rejection, so I'm not sure that would make the problem worse than it already sometime is.