Preston McAfee Shakes Things Up in Academic Publishing

Most of our blog readers couldn’t care less about refereeing in academic publishing. And they should be thankful.

For a tenure-track economist, getting published is a brutal process. You spend a year or two coming up with an idea, try to think over every aspect of the problem, collect and analyze data, and finally produce a 30-page paper summarizing all the work. You can only send it to one journal at a time. Then you wait six to nine months to get a decision from an editor.

At the top journals, 90 percent or more of the papers get rejected, which means you then head back to square one at another journal. Even if the referees and editors like the paper, the best you can hope for is a “revise and resubmit” invitation, which entails pages and pages of demands for changes before it will be reconsidered for publication. In a few cases, the comments I got back from referees and editors were longer than the paper I submitted in the first place!

In the off chance that your paper ever gets accepted, there is often a delay of a year before it appears in print. Thus, from start to finish, the whole process takes 3 to 5 years.

For many young economists, the tenure clock expires before the publication process completes. These untenured economists don’t have any choice but to try to navigate the system. For many tenured economists who don’t actually need to publish in these journals, the headaches associated with peer-review are just not worth it. As economist Glenn Ellison recently noted, the top economists are increasingly turning to edited volumes and “special issues” of journals that either have no peer review or only a wisp of it.

Preston McAfee is trying to shake things up. I’ve had a deep admiration for McAfee since I first met him back in 1997 in Barcelona. I was giving an academic talk there — one I had given at least a dozen times before. By the fifth or sixth time I gave the talk, the questions had become routine. I hadn’t heard a good new idea on the paper from an audience member in a long time. Which is why I was so shocked when some guy I had never seen before asked two incredibly insightful, totally new questions that day. That guy turned out to be McAfee. He is an original thinker, as you can tell from his open-source Principles in Economics textbook. (He also has adorable children.)

McAfee just took over as the editor of a journal called Economic Inquiry. It has been a long time since I submitted a paper to that journal, in part because the last time I did, the editorial process was especially onerous. Just a few weeks ago one of my co-authors suggested sending our paper there, and I vetoed the idea for that very reason. Luckily, I’ve been slow in getting around to submitting to another journal, because with a new sheriff in town, things are changing at Economic Inquiry. After taking over as editor, McAfee posted this on the journal’s web page (hat tip to Marginal Revolution, who blogged about it):

Editor’s Announcement: No Revisions Option

Journal time to publication lags have become embarrassing. Many authors have 5-year submission-to-print stories. More insidious, in my view, is the gradual morphing of the referees from evaluators to anonymous co-authors. Referees request increasingly extensive revisions. Usually these represent improvements, but the process takes a lot of time and effort, and the end result is often worse owing to its committee-design. Authors, knowing referees will make them rewrite the paper, are sometimes sloppy with the submission. This feedback loop – submitting a sloppy paper since referees will require rewriting combined with a need to fix all the sloppiness – has led to our current misery. Moreover, the expectation that referees will rewrite papers, combined with sloppy submissions, makes refereeing extraordinarily unpleasant. We – the efficiency-obsessed academic discipline – have the least efficient publication process.

The system is broken.

Consequently, Economic Inquiry is starting an experiment. In this experiment, an author can submit under a “no revisions” policy. This policy means exactly what it says: if you submit under no revisions, I (or the co-editor) will either accept or reject. What will not happen is a request for a revision.

I will ask referees: “is it better for Economic Inquiry to publish the paper as is, versus reject it, and why or why not?” This policy returns referees to their role of evaluator. There will still be anonymous reports.

Authors who receive an acceptance would have the option of publishing without changes. If a referee noticed a minor problem and put it in the report, self-respecting authors would fix the problem. But such fixes would not be a condition of publication.

During the course of the experiment, an author may opt for submission under the old system. The old system remains the default; to opt in to the new system, please add “I submit under the ‘no revisions’ policy.”

I love McAfee’s idea. He’ll be getting my paper next week. I suspect I will make history by being the first paper to be rejected under his “no revisions” policy.


EmilyAnabel

It's great to see people rethinking academic journals as there are many respects in which they are broken but inertia seems to prevent much progress.

I don't know much about Econ journal pricing, but there have been the beginnings of a revolution in CS and math journals regarding the pricing. A number of people have signed the "Banff Protocol" http://members.cox.net/banffprotocol/ which is an agreement to not referee or submit to expensive, for-profit journals. There have been some mass resignations of editorial boards when journals didn't agree to have more reasonable prices. The point is that the bulk of the work (writing, editorial decisions, refereeing) is done by uncompensated volunteers and journals are quite expensive. There are plenty of affordable journals run by professional societies as alternatives to the for-profit (Elsevier etc.) journals. Still, if you look at the list of signers of the Banff protocol, they are all researchers who are tenured including some vocal bigshots like Rob Kirby and Herb Wilf.

Read more...

bgoffe

In some ways, Economic Inquiry is moving to the way "things used to be." In "The Slowdown of the Economics Publishing Process," JEP, Oct. 2002, Ellison describes why the slowdown has occurred. Part of the reason is the dramatic increase in "revise and resubmits." My favorite quote from the paper: An anecdote I find revealing is that a senior economist told me it looks odd to him to see young economists' resumes trumpeting that papers have been returned for revision. When he was young he never would have listed a revise-and-resubmit on his resume because he would have been embarrassed that something was wrong with his initial submission." In "Evolving Standards for Academic Publishing: A q-r Theory," Glenn Ellison, JPE, Oct, 2002, describes a theory of how changing social norms in publishing can explain the increase in revise and resubmits. (FWIW, I posted this same comment in Marginal Revolution and DeLong's blogs -- what are the norms in multiple posts to the same idea ?).

On the “Banff Protocol” I must admit I'm a bit pessimistic. Something similar was done in the life sciences and it seemed to have little impact. However, fields vary dramatically so perhaps it will work in math and CS.

Read more...

zbicyclist

Under the "no revisions" policy, why not have referee's comments included after the paper (perhaps cleaned up a bit)? This might not be appropriate in most cases, but in others it might be extremely appropriate.

Result: (1) faster publication, (2) author's intent clear, but flaws/alternative explanations/suggestions for others also included.

kah

Does the "no revisions" policy include a "no resubmitting a rejected paper to the same journal after doing self-motivated revisions" clause?

bgriffs

Dr. Levitt-

Are you implying it would take you a week to come up with something that would be rejected? :)

I think I could beat you to the punch on that one.

bsci

This is more economist exceptionalism. There are many problems with peer review, but most other fields don't have 5 year periods between initial submission and publication. I've head the median lifetime # of publications for an econ PhD is 0 since the majority get a non-academic job and never get their dissertation research through the publication process.

The Economic Inquiry policy seems to be designed more to get attention than to fix the problem. The obvious (harder) solution is to demand reviewers respond in a reasonable amount of time and they review the articles based on clear guidelines, not on what additional studies they'd like the authors to do. Perhaps reviewers could be asked a simple question, "if the authors devoted another 80 hours to data processing to address your concerns, is this paper of a quality to get a revision accepted." This allows for constructive comments while limiting the requests to a modest about of time in the authors' lives.

Read more...

JanneM

I'm not an economist, but I do publish academically in another field; thus my comments are not absolutely certain to pertain to economics, though I think they do.

First, there's journals and journals; some journals are tougher (and worth more to publish in) than others. People know this, and they aren't stupid. You usually have a pretty good idea of where your paper could fit, and where you stand a decent chance of getting it published without "throwing it away" on a substandard place. People do take this into account. With that in mind I'd guess the normal submitted paper has well over 50% chance of getting published at the first submited journal.

Second, yes, sometimes reviewers go way over board. Especially embarrassing is it when a nominally anonymous reviewer strongly "suggests" including a dozen new references in the paper, all co-written by the same author and only tangentially about the subject of the paper. Usually, however, the problem is the opposite, with reviews giving you terse, vague and singularly unhelpful comments that do not suggest what the reviewer actually wants to see. And in the end, the reviewer is not the arbiter, the editor is. You are free to reject suggested changes and write why you do in the comments to the editor. If your reasons have any merit ("I'd love to add these references to the mating habits of the starspeckled shinwarbler; but as the paper is about precambrian granite formations I do not feel it appropriate") then the paper will be accepted nevertheless.

Third, once it's accepted, it's accepted. As far as your career and your merits are concerned, that's what counts. Put it on your CV and use it for grant applications. The time when the actual paper copy is pushed out the door is of rather minor importance.

And fourth, while it can take time (and in the sciences the time between submission and print is more like a year, not several years), it's not like you're sitting on your hands while the process is moving along. In all probability you're already at work on your next project in tandem with writing up the paper on your last one, and the paper revision is probably concurrent to you writing the paper on the next project (and while you're working on the step after that).

We do have book compilations, and we frequently have special issues. Publishing in a book, however, has only marginally more value than posting a note about your research in red and green crayon on the lab refrigerator. And special issues hare peer-reviewed; depending on the class of journal the rejection rate may be higher for special issues than regular ones.

From what it sounds like, the only issue you seem to have is the review time. And while the "no revisions" idea is interesting, I'm afraid it would end up becoming an impediment, rather than a speed-up. Before, people would get a list of required changes, which would be done, then the changes submitted. Now, people will get rejected with a list of reasons - and so the paper will get edited according to those reasons, and then resubmitted to the journal. The only difference is now that the paper will have to reenter the queue from the back and perhaps assigned to another reviewer, resulting in even longer lead times and higher risk of multiple rejections.

How about cutting the time allocated for review, and limiting the amount of commentary? If a reviewer knows they have six weeks to submit their review, and the review is max one A4 page long, then the amount of gratuitous changes required will be small; only the things that really matter will be brought up, and only at the the level of statign the problem. It would leave free hands to the authors as to how they address those issues.

Read more...

gsiemens

We (Learning Technologies Centre, U of Manitoba) have been working on an alternative model of scholarship over the last few months: http://www.elearnspace.org/Articles/journal.htm . As has been noted, the time cycle of scholarly publication is too long...and, perhaps more importantly, the lack of dialogue following publication overlooks the value readers (not nec. only the peer review members) can provide in enlarging an idea or concept.

pirot

The "no revision" model seems to be very similar to the conference mode in Computer Science. There you have a rigorous reviewing process, with very low acceptance rates for the conferences. A good model, with its own shortcomings (e.g., when the initial submission is not accepted, when you resubmit you get a different set of referees, which may request completely different things to be addressed and so on.)

Giselle

there seem to be 3 Academic Networking Sites sprining at the same time : www.pronetos.com , www.academia.edu and www.hypertope.com

The big question is which one is going to lead the way?
I've tested out all of them and here is what I think:

Pronetos.com is simply a premitive site - they have a long way to go before they can call themselves a networking site. It requires a lot of work before it can move out of Alpha stage. The repository feature is okey as far the principle is concerned, but it is just too primitive to address academic needs. In short, I cannot see Pronetos developing beyond a tiny community.

Academia.edu is another website where PhD students can create a page for themselves. But since membership is for .ac.uk and .edu e-mail accounts only, and because members' e-mail addresses appear on their pages -- it is nothing less than a goldmine for spam-bots. There is also a "paper-tracker" mechanism which is supposed to be collecting all academic papers from internet, but it just doesn't work :( I can only imagine how much money was spent on the name "academia.edu" alone, yet in 4-5 months that it this website has been around it still didn't develop any new feature and is still in Alpha phase of development. Plus, it doesn't seem like any new members are joining. I could be wrong, but it looks like that project has been abandoned, because nothing has changed since the Alpha launch.

now... http://www.hypertope.com is my personal favourite of the 3 academic networking sites (so please excuse my bias). And it is packed with features designed for academic and research needs (categorising contacts, discreate e-mailing, private messaging, flexible privacy features, research groups, dinamic networking, integrated blogging, Biblio Repository, Calls for Papers, filterable calendars of upcomming events, RSVP technology for promoting events, etcetera etcetera -- and it all works like a clockwork. It's proper cutting edge Web2.0. It was launched last week, so it is the youngest, but it is rapidly growing and seems to be popular both with professors and PhD researchers. And the community is very vibrant. To join hypertope.com one must receive an invitation from an existing member, but for a brief promotional period of time they opened the gates so that initial members can join and invite their contacts.

It's all very exciting. SNS has been used to reconnect with classmates, and then for business purposes and then for medical practitionors and now scholars.

Read more...

Preston McAfee

I'm still waiting for your submission! Moreover, you now do not have the opportunity to be the first rejected or the first accepted under the new policy!

To the naysayers: it is an experiment. It is an author's choice to submit under the no revisions policy or not. If you have a good paper and are sick and tired of referees making you make marginal changes for no discernible purpose or a 5th year assistant professor in great need of a decision quickly, it is nice to have the option. If you don't really know what you want to say or how to say it, it is a bad choice to submit under no revisions.

So will the critics finally remember that there is a market, and if this isn't an improvement, it will fail. It is worth trying. If it is an improvement, EI will get better. If I just get Steve's paper, EI will surely get better.

Preston McAfee

Oops, my bad. Steve did submit a paper.

At least I was right that the policy would improve our submissions.

Wyatt Werner

RE: Pronetos, we (yes, I'm the CFO so I'm quite biased) are in our infancy. We're up to 120 users as of noon today. We launch new features each month, so keep your eyes on us.

Pronetos isn't just another social network. It combines a content repository with a social network. This means that scholars can comment and rate the content in the repository. The social networking revolves around the content. Soon, members can select their favorite content to combine into a custom textbook. We're an open access repository using creative commons licensing, so anyone can view and use the content on the site, unlike siloed repositories and closed subscription-only journal databases. That's our main value proposition at the moment.

Coming soon, a peer-review process by which the most impactful content will be published in a series of academic journals. We take a lot of the pain out of the publication process, which gets the best research out to the academic world much faster, and to a larger audience. This is the pain-point Levitt mentions in this post. We streamline publication and accelerate discovery.

Read more...

DW MacKenzie

Has anyone thought of setting the marginal cost of resubmission above zero? As it stands now some journals charge a fee for initial submission, but no journal charges a fee to revise and resubmit. I am not sure how common it is to revise and resubmit multiple times (I have never had to), but if this is part of the reason why some articles take five years to finish the review process, then editors might consider a progressive resubmission fee, whereby the fee increases with each additional resubmission. Resubmission fees might internalize any externalities imposed by slothful authors. I have not heard such a thing suggested before, but c'mon guys, were economists. If we can figure out marginal cost pricing, who can?

science minded

Now I know why I write books. But even there, my first edited book's editor edited out my main point and there was nothing I could do about it as it was a contract situation. Curiously, it is still in print after 13 years. I have since learned to make sure that what I send to a publisher is correct (as correct as it can be at the time) and to protect myself against predators (those who don't understand and rather than ask for a revision from the author by means of which the author is given the opportunity to make their understanding clear, they revise it in a way that the author is edited out. so much for science and more for vanity. so the idea of a no revision policy sounds good. Perhaps I will try it.

steve

Great idea,

But could be extended to even a revision's process whereby the item is either

1."accepted with or without revisions" - where the author could choose to make those small revisions or not.

2. "Only accepted with revisions" - the author can make all revisions and be published; or where some revisions are rejected the paper becomes part of the usual process.

3. Rejected.

Overall though, a great idea for any "untenured" economists

EmilyAnabel

It's great to see people rethinking academic journals as there are many respects in which they are broken but inertia seems to prevent much progress.

I don't know much about Econ journal pricing, but there have been the beginnings of a revolution in CS and math journals regarding the pricing. A number of people have signed the "Banff Protocol" http://members.cox.net/banffprotocol/ which is an agreement to not referee or submit to expensive, for-profit journals. There have been some mass resignations of editorial boards when journals didn't agree to have more reasonable prices. The point is that the bulk of the work (writing, editorial decisions, refereeing) is done by uncompensated volunteers and journals are quite expensive. There are plenty of affordable journals run by professional societies as alternatives to the for-profit (Elsevier etc.) journals. Still, if you look at the list of signers of the Banff protocol, they are all researchers who are tenured including some vocal bigshots like Rob Kirby and Herb Wilf.

Read more...

bgoffe

In some ways, Economic Inquiry is moving to the way "things used to be." In "The Slowdown of the Economics Publishing Process," JEP, Oct. 2002, Ellison describes why the slowdown has occurred. Part of the reason is the dramatic increase in "revise and resubmits." My favorite quote from the paper: An anecdote I find revealing is that a senior economist told me it looks odd to him to see young economists' resumes trumpeting that papers have been returned for revision. When he was young he never would have listed a revise-and-resubmit on his resume because he would have been embarrassed that something was wrong with his initial submission." In "Evolving Standards for Academic Publishing: A q-r Theory," Glenn Ellison, JPE, Oct, 2002, describes a theory of how changing social norms in publishing can explain the increase in revise and resubmits. (FWIW, I posted this same comment in Marginal Revolution and DeLong's blogs -- what are the norms in multiple posts to the same idea ?).

On the "Banff Protocol" I must admit I'm a bit pessimistic. Something similar was done in the life sciences and it seemed to have little impact. However, fields vary dramatically so perhaps it will work in math and CS.

Read more...

zbicyclist

Under the "no revisions" policy, why not have referee's comments included after the paper (perhaps cleaned up a bit)? This might not be appropriate in most cases, but in others it might be extremely appropriate.

Result: (1) faster publication, (2) author's intent clear, but flaws/alternative explanations/suggestions for others also included.

kah

Does the "no revisions" policy include a "no resubmitting a rejected paper to the same journal after doing self-motivated revisions" clause?