Interview with Peerage of Science

published by scottchamberlain on Wed, 01/25/2012 - 06:09

Peerage of Science(PoS) is a new venture in peer review for research papers. What is PoS about? What do they do? We caught up with PoS the other day and asked them a couple questions. Let us know what you think. Discuss below in the comments, or on Twitter or Facebook.

Is peer-review broken? If so, what needs to be done to fix peer-review?

No, peer review is evidently not broken as a whole: some editors consider the traditional process to work very well for their journals. Partly this reflects the different prestige, resources and management arrangements different journals have, influencing how much effort academic editors need to put into obtaining useful reviews in decent time.

Some say "if it ain't broke, don't try to fix it". But anyone who has owned an old car knows very well that when an integral part of the vehicle starts to make troubling noises, it is high time to do something about it before the part breaks. The peer review part of science is indeed starting to make troubling noises (recent report here ). If you are a prominent scientist in your field, you are likely to receive several review requests every week, so you either have to decline most or do the sort of hasty reviews all scientists have sometimes been misfortunate to receive. One editor of a very prestigious journal told us he just had asked 11 persons to review a manuscript, and all but one had declined. Another told the number of people editor has to ask in order to get the two reviews required is continuously increasing. Studies show the inefficiency of peer review causes significant publishing delay, the reports are poor in predicting future citation or access rates, and so forth.

But I do not want to fixate on the problems. Even if peer review was running smoothly, we should be interested in making it better. There are several things Peerage of Science is doing to achieve better peer review, but the most important by far are to provide scientists incentives to voluntarily write high-quality reviews, and to provide a better method for matching manuscripts to appropriate journals than the trial-and-error process used currently.
What are the current goals of Peerage of Science?

Overall, the goal is to empower scientists, and make the evaluation of science, scientists and scientific organizations more efficient, more accurate and more rewarding.

Scientist-as-Author should no longer need to find appropriate outlet for one's results by time-consuming trial-and-error process; instead, editors make the decision what manuscript fits where and authors choose from offers they get.

Scientist-as-Reviewer who does reviewing well should be rewarded for doing that.

Scientist-as-Editor should no longer need to manage peer review processes at all; instead they should be able to focus on advancing the journal's field of science as editors, and on guiding the journal to the direction they want to with data and tools for easy, efficient and proactive decision-making.

Immediate goals are to accelerate the community growth, get more selected journals to engage in trial use of the service, and to start including other fields of science, medicine and technology.
If you were to choose only one of your goals for Peerage of Science, what would it be?

The opportunity to gain merit as a reviewer is key to the accomplishment of all the other goals, so if I had to choose one it would have to be that.
What is the process of using Peerage of Science? Walk us through how Peerage of Science works.

The process has many important novel features beyond this short description, but essentially it is an automatically controlled process flow of four stages with deadlines defined by authors at the start: 1) author sends ms, reviewers discover it and voluntarily write reviews that author immediately receives; 2) reviewers evaluate each other; 3) author send revised manuscript; 4) reviewers evaluate the revised manuscript. And in any stage after 1, editors can send publishing offers for authors.

I recommend those interested in the details to first read the Overview, and then look at the Process Flow.
What does Peerage of Science do differently than the traditional peer-review process?

So many things are different, that it is easier to describe what good features of the traditional process are maintained in Peerage of Science. It is similar to the traditional process in that it allows editors to suggest reviewing to scientists they trust to be good reviewers, maintains reviewer anonymity (but encourages post-review openness), requires that authors revise the manuscript in response to the comments, and is completely independent of the publishing business.

The differences that have the largest impact are that reviewers voluntarily choose what to review, that authors receive reviews immediately upon upload, that reviews are themselves peer-reviewed, that scientists can gain reputation from reviewing, that journals can offer to publish manuscripts that otherwise might not have been submitted to them at all, and that authors sometimes may get several offers from which to choose. But there are many, many little details in addition to these that are made possible in Peerage of Science.

Some old concepts do not exist at all in Peerage of Science. One of them is the concept of "rejection". In Peerage of Science the editors are anonymous when they track peer review processes they are interested in, and nobody else gets to know if they decide the process no longer warrants interest and choose to stop the tracking without making an offer. No more hard feelings due to having to send or receive a rejection letter.
Is the goal for Peerage of Science to eventually entirely replace the current incarnation of peer-review?

Things change slowly in science; the "current incarnation" does not differ very much from the way things were done over 300 years ago when The Philosophical Transactions of the Royal Society first started to send manuscripts to reviewers. So such a goal would be a little too grand. However, we do believe that if scientists in some fields choose to routinely send their manuscript first to be peer reviewed in Peerage of Science regardless of where they plan it to be published eventually, many journals in those fields will no longer see reason to hold on to the old process with all its downsides. This may happen quickly too, but it is really up to the choices of the scientific community.
How might Peerage of Science facilitate the training of graduate students and early-career researchers in how to do good peer-review?

This is very important aspect that may really impact science for the better in the long run. Right now you only see the reviews you receive for your own manuscripts, and learn the difference between a good and a poor review by experiencing them. And when you start to write review reports, there is little feedback to tell you if you did well or failed.

In Peerage of Science you not only get to see a lot of reviews, but also see what others think of them, and you get clear, quantitative evaluation of your own performance, too. When some people start to gain reputation as excellent peer-reviewers, their Peerage Essays (i.e. the reviews done in Peerage of Science that they choose to publish) are examples to students of what a high-quality peer review looks like.
Have any participating journals been willing to forgo their own round of internal peer-review before publishing a manuscript they originally found via Peerage of Science? Do you think this will increasingly be the case as Peerage of Science becomes further established?

It is very early days, as we just opened the service on November 22. The first manuscripts are coming out of the peer review process now, so naturally none of them are published yet. We will find out in few weeks what starts to happen when authors export the reviews to accompany traditional submission to journals of their choice. We expect that initially many editors will want to have additional reviews, but believe that when editors find out that Peerage of Science reviews can be trusted and management of their own additional process is just unnecessary extra work for themselves and the reviewers, most will be happy to take the advantage of quick and efficient decisions.
How do you think scientific research will look in 2050?

Hmm, one way to approach that question is to look how scientific research today is different from 1971. The answer is "not very different at all", except for the technological ease in accomplishing routine tasks and some new ways of gathering data. On the other hand, before 2050 we may have to face technological singularity, or more likely the consequences of greed multiplied by over 9 billion individuals of a species prone to anger with only one planet to live on; in both scenarios scientific research as we know it has disappeared.

Further reading on peer review and Peerage of Science:

Fox and Petchy propose a banking system for peer review called "PubCred" here
The Scholarly Kitchen blog discusses the PubCred idea here
Lortie, of the Oikos blog, proposes discusses potentially paying reviewers here
Jeremy Fox, also of the Oikos blog, discusses Peerage of Science here
There's an interesting discussion of Peerage of Science on Ecolog - go there to check it out (search "peerage of science")
Mike Folwer's blog post about Peerage of Science here
Science Insider story on Peerage of Science here
The Peerage of Science blog here