Dedicated to Prof. K. Peter C. Vollhardt on the occasion of his 75th birthday
1
Introduction: The History of Select Crowd Reviewing
The foundation for Select Crowd Reviewing at Synlett was laid several years ago, when drawbacks of traditional peer review became continuously
more apparent. The two of us were working together in the Synlett Office, Manuel van Gemmeren serving as editorial assistant and at the same time doctoral
student of Benjamin List, then Associate Editor, now Editor-in-Chief of Synlett. Striving to ensure a fast, fair, and balanced evaluation of the manuscripts submitted
to Synlett, we had very high expectations regarding what the peer review process should accomplish.
Besides providing the required feedback that would allow us to judge a manuscript
to be suitable or unsuitable for publication, we expected that a good peer review
process should help the authors to improve their manuscript as much as possible and
whenever possible. However, like many other journals, we found it increasingly difficult
to find qualified scientists willing to invest the time necessary for providing such
a review (‘reviewer fatigue’).[1] This is almost certainly due to an uneven workload distribution and an ever-increasing
volume of submitted manuscripts. The process was further hampered by an increasing
frequency of nonresponses. While we understand that we have no right to expect anything,
technically not even a decline to review, from potential reviewers we contact, such
silence can noticeably slow down the evaluation process. Even worse, when referees
agree to provide a review and then suddenly ‘play dead’ leaving a manuscript stuck
in review for additional weeks at the costs of the authors. Finally, and again as
many other journals, we frequently encountered cases where after obtaining two or
more referee reports, these were drastically contradictory, e.g., one reviewer recommending
minor revisions and another recommending rejection based on a lack of novelty. Who
should we give more credit to? Wouldn’t it be great if these two referees would talk
to one another and then provide us with a consensus opinion?
These thoughts led us to ponder the possibility of having the reviewers cooperate
in the reviewing process as a form of scientific discussion rather than independently
reviewing the manuscript.
Further technical questions needed to be sorted out, most importantly: In order to
have a real conversation, the reviewing process would have to take place on a similar
timeline. How would this be possible with two to three reviewers who would have a
look at the manuscript over the course of weeks? We concluded that this would not
work and that instead a larger number of reviewers with access to the manuscript over
a shorter timeframe was required, such that actual conversations would evolve, analogous
to discussions on scientific panels but also on social media, but with a carefully
selected group of participants – the reviewing crowd. We quickly realized that such
a crowd reviewing system would actually offer substantial inherent advantages that
could serve to alleviate the challenges traditional peer reviewing faces today. At
the same time many questions remained: Would the response be good? Would the comments
from such a procedure be substantive? Would we be able to ‘recruit’ suitably qualified
scientists to participate in the reviewing crowd?
Overall, at this stage we saw many opportunities and equally many challenges and concluded
that the only way to evaluate the idea was through experiment (what a surprise that
chemical synthesis scientists would choose this approach…).
Together with the Thieme publishing team in Stuttgart and coordinated by Dr. Denis
Höfler, the successor of Manuel van Gemmeren working as the editorial assistant in
Mülheim, a startup company offering an interesting online platform was identified
(Filestage). The software that Filestage then provided was originally designed to
aid advertisement and media companies to conduct collaborative though nonlocalized
work on graphical material. International teams of photographers, writers, and art
directors, located all over the planet, could jointly discuss graphical art. We reasoned
this to be close enough to our concept of having multiple referees working jointly
but not locally on manuscripts. Accordingly, Select Crowd Reviewing was launched as
a pilot project in 2016. At this stage all manuscripts were simultaneously evaluated
through Select Crowd Reviewing and traditional peer review, such that a reasonable comparison could be made between
the two methods. The early conclusion from these trials was that ‘crowd-based peer
review can be good and fast’.[2]
As can be seen from Figure [1], the proportion of manuscripts evaluated through Select Crowd Reviewing remained
comparably low in those days. The decision was made to extend Select Crowd Reviewing
in form of a long-term trial during which authors were given the choice to opt-in
on Select Crowd Reviewing either instead of, or additionally to, traditional peer
reviewing. This phase began in mid-2017, which marks the beginning of a continuous
rise in the proportion of manuscripts undergoing Select Crowd Review (Figure [1]). This extended trial was highly successful. Select Crowd Reviewing was increasingly
well accepted by the community, delivered valuable feedback, and displayed very fast
turnaround times.
Figure 1 Development of Select Crowd Reviewing: increasing percentage of manuscripts evaluated
by Select Crowd Reviewing relative to the overall number of manuscripts sent for external
peer review.
This led to a continuous increase in the percentage of manuscripts at Synlett being evaluated this way. The realization that Select Crowd Review ‘is here to stay’
is also reflected by several decisions made by Thieme as the publisher. Firstly, steps
were taken to implement Select Crowd Reviewing at further journals beyond the boundaries
of Thieme Chemistry (e.g., the medical journals Röntgenforschung and International Journal of Sports Medicine). Secondly, Crowd Reviewing Editors were appointed at both SynOpen (Dr. Philippa Cranwell) and Synlett (Dr. Manuel van Gemmeren).
Finally, in 2019 the Editorial Board of Synlett decided to make Select Crowd Reviewing the default method for the evaluation of manuscripts.
Authors now have to actively opt-out of Select Crowd Reviewing. SynOpen has gone even further – manuscripts here are exclusively evaluated using Select Crowd
Reviewing. This decision explains the rather sharp increase in the percentage of manuscripts
undergoing Select Crowd Review in the first half of 2020 (Figure [1]).
2
The Select Crowd Reviewing Workflow
Many steps of the workflow for a manuscript that undergoes Select Crowd Review are
identical to those of a manuscript evaluated by traditional peer reviewing. The process
begins with the submission of the manuscript by the authors (Step 1, Figure [2]). During submission, an active opt-out from Select Crowd Reviewing causes the manuscript
to be further processed via traditional peer reviewing. Irrespective of this decision, the manuscript first undergoes
technical checks at the Synlett Editorial Office in Stuttgart.
Figure 2 Workflow of Select Crowd Reviewing at Synlett/SynOpen
These checks do not consider the scientific novelty or relevance of the manuscript
but exclusively focus on technical and formal aspects, i.e., plagiarism. Manuscripts
that do not comply with the standards of Synlett may be sent back to the authors at this stage. Otherwise, the manuscript progresses
and is assigned to an Editor (Step 2). The Editor then performs a first scientific
evaluation of the manuscript. Manuscripts judged to be outside the scope of Synlett or unsuitable for publication in Synlett for another reason are rejected without peer review at this stage. Until now, no
differentiation has been made between manuscripts that undergo Select Crowd Review
or traditional peer review. For all other manuscripts, the Editor decides whether
to send the manuscript for traditional peer review, Select Crowd Review, or both.
Note: Authors who do not opt-out of Select Crowd Review are not guaranteed that Select
Crowd Reviewing will be used to evaluate the manuscript. Editors may decide against
this, e.g., when manuscripts are considered too specialized to ensure that an evaluation
by the members of the crowd would be appropriate or when potential conflicts of interest
are foreseen.
Despite these limitations, a sizable proportion of manuscripts is ultimately evaluated
in this way. In these cases, the Editor sends a request for Select Crowd Reviewing
to the Crowd Review Editor (Step 3). This is fully analogous to the invitation of
an external reviewer. The Crowd Review Editor then uploads the manuscript to a secure
online platform to which the members of the crowd have access, informs the members
of the crowd that one or more manuscripts have been uploaded and invites them to contribute
to the reviewing of these manuscripts. The members of the crowd receive invitation
e-mails containing the titles and authors of the manuscripts in question as well as
direct links to the manuscript (Step 4). Amongst the members of the crowd, those who
are currently available and feel like they may contribute to its evaluation then use
these links to access the manuscript in an anonymized form. Each reviewer gets assigned
to a permanent alias, such that the Crowd Review Editor is able to determine the identity
of a reviewer, e.g., in a suspected case of scientific misconduct or inappropriate
commenting. It should be noted, however, that we have not observed such behavior so
far.
The core of Select Crowd Reviewing now takes place (Step 5). The members of the crowd
all have access to the same version of the manuscript and see all previous comments
on the manuscript. The reviewers therefore react to the manuscript AND the comments
already made. They can thus agree to comments already made (instead of writing down
the same comments in a different phrasing). More importantly, the reviewers can disagree
with previous comments and thereby start a discussion. Oftentimes, the same reviewer
who posted the original comment will return and defend his/her position or consent
to the criticism and change his/her recommendation. This process also allows for third
and fourth reviewers to contribute as well. At the end of such a discussion, a consensus
opinion can very often be identified or alternatively, the arguments for both positions
have been exchanged in a way that allows the Editor to make a balanced decision based
on these arguments. At the end of a typical crowd reviewing process, a review report
in form of a PDF containing the annotated manuscript is obtained (Step 6). This report
and a brief summary of its content are then sent from the Crowd Review Editor to the
Editor, who receives this report in the same form a single peer review report would
usually be obtained (Step 7). Based on this report (and in some cases an additional
traditional peer review) the Editor now makes the decision regarding the manuscript,
providing the authors with a typical decision letter accepting, rejecting, or requesting
revisions to the manuscript (Step 8). As part of this decision letter the authors
receive the crowd reviewing report with all original and anonymous comments made by
the reviewers in the crowd.
Figure 3 Screenshot of a real-life Select Crowd Reviewing process
To further illustrate this process, Figure [3] depicts a screenshot from a real-life crowd reviewing process.
3
Questions We Have Received Regarding Select Crowd Reviewing
In this section, we paraphrase some of the most important questions regarding Select
Crowd Review that we have been asked or asked ourselves over the last few years and
provide you with the answers we have/should have given to these.
The peer review system we have in use at most scientific journals works fine and has
been honed over centuries – why are you trying to fix something that is not broken?
First, we must stress that we are not advocating the abolition of peer review. Select
Crowd Review is peer review, just organized differently. The traditional peer review system has a
number of very important merits that we acknowledge and that should be retained in
any system implemented for the evaluation of scientific manuscripts.[1] The most important aspect is the evaluation through peers. Who, if not highly qualified
scientists from the same discipline, should be qualified to judge if a manuscript
is scientifically sound? Only a scientist who is sufficiently educated and specialized
in a similar way can spot methodological flaws and suggest meaningful experiments
to improve a manuscript. Only the participation of a broad group of peers, i.e., ‘the’
scientific community, can combat the bias that would undoubtedly occur if such decisions
were to be made by persons outside the academic system. Another key aspect in peer
review that must be retained is the reciprocity. Highly educated scientists provide
their expertise to evaluate manuscripts because they can trust that their own manuscripts
will be reviewed accordingly.
On the other hand, one could argue that the traditional peer review system is ‘broken’
in many small ways. Anyone who has seen the ‘Reviewer 2’ jokes on social media and
recognized their own experiences in this will certainly agree that bias does exist
in the current peer review system, that some reviewers protected by the single-blind
review system are tempted to provide harsh, sometimes even demeaning comments or even
unethically hamper the work by their direct competitors.[3]
The traditional peer review system in place today, in which typically two to three
referees provide full reports on a manuscript, originates from a time in which the
manuscripts and reports were sent back and forth via paper mail and where cooperativity in the reviewing process was simply not a realistic
option. With Select Crowd Reviewing, we strive to preserve the indispensable strengths
of peer review and at the same time alleviate some of the weaknesses in the traditional
procedures by using modern tools enabled by the internet and information technology.
You mentioned the fact that sometimes contradictory recommendations result from traditional
peer review. Isn’t this desired? Or, phrased differently, does crowd review result
in a situation where you have just one viewpoint?
Of course, Select Crowd Reviewing also shows when different opinions on a certain
manuscript or aspect thereof exist. However, these do not stand beside each other
without a possibility to weigh them. Since each reviewer can see the others’ comments,
the contradictory opinions ‘clash’ during the reviewing process. This can lead to
a discussion and ultimately to a consensus opinion instead of an Editor having to
weigh-up which of two opposing recommendations to give more credit.
But doesn’t this create the possibility for ‘peer pressure’ in the reviewing process,
where one reviewer posts an opinion and then the others just agree without proper
reflection?
We do see situations where one reviewer posts an opinion and then various others agree.
The real question here is whether this is automatically a bad thing. It can in many
cases save a lot of work to the reviewers, because instead of independently and lengthily
writing the same points, they can agree to an already existing comment. It would only
be a problem, if the agreeing reviewers did agree without reflecting on this step.
We have no reason to assume that this is the case. Firstly, the reviewers are anonymous
amongst each other, such that there is no reason to agree with someone for any reason
except that you do in fact share their opinion. Secondly, we do see disagreement often
enough to conclude that there is no hesitation to enter a discussion when conflicting
opinions exist. Consider this: In traditional peer review it also happens frequently
that several reviewers reach the same conclusion/recommendation without having influenced
each other. This is actually what Editors hope for because it renders their job to
make an overall decision much simpler than highly conflicting independent reviews.
You mentioned ‘reviewer fatigue’, but does in not create disproportionately more work
if a larger number of reviewers has to review each manuscript?
Three aspects should be considered here. A reviewer contributing to the evaluation
of a manuscript through Select Crowd Reviewing does not automatically have the same
amount of work as in traditional peer review. It is possible for reviewers to log
in, give a single comment ranging from an overall recommendation to highlighting spectra
of insufficient quality, and not give any further comments on the manuscript. Through
multiple such contributions full referee reports can be obtained. This is ‘crowd intelligence’
at work. Secondly, not all members of the crowd comment on all manuscripts, they ‘self-assign’
when they are available for commenting. Thirdly, the traditional peer review system
has the tendency to focus reviewer requests on the most visible members of a given
field, simply because these are the first names that come to mind when the authors
are asked to recommend reviewers or the Editor herself/himself has to identify suitable
reviewers. At the same time many members of the scientific community that also possess
the qualification to provide review reports and are willing to contribute to the scientific
community of which they are part as authors receive little or no requests to review
manuscripts. Select Crowd Reviewing thereby has a much larger potential reviewer base
than traditional peer review.
You mentioned that you have a larger base of potential reviewers. How did you mean
that? Or phrased differently, who is part of this ominous crowd and how do you become a member of the crowd?
The members of the crowd are often highly trained assistant professors and postdoctoral
researchers, but also senior professors and chemist who have left academia to lead
industrial research laboratories. By asking such scientists to join Select Crowd Reviewing
and encouraging them to apply to become members of the crowd, we create a system that
distributes the burden of, but also honor and responsibility associated with peer
review onto a broader basis of scientists. There are several main pathways to become
a member of the crowd. The Editors at Thieme Chemistry are always looking for suitable
candidates, who they then either contact directly or via the Crowd Reviewing Editor and invite them to join Select Crowd Reviewing. Additionally,
applications to join the crowd, especially from postdoctoral researchers who wish
to pursue academic careers, are frequent and are considered based on the merits of
the applicant.
Even if the members of the crowd are well-trained chemists, isn’t the evaluation by
two or three highly specialized leaders in the field more rigorous?
The need for extreme specialization may be given in particular manuscripts when topics
at the border of the scope of Synlett are covered. However, in general, the fact that Synlett is in itself a specialized journal focusing on chemical synthesis leads to a comparable
specialization of the crowd members. We simply aim to have a crowd of suitably specialized
referees. Additionally, it should be considered that besides the qualification to
perform a detailed and qualified review the quality of the referee reports obtained
also strongly depends on the time invested by the reviewer. Here, Select Crowd Reviewing
offers an inherent advantage. Traditional peer review, by focusing the reviewing burden
on fewer individuals also reduces the time invested for reviewing a manuscript, which
can overall erode the quality of peer review reports provided by prominent leaders
in a given field. In contrast, the members of the crowd are often willing to invest
more time and end up providing much more thorough referee reports, also because double
work is avoided as explained above.
How can this ever be scalable?
Over recent years Select Crowd Review at Synlett has evolved from a handful of manuscripts over several months to the default method
with which hundreds of manuscripts are evaluated. Naturally, we had to consider how
to cope with an increasing number of manuscripts to be reviewed. As a result, we have
expanded our crowd and now use a system, where each manuscript is evaluated by a subset
of crowd members. Larger journals or journals with a broader scope and interdisciplinary
journals where not all crowd members are suitable for all manuscripts, could easily
adapt this approach to create sub-crowds by keywords describing their respective specialization.
Overall, the solution is simple, the more manuscripts have to be evaluated, the more
crowd members are required. Since, as discussed above, the overall amount of work
is not increased and it is distributed more fairly throughout the community, we can
conclude that this is more a matter of management, e.g., by the Crowd Review Editor,
and a technological matter in the sense that the databases and artificial intelligence
software to manage these increasingly larger and more diverse crowds must be available.
To this goal, Thieme constantly works with the companies involved in the management
and evaluation of manuscripts (ScholarOne and Filestage). Two more aspects should
be reiterated here: (1) with Select Crowd Reviewing we tap into a much larger refereeing
community, and (2) it should be noted that the overall amount of reviewing material
that is generated in the process, which we typically quantify in terms of number of
substantive points, is not massively more than that coming from traditional peer review.
Overall, our referees work less rather than more and can even select on which manuscripts
to put their effort.
How do you ensure appropriate ethics of the crowd?
On the one hand, we have reviewer guidelines as for traditional peer review, requiring
reviewers to disclose potential conflicts of interests and not contribute to the evaluation
of affected manuscripts. Secondly, the Crowd Review Editor has the possibility to
access the identity of each reviewer despite the anonymity during the reviewing process
itself. This allows us to probe suspected cases of ethical misconduct, although this
has never been needed. In contrast, the crowd has already uncovered cases of ethically
questionable to unacceptable behavior by authors that might have been missed in traditional
peer review, such as self-plagiarism, plagiarism, and even digitally modified analytical
data in the Supporting Information of manuscripts. Finally, and most importantly,
the Crowd is self-controlling. An inappropriate reviewer comment will simply receive
opposing comments by other members of the crowd. This renders it much more difficult
to provide a biased report than in traditional peer review.
How do you deal with bad manners in the crowd?
Again, we have the same options at hand as in traditional peer review. We have had
occasional impolite comments by reviewers. These reviewers were contacted and informed
about our expectations from reviewers. In virtually all cases the reviewers changed
their conduct after this instance. Only in a single case over the last years was a
reviewer removed from the crowd for repeatedly providing comments in an inappropriate
manner. One point of note is, however, that the referee reports obtained from Select
Crowd Reviewing are built from the compiled comments by various people and constitute
‘chat protocols’. This format instead of the flow text obtained in traditional peer
review may seem harsher to authors experiencing the format for the first time.
Until now, we have discussed potential weaknesses/challenges of Select Crowd Reviewing,
but what are the inherent advantages?
The key advantages of Select Crowd Reviewing result from resolving challenges in traditional
peer reviewing.[4] As stated above, Select Crowd Reviewing reduces an overload of certain reviewers,
by calling upon a broader reviewer base. This also renders the format more inclusive
towards otherwise underrepresented groups of scientists (with respect to the career
stage, country of origin, etc.). The interactive format provides several additional
advantages. It acts as a safeguard against biased referee reports and instead of contradicting
statements by independent reviewers a discussion is obtained as review report, which
allows for much more informed decisions on the Editor’s side. Further advantages result
from the composition of the crowd. The participation of scientists at earlier stages
of their careers provides a great opportunity for them to contribute to the scientific
community. Furthermore, the balanced composition of the crowd leads to very substantive
referee reports. Different members of the crowd tend to comment on different aspects
of a given manuscript. Some focus only on the big picture, others have a keen eye
for the quality of analytical data or put particular emphasis on the quality of the
graphical material. Finally, the use of this modern format inspired by social media
is extremely fast compared to traditional peer review. Typically, a manuscript is
online for our crowd for only two to four days. This short period is usually enough
to obtain a sufficient amount of quality feedback such that by now the administrative
parts of the workflow shown in Figure [2] account for the larger part of the time required to evaluate a manuscript.
Doesn’t this cause much extra work for the authors?
It depends on the reference. In our experience Select Crowd Reviewing can produce
very detailed comments. Instead of a comment such as ‘the language must be improved’,
dozens of specific comments may be given to help the authors improve the writing.
Reviewers may comment on spectra of insufficient quality and make specific suggestions
on further experiments, etc. However, all of this is in principle expected from a
very good traditional peer review report. Unfortunately, we have grown used to traditional
review reports that do not live up to these standards and sometimes just contain a
general recommendation to the Editor, but no specific comments that could be used
to improve the manuscript. In that sense, the reports from Select Crowd Reviewing
will sometimes result in more work for the authors than traditional peer review. However,
this work is certainly justified if it results in better manuscripts and better science.
4
Conclusion
In this Account we have reported how the idea and process of Select Crowd Reviewing
originated, have detailed the workflow used at Synlett/SynOpen and have attempted to address typical questions we receive regarding this modern
implementation of peer review. While we are very pleased with how Select Crowd Reviewing
has worked over the past few years, this comparably young technology is still in a
dynamic development. We therefore always welcome questions and suggestions and invite
readers interested in experiencing Select Crowd Reviewing themselves to contact us
and join the crowd.
Testimonials from Crowd Members and Editors
‘Due to the diverse members of the crowd, CR allows indeed to obtain more opinions
on manuscripts from a broader basis of reviewers. This is absolutely true and also
prevents referees from following an agenda and weakens extreme opinions (in one direction
or the other). […] For me it is a lot of fun in principal. As a young researcher,
I am not yet overwhelmed by peer-review requests. At the same time, I can give something
back to the community, I can participate actively and help to shape the journal, as
I work on many of the submitted manuscripts. But especially, I also get insights into
the comments of other referees and enjoy reading about, what I might have overlooked
while reading! It helps me to grow as both scientist and referee. It is a great concept
and I feel it already works very, very well! Congratulations!’
Dr. Bernd M. Schmidt, Assistant Professor, Heinrich-Heine-Universität Düsseldorf
‘The crowd review program (CR) is a great improvement for the peer-review process where
you could potentially see a fast-paced publication. It certainly has several advantages
over conventional peer review and takes significantly much lesser time for both the
manuscript evaluation as well as for publication (few weeks). In addition, crowd review
completely nullifies the conflicts of interest and brings a precise conclusion to
the fate of the manuscript, which you can occasionally miss in conventional peer review.
[…] I strongly believe that the reviewers also benefit from the scientific discussions
during the review process and share a joint responsibility to properly evaluate the
manuscript. A reviewer does not enjoy these important returns from conventional peer
review.’ Dr. Nagaraju Miriyala, Research Associate, Medicinal Chemistry Facility, J. G. Brown
Cancer Center
‘As the Editor for Account and Synpacts articles, I have been gratified by the increasing
number of authors opting for crowd review (more than 50%). The manuscripts are invited,
hence the aim is to get them published, the reviewing process serving to flag problems
with content and to improve the presentation of the subject matter. The latter gets
short thrift in traditional peer review, which (rightly so) tends to focus on the
intellectual gist of the manuscript. The crowd has been exceptionally good at addressing
both aspects, providing valuable feedback on the scientific content of the manuscripts,
but at the same time pointing out stylistic and grammatical errors, inconsistencies,
format problems, and ways to improve the reading of the narrative. Only occasionally
do I have to supplement the comments of the crowd by those of additional experts.
Needless to say, manuscript processing times have been reduced dramatically.’
Prof. Dr. K. Peter C. Vollhardt, University of California at Berkeley
