The problems of peer review

Comments Subscribe by RSS

1. Nick Dickens 03 February 2010 at 09:39 AM

   permalink report this comment

   I completely agree, double blind would be ideal - in a perfect world I would say that an open system would be great, it is much easier to take the hard line and slag off someone's paper if they don't know who you are...but the open system would also lead to the "they rejected my paper so I'm going to reject theirs" attitude (not that we scientists are all petty...!). I have personally witnessed a not-so-ethical PI come rushing into the lab after seeing a paper that was direct competition, asking a researcher to look at a specific gene and then sitting on the paper being reviewed until we could get our own paper out and published. Needless to say, I couldn't work for this particular PI any more after that.

2. Blanka Sengerová 04 February 2010 at 06:32 PM

   permalink report this comment

   On a slightly related topic, I am a member of the Voice of Young Science (VoYS) network, set up by Sense about Science. Via their mailing list I have received details about a workshop they are running about peer review, whose main aim is to discuss how and whether the peer review process is well set up to catch fraud in science. If anyone is interested in attending, here goes: Research: publishing it, reviewing it and talking about it publicly Sense About Science will be holding a workshop on Peer Review at the University of Sussex on Friday 5th March. This afternoon event is free and for early career researchers in all sciences, engineering and medicine (PhD students, post-docs or equivalent in first job). During the workshop we discuss the process of peer review in journal publishing and explore the criticisms of the peer review process. What does peer review do for science? Does it detect fraud and misconduct? Will it illuminate good ideas or shut them down? Please find further information at www.senseaboutscience.org/PDF/sussexpeerreview10.pdf. These workshops are very popular and places are limited. To apply send a CV and covering letter explaining your reasons for applying and stating any affiliations you hold to Julia Wilson jwilson@senseaboutscience.org. Closing date for applications is Friday 19th February

3. Tennie Videler 05 February 2010 at 10:27 AM

   permalink report this comment

   The following is very definitely my personal opinion and not Vitae's (and to be honest, all my posts here should be read as being my personal opinion!!!) I think the fundamental problem with peer review is that the reviewers are just humans. So a tiny minority will be bad (as in Nick’s example and I have definitely heard accounts of this in my time) but most aren’t. But they will form judgments based on prejudices even if they don’t mean to. A gender bias has been shown towards authors. I have seen in action that people go into reading an article of an author they respect far more positively than authors unknown or previously disagreed with. Double blind would go some way to eliminate this of course. When I have had to review, I have found it hard to not be negatively influenced by poor language and especially sloppiness (not labeling figures, referring to the wrong figure, mistakes with references). There is no way round this of course but there is a lesson to learn: be as thorough as humanly possible when submitting a paper for peer review yourself and make it as easy as possible for the reviewers to have a positive stance from the start….

4. Simon Smith 05 February 2010 at 05:04 PM

   permalink report this comment

   I'm surprised to hear that the journals you have written for do not use double-blind peer review! In my experience as both a reviewer and a writer in the social sciences, it is the norm (to the extent of being asked to anonymise all references to my own publications when submitting an article for consideration). Peer review has taken a bit of a battering recently, with the Lancet forced to eat another dose of humble pie after the MMR case resurfaced in the media, but I agree with the British Academy (see below) that it is the least worst solution the academic community has found for identifying quality. And don't forget that it also has an important role in improving quality - I have genuinely felt gratitude to my anonymous reviewers on many occasions for the contributions they have made to the development of my ideas. From the conclusion to a 2007 report for the British Academy on peer review in the humanities and social sciences: "Peer review has its critics, who allege that it is costly, time-consuming and biased against innovation. None of these criticisms is entirely without force, but the Working Group concluded that there were no better alternatives and that often the criticisms were directed at deficiencies of practice rather than the principle of peer review. Peer review is both a mechanism of selection - only those grants and publications are favoured that are positively judged by peers - and a force making for enhancement. Work is better as a result of peer review. Importantly, it retains widespread and deep support among members of the academic community." http://www.britac.ac.uk/reports/peer-review/execsum.cfm

5. Sarah Davies 09 February 2010 at 09:23 PM

   permalink report this comment

   Just to add my two penny's worth: many of the social science journals I submit to and review for use double blind reviewing. The natural sciences seem to be lagging behind...

6. Joanne Oates 10 February 2010 at 12:57 PM

   permalink report this comment

   I'm very interested to here that the social sciences use double blind reviewing. I wonder why this is not the case in biosciences...?

7. Hannah Dee 10 February 2010 at 01:55 PM

   permalink report this comment

   I'd say in computer vision, it's about 50/50. Strangely, more conferences than journals are double-blind.

8. Tennie Videler 11 February 2010 at 11:19 AM

   permalink report this comment

   How fascinating that there are such discipline specific differences. The situation you describe Hannah seems even odder, to have a mixture... Does anyone have knowledge or views on where 'invited articles' fit into all this?

9. Chris Thomson 11 February 2010 at 01:33 PM

   permalink report this comment

   One problem in some parts of natural sciences and computing is that it is very easy to tell were the research was done (down to the research group level at least) due to the nature of the research (technique,equipment and subject) and perhaps the cliquishness of research groupings where most people are familiar with what others are doing. Because of this double blind reviewing only gets you so far along, even when it exists.

10. Oliver Harris 15 February 2010 at 09:01 PM

    permalink report this comment

    I'd have thought Chris's point would apply, at least at times, in most disciplines: we all move in tight little circles of our specialisms, many papers submitted for publication will have had previous outings at conferences etc, and it must be pretty obvious much of the time who's written them. @ Simon: I'm fascinated by the concept of 'being asked to anonymise all references to my own publications when submitting an article for consideration', which I've never come across. How does that work, then? Do you actually remove the references entirely, or do you literally anonymise them? If (a), then doesn't it just lead to comments along the lines of 'The author has failed to refer to the important study ...'; and if (b), i.e. you remove your name, but leave the titles and places of publication, then isn't it the easiest thing in the world for the referee to identify you? I agree that referees' reports can often be constructive, and make for a better paper. However, I've also been very annoyed at times by comments such as 'paragraphs X, Y and Z are irrelevant', or 'something could have been said on A, B and C' - to which I sometimes want to reply, 'OK, maybe that's the paper you'd have written, but you've actually missed the point of the paper I did write'. The anonymous referee's report always seems to be presented as a judgement from on high, and I'm never really sure how much room there is for negotiation. If I do disagree with the referee's opinion, I tend to look for some kind of compromise (and explain what I've done to the editor), and I haven't run into serious difficulties yet - but there have been occasions when, once I've seen the piece in print, I've regretted the compromise, and wished I'd stood my ground. Any comments?

11. Simon Smith 19 February 2010 at 04:43 PM

    permalink report this comment

    Hi Oliver, It's B in the cases I'm thinking of - the convention is to replace the name(s) of author(s) with XXXX. So you're right, an inquisitive referee could easily find out who I am, but peer review requires trust to be exercised by both parties. It's a system that relies on some basic ethics after all! I can empathise with you in your frustrations with some reviewers' comments. They do often appear not to have read the paper closely enough. In that case, there ought to be room for dialogue, and I've found there can be, but you do take a risk in holding your ground, and sometimes it's easier to go along with their recommendations when you have an opportunity to re-write a paper. If you defend your position, you're putting a lot of trust in the editor, who becomes the arbiter in those situations. This is not something I've ever done, but it occurs to me that a informal enquiry to the editor prior to submitting your revised version, to see what they think about the scope for negotiation, might be worth a try?

12. Simon Smith 19 February 2010 at 08:22 PM

    permalink report this comment

    Hi Oliver, It's B in the cases I'm thinking of - the convention is to replace the name(s) of author(s) with XXXX. So you're right, an inquisitive referee could easily find out who I am, but peer review requires trust to be exercised by both parties. It's a system that relies on some basic ethics after all! I can empathise with you in your frustrations with some reviewers' comments. They do often appear not to have read the paper closely enough. In that case, there ought to be room for dialogue, and I've found there can be, but you do take a risk in holding your ground, and sometimes it's easier to go along with their recommendations when you have an opportunity to re-write a paper. If you defend your position, you're putting a lot of trust in the editor, who becomes the arbiter in those situations. This is not something I've ever done, but it occurs to me that a informal enquiry to the editor prior to submitting your revised version, to see what they think about the scope for negotiation, might be worth a try?

13. Oliver Harris 23 February 2010 at 09:28 PM

    permalink report this comment

    Sorry, Simon, but the XXXX convention still sounds daft and unworkable to me! It's flagging up the fact that you (the anonymous author of the paper under review) are emphatically also the author of the easily-identifiable anonymised publications in the bibliography - whereas if you'd left them untouched, you could still be another researcher entirely. However, not really my problem! I have to say that in my field (humanities) there can be a fairly relaxed attitude towards refereeing. I've written for respectable (though not top ranking) journals where it barely exists at all: the editor accepts the piece in principle on his/her own judgement, then passes it to a couple of (named) colleagues for comment. Even where refereeing is more formal, it's by no means unusual for referees to waive their right to anonymity (normally only if they're feeling positive about a paper!), which opens the ground to constructive dialogue at the fine-tuning stage.

14. Sarah Davies 23 February 2010 at 09:51 PM

    permalink report this comment

    Actually when I've anonymised my publications I've simply cited in the text as (Author) and excised the references from the bibliography altogether. (Sadly no-one has ever responded by saying that I've ignored my own, very important work...) In a small field it's still pretty easy to work out who's writing, but if you're publishing somewhere more general (a big, whole discipline journal like Sociology, for example) it's pretty effective. I agree, excising your name but leaving the piece's title and date is seriously underestimating the power of Google...

15. Hannah Dee 24 February 2010 at 09:13 AM

    permalink report this comment

    Check out Section 1.3 of this for details of the way we're advised to do double-blind review in computer vision - in short, it's a case of "refer to your own work in the third person", but the longer version is worth a read: http://www.cs.ucl.ac.uk/staff/s.prince/bmvc2009/bmvc_review.pdf