Man, I am SO far behind on my blog posts; I have 5-6 topics swirling in my head, half-written, half-researched, or in some alternative form of partial existence. But a recent comment by David Featherstone on the previous entry warranted a new thread to see if we can spur on additional discussion. At issue is competition and how it affects the way people do science, and here is David’s fear:
Which leads us into a discussion of Web 2.0 or Publishing 2.0 or whatever the heck the term is for the recently popular idea that science should be published fast online and decorated with ongoing reader feedback. Carrying this model to a logical extension, will I be able to do an experiment in the morning and have it online by the end of the day? What will that do to the careful pace of science? Will publishing turn into a continuous frantic scramble where priority in top journals/websites can be lost by getting a good night’s sleep before submission, reflecting on the data a couple days, or getting feedback from colleagues?
Well let’s hear it. Is Web 2.0 technology actually harmful to the future of careful scientific research?
For me, I believe that there will always be people out there who view science as their occupation as opposed to their passion. There is absolutely nothing wrong with this. It’s a fun, flexible job that can help pay the bills (well, as long as you don’t live in NYC, SF, LA…okay fine, the pay sucks for the training required; a topic for another time…). These researchers may indeed follow a more frantic careless pace, only worried about the short-term personal gains as opposed to the long-term value of the discovery. Therefore, as long as the results are mostly correct, why not get it out there first to get the credit and, hopefully, the advancement that can come with it?? This could very well produce a poisonous, competitive environment that could cause the rigor of science to suffer, to which David alludes.
However, I feel that the vast majority of researchers actually have a passion for scientific research, and it just so happens that being a scientist is also their occupation. These researchers will be loathe to have their name attached to anything that is not well-described, understood, or potentially erroneous. The pace of research may indeed pick up with the introduction of web-based publishing (or blog-based, or other such means), but not because of a worry of competition, but because information dissemination will be vastly improved and quickened. With the use of Web 2.0 technology, researchers will not have to wait months to see important results if they happen to miss their big subfield-centric meeting. In fact, look already at the popularity of “AOP” (advance online publication) and how that has improved the turn-around time for getting results out. And, as always, pre-print servers like Nature Precedings still offer a way to get a full manuscript out in the public at whatever time the researchers feel comfortable tipping their hand.
Perhaps I am being naïve, but I just can’t accept that scientists will short-circuit their own personal comfort zone of feeling confident in results on a broad, wide-scale, pathological, and recurring manner. It would only take one marginally significant foul-up, or the embarrassment of posting inaccurate data, to eliminate the bad habit of pulling the trigger too soon. Just like single-trial fear conditioning in rodents, the great 80s band Great White said it best – Once bitten, twice shy.
Last updated:
Wednesday, 24 Sep
2008 - 21:12 UTC
If there was a lot of competition to publish online early, then our norms could change, so that “publish early and worry about it being right afterwards” was Officially Good. Making mistakes would then be part of the process, and nobody would mind too much because we would all know that the thinking and correction would follow.
I suspect we won’t go that far: , but my point is that there is also the potential for social evolution, and what we may think of as bad now may be seen as good (or not so bad) later. We will accommodate the downsides somehow: either by abandoning the new technology, or by changing the way we work and think to work round them.
I’m not sure I agree with your characterization of two types of scientists—it’s certainly more of a gradient than it is two buckets. Some of the most brilliant, best scientists that I know are also the most competitive and most cutthroat. Caring about discovery and quality of work are not necessarily contradictory with ambition and ladder-climbing.
We are indeed seeing faster turnaround times from the writing of a paper to the release of that material to the public (arXiv as the obvious example). The question is whether there’s a difference in the amount of experimentation and time put in to gathering the results before the paper is written. And I’m not aware of any trend in that direction. I think it’s important to take some time to think through and fully understand your results before you put them out to share with others. As I’ve written elsewhere credit is given to the person who makes the intellectual leap, not necessarily the person who collected the data that enabled that leap.
I’m not sure I agree with your characterization of two types of scientists—it’s certainly more of a gradient than it is two buckets. Some of the most brilliant, best scientists that I know are also the most competitive and most cutthroat. Caring about discovery and quality of work are not necessarily contradictory with ambition and ladder-climbing.
Well, I never said that there was a contradiction between competitiveness and doing good science. I simply stated that there are some scientists out there who simply “punch the clock” and thus may not be too concerned with shoddy data, while the vast majority of researchers actually see science as their passion. The former group is small because typically, once individuals recognize their lack of passion, they usually look for another job. Oh, and of course its a gradient. I thought that was a given.
As I’ve written elsewhere credit is given to the person who makes the intellectual leap, not necessarily the person who collected the data that enabled that leap.
Congratulations on choosing one of the most controversial examples in the history of scientific discovery to make your point. I’m sure that plenty will debate you as to the value of Franklin’s contribution to Watson and Crick’s “discovery”. Since the Nobel prize is never given posthumously, Franklin was not considered for the prize, even though her colleague, Wilkins (who actually presented W&C her data behind her back) did receive a portion. Do you really believe that had she been alive, she would not have also been a part of that group receiving the Nobel?
Regardless, we are in agreement that most scientists will definitively take care before disseminating results. The point of the post was that the face of research communication could change, and as Bob alluded to, may produce standards that currently seem completely foreign to us.
Maybe we can learn from software developers on how to approach scientific discovery. Agile Software Development has many of the features of the Web 2.0 approach to science that you describe, e.g.
Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
What about conference presentations? These already provide a way of releasing not fully-worked out results, or so I understood. Perhaps there are occasional abuses, but mostly people know that if they get it badly wrong that’s going to impact on their reputation. I would think the same will apply to new web-based publishing practices.
It is a good point Frank, but I think that one can always be a little looser with the spoken word since it is rare to have conference presentations taped and archived for the long-term (some conferences offer very short-term access to recordings of talks only to attendees). This lack of enduring public record always provides the researcher with “plausible deniability” later.
Once published on the web, there is an explicit record of the data, which is a good thing if you would like to claim intellectual property, but a bad thing if it is wrong.
I agree with Noah – the transient and unrecorded nature of conference presentations allows for a bit more freedom of expression. Plus, since you rarely have time to present full methodological details, there is a some protection from copycat scientists and the presenter may therefore be more willing to share new findings.
I still don’t get this Web2.0 motivation to post results online asap. I take the point that advanced online publication has speeded up the release of new data but that route still allows for peer review and therefore requires internal and external quality control.
There’s another aspect. My work has certainly benefitted from referees’ comments before going public – and then I get the credit for the improved publication. (Of course, I provide a reciprocal service when reviewing others’ manuscripts!) Via Web2.0, my mistakes/oversights would remain—for all time—and it would be others who, perhaps rightly, would be seen point out to the world the errors of my ways. I’m not sure how keen I am to go down that route… Is that just cowardly?
I think there is a culture of more rapid publication in the physical sciences than is found with the life sciences. arXiv has been running for a long time as a preprint server. A number of journals pride themselves on rapid publication too. This culture has led to high profile errors (or even fraud). Even though in this case the results were peer reviewed. Peer review is a check but not necessaraly a sufficient check.
Speaking of peer review.. this is of course getting more and more tricky, with more and more people declining requests to review. So there’s a quality control issue with ‘traditional publication’ as well!
Why are less people prepared to review? The general impression is that most are too busy due to administrative workload, funding situations, etc. You’re not getting paid to review, and people don’t just do if for the ‘honour’ of being asked anymore either. There are probably also more publications that need review these days (Martin, I’m guessing that you have a figure or a link demonstrating that somewhere).
Soo.. would an increasing amount of work published online dilute the whole process even more?
Stephen said: Via Web2.0, my mistakes/oversights would remain—for all time—and it would be others who, perhaps rightly, would be seen point out to the world the errors of my ways
I think this is the most important point. Yes you can play fast and loose and it may be easier to do that with rapid online release, but there’s a risk to doing it and you may end up with a dis-reputation.
In terms of peer review I think the reason people are less prepared to do it is two-fold. One is the time, and the other is the lack of any tangible reward for it, particulary in a world where everything is costed. We’ve been around this loop before but in many ways the amazing thing is that anyone is prepared to do peer review at all in a world where everything depends on getting tangible pieces of work in the right places.
In terms of the raw data/publication/Web2.0 thing some of us do do this for a variety of fairly complicated reasons. But David Crotty’s recent post on digital intimacy in many ways hits the nail on the head.
No-one really looks at these things except the people who are directly interested. The vast majority of those are people directly involved in the work. By putting stuff up our major aim is to enhance communication between people in the same lab. Its just we choose to leave the door open in case someone with some interesting ideas pops in.
Sorry, just one followup re: mistakes being on the record for all time. Ummm aren’t they already? Make a mistake in a published paper and that is on the record for all time. Probably end up being highly cited as well…
It’s funny though, the strength of science surely lies in the process by which ideas are subject to community scrutiny – i.e. communication lies at the heart of the scientific process. Yet, rather than communicate and refine ideas, we prefer to let our behaviour be defined by the fear that we might potentially be embarrassed in the future by something we might have not got completely right.
Actually in a world in which everything is recorded you may as well get used to that anyway.
People peer-review because they regard it as a service to the community they are in. Everyone (with a few exceptions) does it, and everyone knows that their own mss will benefit from the process when they are authors. Peer review is a great training mechanism for newer members of a group, in teaching young scientists how to be objectively and constructively critical, and how to write and construct manuscripts themselves.
As Stephen correctly (in my opinion) states, peer-review inevitably improves a manuscript. Editors at Nature journals with many years’ experience of the peer-review process will certainly strongly agree with that view. You would not believe the unsupported claims that get submitted to journals.
Peer-reviewers and authors are all the same people. Those who complain about the peer-review process are essentially complaining about themselves!
More details and links to various Nature journal editorials can be found at this web page
Journals should do more to reward peer-reviewers in my opinion, as peer-review is not only a valuable process but an essential one. The scientific community itself would do well to incorporate peer-reviewing, along with many other “informal” activities (informal in the sense of not being included in current metrics) when it comes to individual assessments (for grants, tenure, new positions).
In sort-of response to Brian, the peer-review system was not set up to detect clever and deliberate fraud. It assumes everyone is honest. Most papers that get submitted to Nature and eventually published are toned-down in their claims as a result of peer-reviewers questioning over-enthusiastic claims and interpretations, but this is not the same as detecting fraud.
There are certain frauds that are easier to detect than others, eg “photoshopping” figures – journals including the Nature journals now have checks in place for this – and plagiarism – software is becoming available so that some (but not all) types of plagiarism can be spotted.
However, a deliberate and intelligent faking of results is hard for referees or anyone to spot – until someone independent attempts to repeat the work. In the context of the discussion to this post, this is one way in which pre-publication sharing of results and data does help (whether on the web or at conferences, or in private between collaborators). However, of course, pre-“official” publication sharing can have other problems (eg scooping, reputation issues) which have been discussed above.
I think its important to distinguish between the standards of peer review at Nature (and other top tier journals) and many (but not all!) other journals.
But I agree with Maxine, there needs to be some more tangible (not necessarily monetary) reward. But I think this idea of ‘racking up credit’ won’t fly. There is no benefit in a head of department appointing a member of staff who does more reviews than others – in fact there is probably a real disadvantage at first order (time, money), and the question is how much does the second order (reputation) benefit balance that.
I seriously wonder whether peer review in its current form for the literature in its current form is affordable. And if not, what do we do about it?
This entire discussion regarding the strains of high volume on peer review is at the crux of why the SfN proposed and implemented the Neuroscience Peer Review Consortium, discussed here.
Although it is taking some time to take-off, and does increase the administrative load for the journals involved, the potential that this entity has to relieve some of the stress on an over-burdened review system far outweighs the drawbacks. Just FYI, Nature is also a member by proxy since NPG has long had its own manuscript/review transfer system in place amongst its family of journals.
The idea is that the more relief the best reviewers can receive in the form of fewer requests will leave them fresh enough to be able to review manuscripts when needed. Since we are all part of the same academic family, why should perfectly good reviews go to waste if the study is simply deemed editorially unsound for the reviewing journal?
Definitely agree that federating peer review is a good way forward for reducing the burden. There is a risk though, given the way that peer review is tied up with journal ‘brand’ that, that this may have negative impact both on top tier journals (who end up with a much greater burden – this is effectively a one way subsidy from higher to lower ranked journals) and lower tier journals (“we publish all the stuff that Journal X rejects!”)
Was also looking at David Featherstone’s comments in the original post. One thing that struck me was whether when it was much harder to modify figures or tables the demands of referees were easier to satisfy? So the burden of reformatting has become much higher? That’s something that could probably actually be tested I guess.
Cameron said: “Was also looking at David Featherstone’s comments in the original post. One thing that struck me was whether when it was much harder to modify figures or tables the demands of referees were easier to satisfy? So the burden of reformatting has become much higher?”
I don’t think reviewers were any easier to satisfy when the mechanics of manuscript revisions were harder. But my impression is that editors were. Nowadays, it seems like I deal much less with editors who actually edit, and more with editors who consider themselves science police. That said, I always have mixed feelings when editors start messing with word choices I agonized over, or making odd and seemingly arbitrary formatting changes. And like others have pointed out, we actually do need some science police to protect the integrity and reputation of the scientific record.
What gets me is the increasing burden of ‘supplemental information’.
Five years ago, before supplemental information really took off, papers were judged at face value. They were either good enough or not. If your paper was not good enough for publication, you generally had to substitute good experiments for bad experiments or resign yourself to the fact that your conclusions were of limited interest and submit elsewhere. Very straightforward.
Now, with supplemental info, it seems like there is no ‘cap’ on what reviewers or editors can expect you do do, and elegance is sacrificed for quantity. Authors no longer try to do a better experiment, they just throw in a bunch more mediocre half-assed crap to marginally bolster their point of view and overwhelm the reviewers with data. Who reads supplemental data anyway? Sometimes detailed supplemental methods are useful, but most of the time supplemental info is crap that doesn’t deserve to be published anyway. Perhaps other reviewers can attest to the experience of criticizing a paper for poor methodology, then having it come back for re-review with no fixes to the original manuscript except a bunch of crappy supplemental stuff and a long letter explaining how they’ve addressed all the reviewer comments. That’s bogus, but as a reviewer it’s really hard to hold the line and say a bigger pile of crap is still crap, because you feel bad about all the work that went into something. So supplemental data makes more work for reviewers, and degrades the need for concise elegance that should be a hallmark of science published in top journals. It also leads to publication of crappy ‘factory science’. I hate to pick on a technique that I think is actually really cool, but why are ten thousand crappy yeast two hybrid experiments publishable in Nature, Science, or Cell when the same journals would reasonably reject a paper wherein the conclusions rested solely on fewer applications of the same technique? As I said above: Crap is still crap, no matter how big you pile it.
So, getting back to Noah’s original question: Is Web 2.0 a good idea? My answer is: Yes if communication is speeded with no degradation in quality. Unfortunately, I suspect that Web 2.0 really means there will just be more crap for busy scientists to pore through sifting for rarer and rarer gems. When it comes to information, less can be more.
Which I why I’ll shut up now. I have a grant proposal, a manuscript, and a tenure package that I should be reviewing right now instead of screwing around on the web. I hope the people in my lab are hard at work.
I would like to take another quote from the Agile Software Manifesto mentioned in my previous comment:
Simplicity—the art of maximizing the amount of work not done—is essential.
A good Web 2.0 tool for scientists should do exactly that. Instead of more crap for busy scientists to pore through it should require less of our time. The process of manuscript writing, submitting and reviewing has a lot of potential for simplification.
Martin, maximizing the amount of work not done (while also getting the maximum value out of the work that is done) sounds very attractive, but it would make me worry that I’m overlooking something essential.. you’d also have to rely pretty heavily on software to pre-digest input exactly the way you need it to. I guess when your interests shift, you’d still be left to trawl the internet?
Steffi, just to give you one example of what I was meaning with simplicity. Why does every journal want a different formatting for the references in your submitted paper? Wouldn’t it be much smarter to have the references in one standard format (probably XML), and every journal can then decide how to render them.
Sounds great (I would love not to have to fix the formatting according to journal style), but I wonder how that would be achieved. Just think how many different publishers there are in any given field – you’d have to get everyone to the table to agree on this.
I’m all for simplifying, by the way, just in case that didn’t come through before!