Identifying pseudo-science, and policy towards pseudo-scientific articles that have already been accepted
Santosh Patnaik
Friday, 10 August 2007 21:04 UTC
Though Nature Precedings screens submissions for pseudo-scientific content, it is possible for such a submission to get through. An example might be this article on Nature Precedings: this website suggests that the authors are supporting “creationism/intelligent design.”
This leads one to wonder how pseudo-science is identified, and what the policy is towards accepted articles that are later identified as pseudo-scientific.
-
Replies
Jump to resultsResults
-
Nature Precedings does not accept pseudo-science. Spotting this and other inappropriate content is the job of our curators. In the month since launch they have already filtered out many such examples. In this particular case, the curator handling the contribution was concerned about the nature of the content and so consulted a senior journal editor with very considerable expertise in this area. In their opinion, it is not pseudo-science and is worth posting for community comment.
I am not personally qualified to assess this paper, but as the person responsible for Nature Precedings, I’m happy with the process that took place. It’s not completely fool-proof (because no system is), and I’m sure we’ll make the odd mistake. But as far as I’m aware no such mistake was made here, and I think our system is about as good as you can get for an open preprint server.
Of course, we’re not complacent and we’re always on the lookout for ways to improve things, so if you have a suggestion then please make them in this forum or drop an email to precedings-at-nature-dot-com.
-
If someone finds problems with a preprint, s/he should criticise the publication rather than the publisher of the server. Of course publishers should implement certain quality control but I believe readers should also take some responsibility to maintain the quality of preprints to make the system work.
-
Identifying pseudo-science is a difficult task since there are no clear criteria for doing so. Dictionaries and encyclopedias suggest that science can be identified by its use of the scientific method (observation, hypothesis, prediction, establishing controls), however many of the manuscripts being published today as science articles do not strictly adhere to this. For example, many of the reviews articles, such as those found in the Nature Reviews series, do not strictly follow the scientific method (Example), yet I would argue that these papers are still worthy of scientists’ interest (and no one is claiming that NPG should not publish them).
This “demarcation” problem has been addressed by a number of philosophers of science. Karl Popper stands out as one of the first to try propose a comprehensive set of criteria for distinguishing science from pseudo-science based on what he called “falsifiability” (the idea that theory is scientific only if it makes “risky” predictions, which could conceivably be false). Popper’s program was later criticized by Imre Lakatos and Thomas Kuhn who each proposed their own criteria. Other philosophers who have written on the problem include Carl Hempel, Paul Feyeraband, and Larry Laudan. Personally, Kuhn’s account appeals to me because of his emphasis on the practice of scientists. (Note: Nature Precedings does not explicitly attempt to use any of the theories above in making curatorial decisions.)
The Supreme Court has also weighed in on the issue in Daubert v. Merrell Dow (pdf), concluding that peer review is relevant for distinguishing science from pseudoscience.
Nature Precedings generally tries to include documents that will be of interest to the scientific community and that will help scientists with their own research (though the term ‘scientific community’ is probably also open to debate). To that end, we do consult other scientists both within and external to NPG. Questions about the value of certain contributions to the scientific community are always welcome.
I also find the science/pseudoscience question fascinating and would love to hear other people’s thoughts.
-
Hilary,
I think that a first prerequisite for something to qualify as science versus pseudoscience is a the presence of coherent, understandable argument(s). I tried to read the paper, and while I am not an expert in the field, I found the grammar and general quality of writing made the paper impossible for me to follow. I am amazed that your curators were able to assess its validity at all, and I doubt that anyone will have the time to critically assess such a poorly-written paper and post review-type comments.
So amongst other things, I guess this paper fails in the area of being falsifiable, since you can’t disprove incoherent language.
I do, however, disagree with what the above blogger seems to be implying, that science is not valid if it backs up non-Darwinian models. If the arguments are rational, and the evidence is valid, then it’s good science. But in the case of this paper, lack of clarity would seem to make this impossible to assess.
So maybe curators at Nature Precedings need to be looking for well-stated arugments, solid reasoning and some mention of evidence.
I’m almost hoping this paper is a bad joke or a Nature Precedings experiment?
-
In an atmosphere were we are trying to establish a new medium for publication there are always going to be those who will exploit its openness. However, since this site is really a marketplace for ideas we should not dwell on the occasional misrepresentation that manages to make it through. Just as those who frequently read a particular blog learn how to spot the “trolls”, so to will the curators of this site begin to recognize the more subtle attempts by people trying to publish a piece that may have a specific agenda.
There is a larger issue here than the occasional pseudo-science posting that will make it onto this site and that issue has to deal with why isn’t the scientific community at large moving more rapidly to the open access form of publication. If we acknowledge that there will always be pseudo-science in society than we as researchers are somewhat called to task to ensure that the society at large has access to “real” science.
-
Anthony—thanks for your comment. I tend to think that one can indeed criticize papers for being incoherent by pointing out inconsistencies in the argument or places where the paper is self-contradictory. I do think, though, that you are correct in suggesting that most people won’t comment on poorly written papers. We hope to see the converse: that people will become involved in discussing papers that look promising or discuss interesting material.
Anthony and Craig—are there other features or tools that would make it easier for researchers to highlight papers that make a meaningful and impressive contribution to the scientific community?
-
Hilary-
I suppose one way to do this would be to allow readers to rate the paper’s overall relevance, as well as in specific areas, such as technical validity, novelty, relevance, general interest etc., on a 1-10 scale or with a simple thumbs up or down. Pooled results could be used to rank papers.A more intricate system would also somehow rank the reviewers themselves (for example according to their citation rate) and weight each reviewer’s evaluation of the paper according to their own ranking, to obtain a pooled ranking that reflects the credibility of each reviewer.
Such a system would not be perfect, so one could imagine giving users flexibility by allowing them to sort the papers listed on the website by customizable criteria as per their own interest. For example, one user might wish to list papers in order of technical validity, whereas another might be more interested in papers rated as highly novel. Within these rankings, some might also be most interested in browsing those papers most highly ranked by high-ranking reviewers, whereas others might find this less important. Perhaps one would be interested in seeing which papers were most highly though of by professors, versus institute lab heads, versus grad students, versus undergrads, versus members of the general public….Rankings according to the reviewer’s field and total number of publications would also be interesting, and so on and so on…
-
BTW Just noticed it is possible already to vote for a submission on Precedings. Good idea!
-
One way that helps to identify pseudoscience is the frame of reference supplied. If the work is building on a previous body of work, then it seems to me to have passed one hurdle towards being considered seriously. If, however, it only cites the author, or only cites anecdotal rather than formal publications, or doesn’t cite anything at all, I’d begin to be suspicious.
At Nature we receive thousands of non-scientific or pseudo-scientific submissions. Many of them are by individuals who are working alone, so they don’t have colleagues or employers or a community with whom to test, discuss and refine their arguments. They often believe that they are uniquely inspired and that the world will gasp at their insights, and hence do not need to go through any process of “moderation” first.
So, there are usually some pretty good ways to be alerted to possbible pseudoscience. I think that Hilary’s description in her first reply of how Precedings handled this particular submission is spot-on. And, from subsequent discussion, it seems as if this particular contribution (which I have not read myself) is “enjoying” the verdict of “the wisdom of the crowds”. -
Maxine’s comment reiterates a number of points made by Robert Park in his essay The Seven Warning Signs of Bogus Science
These include:
- Direct contact with the media (prior to peer-review)
- Claims that “the man” (or other powerful establishment) is trying to suppress the research
- Focusing on a “barely perceptible” event (perhaps without statistical significance)
- Evidence is anecdotal (and not from a randomized double-blind experiment)
- Evidence is not necessary because of “folk wisdom”
- The discoverer has worked in isolation
- Introduction of new laws of nature
The original article is fairly short and worth a read. I would add to the list that a lack of citations or references to research already accepted by the scientific community is definitely a warning flag.
Results
-