Tagged citations
Thomas Kluyver
Monday, 23 February 2009 16:26 UTC
Another brainstorming post—please forgive me if someone’s already suggested this.
What would the world of scientific publishing look like if citations were habitually tagged, to describe how the new article relates to the cited one? I envisage tags like “used to back up specific claim”, “evidence supporting”, “evidence against”, “technical criticism”…
The potential advantages:
- More advanced citation metrics—e.g. you could do “number of papers that have used this to back up a claim” or “evidence supporting minus evidence against”, and so on
- Quick ways to get a rough impression of the feeling surrounding the paper, from these sort of metrics: is it important, well accepted science, a controversial idea, or something that is generally felt to be wrong?
- Quick ways to follow reference links forwards and backwards. E.g. if you want to cite something that backs up an idea, you can select papers citing it with “evidence supporting”, or if you want to see the main papers a paper is responding to, you can exclude “used to back up a specific claim.”
- Post-publication peer review: Nature’s open peer review trial found little interest in actually offering comments (1). Peer review, I gather, works from a sense of duty, but often the top scientists are very busy, and I expect there is less interest in doing it for smaller, low-impact journals. If comments were routinely done as separately published papers, this might be an incentive to spend more time critically examining what other people are doing, as opposed to trying to do more experiments yourself to get the results published.
Of course, there are also problems:
- Technically, someone would have to go through and create the tags. Word processors do not include this functionality, although perhaps it could be added to reference management software. Authors? Editors? Or crowd source it (get readers to tag citations)?
- Would it be possible to represent all of the nuances of relationships between papers well enough in a manageable number of tags?
- If this were to ultimately happen without a pre-publication filter (see last potential advantage above), would we need some way of identifying ‘scientists’, so that e.g. creationists could not flood the web with comments tagged “evidence against” for papers about evolution?
I’m sure you’ll all spot some extra problems!
1) http://www.nature.com/nature/peerreview/debate/nature05535.html
-
Replies
-
Thomas, your post brought to mind Cohere.
This is a tool which uses argumentation theory to organise ideas into webs where ideas or concepts are connected by relationships such as ‘supports’, ‘refutes’, ‘responds to’, ‘is an example of’, ‘is consistent with’, ‘is analogous to’, ‘proves’. I think it is quite early days yet, but it is an interesting idea. The people at the Knowledge Media Institute of the OU that are developing this are looking at how it can be used to investigate how new media can shape the development of collective intelligence to better analyse complex problems such as climate change.
It’s an interesting idea to tag citations in this way. -
I think the system explained by Sarah is pretty interesting and exemplifies what you’re suggesting; in fact, some of the categories suggested are a lot like how the Discussion section of a research report works, so it wouldn’t be difficult for authors to put in their two cents (thus representing the first node?) while readers add theirs. It would be a terrific teaching tool for young scholars and would allow profs to assemble example papers quickly (I know, not the intended purpose, but a very nice benefit!). Getting around the ID filter is a hard one, though perhaps a voluntary system whereby reviewers make their “pedigree” available is not a bad idea — would at least point to likely biases.
-
Having visited Cohere (which I may try b/c it is a way to visually represent ideas that looks easier and more functional than a lot of mind/concept mapping programs), I wonder if a science-based tagging system wouldn’t work better as a kind of app rather than as a “place” to go to enter info — I know that when I am working quickly, I won’t likely go to a site to look for info but would rather click a button to expand to see more info while I’m there looking at the article. That would offer more flexibility, something any person/publisher could use.
-
In a bit of a hurry, but sounds like you are talking about semantic markup? Currently some text-mining projects are doing this partly or wholly automatically, by assigning markup retrospectively. But the holy grail is to develop semantic markup tools for researchers to use as they write. I’ve not seen any of these tools as yet.
For examples of textmining see TextPresso – I think the C. Elegans database is the most developed there. Also Medie .
There was a bit of discussion about semantic markup at the 2008 Science Blogging event and I expect at the Science Online 2009 conference, though I wasn’t there.
-
Thanks everyone, these are some fascinating systems that I’d never come across before.
TextPresso: I can’t work out quite how to search it, but looking at the options available, this is a similar idea, but for the papers themselves, rather than their subject matter.
Cohere: This seems to be closer to what I meant. It would be possible to implement this idea roughly on Cohere—imagine, for example, a node for Wakefield’s MMR-Autism study, with a lot of subsequent studies linked to it with “challenges” or “refutes.” Their example sheet on Black intelligence is a similar example. This would, however, need to be much more closely built in to the whole process, although I appreciate that Cohere at present is more or less a proof-of-concept thing.
-
Just for the fun of pushing this a bit, any system doing this would need to take into account the disciplines of the contributors…I have a set of excerpts from a bunch of articles on laminectomy all using the same basic stats about success…students note that one group (the med doctors) use the data to show how well the procedure works. Therapy-oriented specialists, on the other hand, use the same data to show the procedure fails. So whether any particular piece of research is considered support/extension/contradiction/falsification, etc. will depend on the reader’s biases.
-
The following paper describes a suite of tools for allowing the semantic markup of many of the activity around research: http://bib.oxfordjournals.org/cgi/content/abstract/bbn052v1, including whether papers make claims, counter claims, provide support or otherwise for propositions. It’s pretty cool.
-
There is a parallel discussion happening on a similar topic, which might interest contributors to this thread. See the social networking forum .
-
Mickey: That’s a bit worrying, in a way. I’m rather hoping that that sort of thing is the exception, and we can usually agree on whether the evidence supports or refutes something.
Ian: Thanks, another interesting thing I hadn’t heard of. It looks like I don’t have access to the full paper, but following links I can read their own description: http://sciencecollaboration.org/sites/default/files/www08v4.pdf
It seems to have been inspired by social networks and by alzforum.org, who are working on something similar: http://www.alzforum.org/res/adh/default.asp
“a community-driven knowledge base… researchers can annotate scientific hypotheses, claims, data, and information…tie statements made in scientific publications or on the Web, to scientific evidence, biological terminologies, and knowledge bases, and to claims and counterclaims made by other researchers. "Maxine: Thanks for the link.
When I have time, I plan to make a short list of tag categories, then sit down with a paper or two and see how applicable they are. I’ll let you know what I find.
-
It’s a great idea, Thomas, but that’s not exactly tagging, in the web2.0 sense, it it? To me, that’s more of a semantic web kind of thing(like Cohere). I think the time has come for this kind of approach, and I know there are many groups working on it, it’s just a matter of who pulls it off best.
-
