Quotes don’t equate with quality
Reference to your research in other scientific publications can be both negative and poorly deliberated – so the sheer number of citations is a poor quality indicator, concludes a Swedish researcher.
Denne artikkelen er over ti år gammel og kan inneholde utdatert informasjon.
Scientists quote one another’s research in articles. Such references are tallied and used as a benchmark of the research’s influence.
A higher number of citations is viewed as a greater amount of influence. But do such numbers represent real impact, or just empty buzz?
In Sweden and Norway the quantity of quotes is a key factor in incentive and productivity measurement systems. Both countries use bibliometric systems as criteria for measuring research importance. The number of citations a university’s or university college’s researchers accumulate is also factored in when the government appropriates funds.
Sooner or later, Norwegian politicians or authorities will be asking whether it’s possible to gauge quality by counting citations, thinks Gunnar Sivertsen, who conducts research on scientific publications at the Nordic Institute for Studies and Innovation (NIFU).
“The latest government white paper on research called for better research quality. The outgoing minister has publically broached the subject too. And now a special committee is discussing whether the financial system for universities and colleges promotes quality,” he says.
There are many arguments against using citation counts as a quality yardstick. The latest was presented in a recent Swedish doctoral thesis which looked into the citation culture among researchers.
Haphazard
Swedish researcher Gustaf Nelhans concludes in his dissertation that citation counts are a poor way of measuring quality because a researcher can have a wide array of reasons for quoting a given study.
Nelhans thinks authors of articles sometimes simply don’t give the quotes they use enough consideration. They are almost tossed in as a routine as certain studies become standard references which “everyone” cites.
The PhD thesis goes a long way toward suggesting that many citations are added rather haphazardly, according to a press release from the University of Gothenburg, where Nelhans got his doctorate in scientific theory.
Nor should it be forgotten that researchers also quote others in order to refute them or shed doubt on their conclusions.
The Swedish researcher asserts that citations they shouldn’t be used directly as measurements of quality without due consideration of why the quotes are included.
Impact, not quality
Gunnar Sivertsen agrees with Nelhans on that issue, and says so do others who research such bibliometric systems internationally.
But he doesn’t think Gustaf Nelhans has hit the nail fully on the head in his description of common citation practice among researchers.
“In the natural sciences, where citations are counted most, negative citations are less common. Arguments are not being made against the research that is cited.”
“But the concern about habitual citation is justifiable. References that other researchers have used can simply be repeated. There’s a well-known snowball effect. Once you first get cited the ball starts rolling. But most researchers are still meticulous in their selection of references to others they read,” he says.
“You could say there are some odd mechanisms involved in the ways that citations get used, but it is still a valid way of measuring impact and penetration.”
Reporting citations
NIFU, where Sivertsen is a researcher, issues annual reports including statistics on how much Norwegian research gets quoted. The justification given is that these numbers are indicators of the influence of the research.
Political decisions can be partly based on such NIFU reports. But Sivertsen doesn’t think politicians are misunderstanding the situation and equating the influence of research with its quality:
“NIFU says in the indicator report and elsewhere that citations express ‘influence’, not ‘quality’. I don’t think the authorities have allocated funds so far on the basis of misunderstandings of the significance of citations.”
Sweden implemented citation frequency as a measurement of quality in 2008.
“So the issue is extra important in Sweden. The new doctoral thesis should be seen as a response to this,” says Sivertsen.
Belgium is the only country in Europe which uses citations in the same way as Sweden. The Swedish government decided last year to evaluate whether to use peer reviews instead of citations to assess the quality of university research.
--------------------
Read the Norwegian version of this article at forskning.no
Translated by: Glenn Ostling
