Comparing Apples to Apples in Evaluation Research

This essay was originally published in the Current Contents print editions September 12, 1994.

Evaluation tools for journals have been described in the three most recent essays.(1,2,3)  In the comment that follows, we look at another approach to journal evaluation, but also at two techniques designed for evaluation of research groups and individual scientists. The latter was developed at ISI® (now Clarivate Analytics) and includes a measure called Expected Citation Rates (ECR). The ECR is used to compare the citation record of published items to the citation averages for similar items published in the same journal during the same database year. The two other approaches to journal and group evaluation were reported recently by researchers in the Netherlands. Rickie Deurenberg’s study at the University of Nijmegen (4) uses Clarivate Analytics impact factor and obsolescence indicator (cited half-life) to make decisions on journal selection and weeding. R. Plomp’s study at the Free University Hospital in Amsterdam deals with evaluation of a research group’s performance (5). He uses impact ratios and indicators of efficiency to make his determinations.

Clarivate Analytics ECR

Clarivate Analytics ECR system is based on an analysis of the basic structure of the impact factor, as described here in June (1). However, the ECR is the average citation frequency for a specific item type (article, review, note, abstract, letter, discovery account, etc.) in a specific journal during a specific database year.

This strategy allows one to compare apples to apples. After all, the average paper in immunology will be cited more often than a paper in nuclear engineering. However, by making comparisons within the same subspecialty, a balanced view within the subspecialty is discernible.

Design of the Individual Analysis.

Each individual analysis can and should be designed uniquely. An important variable is the time period chosen. In addition to selecting item types, the cited and citing years can also be selected. For ECR, Clarivate Analytics uses a one-year cited window and the corresponding n-year citing window, where n equals the present year (plus 1) minus the publication year. For example, a paper published in 1987 would have a citing window of 8 years (i.e., 1994 + 1 – 1987 = 8). Time period strategies are variable, however. For instance, a cited period of one year (e.g., 1981) and a citing period of 13 years (1981-1993) can be chosen. Alternatively, cited periods of five years (e.g., 1981-85, 1982-86, …, 1988-92), also referred to as moving windows, and the corresponding one-year citing periods (e.g., 1986, 1987, …, 1993) can be chosen. Regardless of the strategy chosen, it is important for the sake of comparison that the same time periods are chosen for both the paper under consideration and the papers used for calculation of ECR. In both examples, the comparisons must be made with material from the same time period.

Interpretation and Use of Results.

The ECR is an important part of a Personal Citation Report (PCR). The PCR is a complete inventory of an individual’s papers published in Clarivate Analytics -indexed journals between 1981 and 1993. The report includes all standard bibliographic information for each paper as well as its total and year-by-year citation counts. Actual citation performance and ECR are juxtaposed and thereby provide a means of impact comparison with similar types of research publications.

Deurenberg’s Periodicals Ranking

In her study, Deurenberg used Clarivate Analytics impact factors and half-life to clarify the journal collection. These data were used to divide each main subject category into four quartiles (4). The product of the cited half-life and the impact factor was used for further ranking. Journal half-life is the number of journal publication years, going back from the current year, that account for 50% of the total citations received by the cited journal in the current year. A similar measure often used is called the Price Index, which uses the last five years instead. The Price in question is Derek de Solla Price (6). I

For selection purposes, the journal with the lowest product (impact factor multiplied by half-life) from each of the three lowest quartiles was identified, and the average was calculated. The mean served as the threshold, with journals ranking above the threshold being kept on the subscription list and those below being rejected. The threshold can be adjusted to meet individual needs, depending upon the budgetary reduction or increase sought.

The similarity to ECR lies in the attempt to position the item being evaluated (in this case, a journal) so that it compares fairly to other items. Deurenberg’s efforts are geared more toward collection management, but the concept of allowing for variances in several journal characteristics is consistent with ECR.

Plomp’s Indicators of Group Performance

Plomp’s study focuses on highly cited papers (HCPs) to compare the work of specific research groups and, especially, the work of individual scientists within the group(5).  He uses a so-called “impact ratio” and an “efficiency” calculation for each paper.

The impact ratio is the number of citations for a set of papers divided by the average for the research field under study. Here again, the attempt is to compare the performance of published works to other similar works. The efficiency is calculated by dividing citations achieved by the number of references cited in the paper.

Plomp concludes that it is probably more important to look at the performance of the individual researcher rather than the group, and that long-term citation studies are often more revealing than short-term ones.

Conclusions

The ECR provides highly focused comparison of the impact of individual papers. Its inclusion as part of the Clarivate Analytics Personal Citation Report reduces invidious comparisons. Similarly, journal impact factors and half-life measures provide fairer comparisons between fields of research with different rates of acceleration.

While comparisons of papers published in one large journal in a single year provide a fairer basis for comparison, an even more specific validation can be obtained by taking into account the research front–that group of papers published on a more specific topic in a variety of journals. Defining research fronts is the area of our next commentary.

Dr. Eugene Garfield 
Founder and Chairman Emeritus, ISI

References

1.Garfield E. The impact factor. Current Contents® (25):3-7, 20 June 1994.

2.——————. Using the impact factor. Current Contents (29):3-5, 18 July 1994.

3.——————. The application of citation indexing to journals management.Current Contents (33):3-5, 15 August 1994.

4.Deurenberg R. Journal deselection in a medical university library by ranking periodicals based on multiple factors. Bull. Med. Libr. Assoc. 81(3):316-9, 1993.

5.Plomp R. The highly cited papers of professors as an indicator of a research group’s scientific performance. Scientometrics (29):377-93, 1994.

6.Price D J D. Citation measures of hard science, soft science, technology, and nonscience. (Nelson C E & Pollack D K, eds.) Communication among scientists and engineers. New York: Columbia University Press, 1986, p. 155-79.