Abstract

Objective To assess the use of common author ordering conventions, their effects on measures of citation impact, and their implications for the assessment of individual researchers and researcher rankings.

Design Analysis of associations between the use of author ordering conventions, measures of citation impact, and researcher rankings using publications by Canadian primary health care researchers included in the Scopus database.

Participants The 49 living Canadian primary health care researchers with the most first-author citations.

Main outcome measures Spearman rank correlations were assessed between rankings based on number of first-author citations and alternative measures of number of citations. Changes in researcher rankings were assessed based on alternative citation metrics.

Results Rank order correlations varied from 0.55 (first author vs h index) to 0.83 (first-author citations vs first- and second-author citations). The proportion of researchers whose rankings changed by 12 or more ranks (25% or greater absolute change) compared to rankings based on first-author citations varied from 14% for rankings based on first- and second-author citations to 51% for rankings based on h index.

Conclusion The variability and inconsistency of author ordering thwart efforts to identify or create valid measures to rank citation impact. Adoption of author ordering based on contribution as a universal convention would enhance the reliability and validity of comparisons and rankings across disciplines and research fields and would facilitate comparisons among candidates for hiring, tenure, promotion, and awards.

Citation analysis is routinely used to assess the quantity and quality of researchers’ contributions to scholarly literature in decisions regarding hiring, promotion, tenure, and research funding, and for ranking researchers’ academic impact. These processes are bedeviled by a lack of uniformity in the criteria by which authors are ordered.1,2

Many citation-based metrics are available, including number of peer-reviewed publications, number of citations, number of first-author citations, h index, and for individual publications, journal impact factor and the Field-Weighted Citation Impact from Scopus.

The International Committee of Medical Journal Editors (ICMJE) offers no prescription for author ordering, stating that “the criteria used … are to be decided collectively by the author group and not by editors.”3 Biomedical journals offer inconsistent guidance—and often no guidance—on authorship order.4

In this study we examined the use of common author ordering conventions, their effects on measures of citation impact, and their implications for the assessment of individual researchers and researcher rankings. Although the limitations of citation-based metrics and the need to broaden the range of impact measures are increasingly recognized (eg, in the San Francisco Declaration on Research Assessment5), bibliometric measures will continue to be used, at the very least as a screen for highly cited researchers and publications, despite their limitations.

Ethics approval was not required for this study as all information used was obtained from publicly available sources. The study was self-funded.

METHODSStudy population

In a previous study,6 we identified 266 living and established Canadian primary health care researchers through a sequential nomination process during 2020. We defined primary health care researcher as a researcher whose focus is primary health care, or a primary care clinician who conducts research, even if that research does not focus mainly or directly on primary health care. The initial list was created in March 2020 and consisted of recipients of research honours from the College of Family Physicians of Canada. Additional researchers were added to the list based on input from senior researchers and primary health care leaders. The list was sent to research directors of the 17 Canadian family medicine departments, who provided names of researchers whose first-author peer-reviewed papers they believed were highly cited. Finally, 6 additional researchers were added from the Stanford University list of Canadian researchers. We eliminated 26 researchers who had no first-author publications, no research in primary care, or spent a minor portion of their research career in Canada. Study methods and results are documented elsewhere.6 This cohort served as the sampling frame for the current study. The researcher sample comprised the 50 living researchers with the most first-author citations. Their characteristics are described elsewhere.6

Data collection

We obtained citation data from the Scopus database between mid-December 2022 and early January 2023. For each researcher, we determined the numbers of

first-author citations,

first- and last-author citations,

first- and second-author citations,

first-, second-, and last-author citations, and

total citations.

In each category we excluded self-citation. For 2-author publications, second and last authorship were only counted once in calculating first, second, and last authorship. We ranked the researchers in descending order of the number of citations in each citation category. We also retrieved the Scopus h index for each researcher and ranked them in descending order. The h index was proposed as a measure combining productivity and impact. It is defined as the maximum h for which the author has published h papers with at least h citations each.7

Data analysis

We excluded 1 researcher from the analysis who had died after the original study was completed, leaving a cohort of 49. Almost three-quarters (73.5%) of the researchers, most of whom obtained additional research training at the master’s or PhD level, are family or emergency medicine physicians; the remainder are from other clinical or academic disciplines.

Through discussion among the investigators, we identified 5 conventions that are used by Canadian primary care researchers to establish the order in which authors are listed (Box 1).

Box 1. Conventions for order of authors

5 author ordering conventions used by Canadian primary care researchers:

Descending order of contribution

Research supervisor of a student, mentee, or fellow is listed as first author

Research supervisor of a student, mentee, or fellow is listed last (“senior author”); student, mentee, or fellow is listed first

Research supervisor of a student, mentee, or fellow listed as second author*

Alphabetical

↵* When the main work is done by a student, mentee, or fellow. This criterion assumes that the research supervisor makes the second largest contribution after the student, mentee, or fellow.

We conducted a sensitivity analysis comparing the researcher citation rankings between the base case of first-author citations and 5 other citation-based metrics: the number of first- and last-author citations; the number of first-, second-, and last-author citations; the number of first- and second-author citations; the total number of citations; and the Scopus h index. We also compared rankings based on second-author publications to those based on last-author publications, anticipating a low or negative correlation if researchers were consistent in using either the second-author or senior-author convention when the work was done by a student, mentee, or fellow.

Using SPSS, version 28, with R, version 4.2.2, we calculated Spearman rank correlations to examine the relationship between rankings of citation impact as measured by the number of first-author citations versus other alternatives. The resulting correlations for these nonparametric data can be interpreted similarly to the more commonly used Pearson correlation. For each alternative measure, we identified researchers whose ranking increased or decreased by 12 or more ranks (ie, 25% or greater absolute change in ranking) compared to their ranking based on first-author citations.

RESULTS

Among the 49 researchers included in the analysis, the number of first-author papers ranged from 13 to 249 (median=48) and the number of first-author citations ranged from 810 to 9081 (median=1192). Total citations ranged from 1077 to 27,155 (median=6631). First-author papers were cited an average of 37 times (range across researchers 10 to 175 per paper, median=82). The h index ranged from 15 to 87 (median=37). The sample mainly comprised mid- and late-career researchers, whose career durations spanned 14 to 45 years (median=28 years); 41% were women. (We used the number of years from researchers’ earliest first-author publication to their most recent year of publication as either a first author or a co-author as a proxy for duration of career.)

Spearman rank correlations for researcher rankings between different measures are shown in Table 1. Rankings based on first-author citations and h index rankings are moderately correlated (r=0.55). Rankings across measures from the other categories are moderately to strongly correlated (r of 0.62 to 0.83).

Table 1.

Spearman rank correlations between rankings based on number of first-author citations and alternative measures of number of citations

The correlation between rankings based on second-versus last-author citations is low (r=0.47). This may indicate that while many researchers are consistent, others are inconsistent in their use of either the second-author convention or the senior-author convention when the work is done by a student, mentee, or fellow.

Figure 1 is an example of a comparison between numbers of citations in different categories in a scatter plot. It shows a positive relationship between the number of citations based on first- and last-author citations versus first-author citations. Scatter plots for the other categories are similar, although for most the scatter is broader.

Figure 1.

Comparison of first- and last-author citations vs first-author citations

Since the rank correlations were 0.83 or lower, we looked more closely at those researchers whose ranks changed substantially. Table 2 summarizes our results for researchers whose ranks increased or decreased by 12 or more based on the alternative citation categories. The number of researchers whose rankings changed by 12 or more ranks compared to rankings based on first-author citations ranged from 7 (14%) for rankings based on first- and second-author citations to 25 (51%) for rankings based on h index.

Table 2.

Frequency of change in rank ≥12 when alternative measures are compared with first-author citations: N=49.

Several notable patterns emerged: Compared to rankings based on first-author citations, 6 researchers’ rankings increased by 12 or more based on first- and last-author citations, indicating that these researchers may have frequently used the senior-author convention for work done by students, mentees, or fellows.

Four researchers experienced increases of 12 or more ranks based on first and second authorship versus first authorship only, suggesting they frequently employed the second-author convention for work done by students, mentees, or fellows.

Not shown in Table 2, 11 researchers’ rankings were 12 or more ranks lower based on second-author citations compared to last-author citations, indicating they may have rarely used the second-author convention. Eight others ranked 12 or more positions higher based on second- versus last-author citations, likely indicating rare use of the senior-author convention. The remainder presumably ordered authorship based on descending order of contribution or were inconsistent in their use of the second- and senior-author conventions.

Some swings in individual researchers’ rankings were dramatic. The researcher ranked 44th based on first-author citations rose to 16th based on first-, second-, and last-author citations, 13th based on total citations, and 21st based on h index. The researcher ranked 10th for first-author citations ranked fifth on first- and second-author citations, and first on the other 4 measures. To generate this pattern, the researcher appears to have frequently used the second-author and last-author conventions (but predominantly the latter) when the work was done primarily by a student, mentee, or fellow. The 28th-ranked researcher based on first-author citations ranked ninth based on first- and last-author citations and third based on h index. The researcher ranked 20th based on first-author citations fell to 34th based on first- and last-author citations and 41st based on total citations.

DISCUSSION

Our results show that, in this cohort of mid- to late-career Canadian primary health care researchers, both the citation impact of the individual researchers and rankings of researchers’ relative citation impact vary substantially using alternative citation-based measures, driven by researchers’ variable use of author ordering conventions. In absence of an accepted reference standard, we have no way of determining which, if any, of the current measures captures researchers’ scholarly impact.

Researcher rankings in the peer-reviewed literature use a variety of metrics; for example, first-author citations8-10; single-author citations10; single- or first-author citations10; single-, first-, or last-author citations10; total citations8-12; h index8-15; m index (h index divided by the number of years since the author’s first paper was published)16; and c index (the number of articles cited more than once by other research groups in the most recent calendar year).11 Additional metrics include the G index (“the largest number n of highly cited articles for which the average number of citations is at least n”17); the i-10 index (developed by Google Scholar: the number of articles published by an author that have received at least 10 citations); and h frac index (citation count divided by the number of authors).17

Web-based researcher citation rankings, including Clarivate Highly Cited Researchers18 and AD Scientific Index,19,20 are released annually. The former includes researchers who “rank in the top 1% of citations for their field(s) and publication year in the Web of Science.... However, citation activity is not the sole selection indicator. This list, based on citation activity is then refined using qualitative analysis and expert judgment.”18 In its Scientists Citation Rankings 2025, the AD Scientific Index reports total citations, h index scores, and i-10 index scores for total years, the past 6 years, and the past 6 years divided by 10.19 Its World Top 100 Scientists ranking is based on total h index scores.20

In a 2016 study, Ioannidis and colleagues assessed the correlation patterns (using Pearson correlation coefficients) of 6 metrics based on citations in 2013 among the top 30,000 scientists across all disciplines and 12 research fields (2 of which were medicine and health sciences).10 Correlations across the 6 measures (single-author citations; single- or first-author citations; single-, first-, or last-author citations; total citations; h index; and Schreiber co-authorship adjusted hm index*) varied from −0.43 to +0.89 overall. The ranges for medicine and health sciences were slightly narrower. The authors found that “different scientists populated the top ranks when different indicators were used”10 and concluded that “no citation indicator, single or composite, can be expected to select all the best scientists.”10

In our background search for relevant peer-reviewed literature, we found no studies that—like ours—assessed the direction and magnitude of changes in researcher ranking using alternative citation metrics.

Examinations of h index validity have found it wanting. In a study published 2 years after the introduction of the h index, Bornmann and Daniel reviewed 5 studies of its convergent validity.21 Four studies supported the convergent validity of the h index “in general,” whereas a fifth study showed the h index to “lack the necessary accuracy and precision to be useful.”21 In a recent large-scale study, Koltun and Hafner found that the correlation of the h index with awards that indicate recognition by the scientific community substantially declined between 2010 and 2019, in part due to “hyperauthorship” (large numbers of co-authors per paper).22 The authors noted that they did not address the role of author order, suggesting it as an “interesting avenue for future work.”22 An unpublished article recently reported the discovery of a service that offers fake citations for a fee.23 Although the evidence suggests this practice is currently rare, the authors concluded that their findings “highlight the need to look beyond citation counts.”23

Limitations

The restrictions in our study of country, research field, and career stage limit the generalizability of our results. However, the authorship ordering conventions, relationships, and patterns we observed are described—but not quantified—in other settings and research fields.1,2

Authorship is sometimes ordered alphabetically, at least in part (eg, most substantial contributor first, others listed alphabetically), especially when the number of authors is large and relative contributions are difficult to determine. We chose not to explore this convention for 2 reasons: first, substantial resources would be required to review the author list of hundreds of relevant papers; and second, when the number of authors is small, alphabetical ordering can occur by chance rather than intent, and it would be impossible to distinguish between the 2. In their examination of authorship order, Ioannidis and colleagues found that in the medicine research field authorship was alphabetically ordered in 17% of 3-author papers and 1% of 5-author papers.10 This finding is consistent with alphabetical ordering mainly by chance and suggests that intentional alphabetical author ordering is seldom used.

Our nomination process may have failed to identify some highly cited primary health care researchers, particularly those not affiliated with a department of family medicine. Finally, we may have failed to detect attribution errors in the Scopus database.

Conclusion

Given the variable and inconsistent use of author ordering conventions, what can be done? For assessing the citation impact of individual researchers, a partial—although awkward—remedy is available: researchers being considered for hiring, promotion, tenure, and research funding should be asked to identify the author ordering conventions they use and in what proportions they use them. Better yet, identification of author order could be done on a publication-by-publication basis, although this requirement would place an onerous burden on researchers with many publications.

The variability and inconsistency of author ordering negatively affects attempts to identify or create valid measures to rank citation impact—the bibliometric equivalent of attempting to fashion a silk purse out of a sow’s ear. No amount of data manipulation can overcome the inherent inadequacy of the raw data. Simply put, the data are not fit for purpose.

The muddy waters of citation impact could be substantially clarified by the adoption of author ordering based on contribution as a universal convention. This would require the ICMJE and individual editors to insist on this convention as the international standard. Adoption of this standard would allow weighting of positions in the author order that is consistent with the purpose of the assessment. Extension of the standard to the full range of scientific journals would enhance the reliability and validity of comparisons and rankings across disciplines and research fields. Implementation of the standard at the institutional level would facilitate comparisons among candidates for hiring, tenure, promotion, and awards. For early-career researchers, the playing field would begin to be leveled almost immediately. For mid- and late-career researchers, the effects on citation-based measures of performance and researcher rankings would occur gradually over time.

Footnotes

↵* The hm index is computed similarly to the h index after the citations of each paper have been divided by the number of authors.

Contributors

Dr Brian Hutchison conceived of and led the design of the project, participated in data analysis and interpretation, and drafted the manuscript. Dr Monica Aggarwal oversaw the literature review and data analysis and contributed to interpretation. Dr Harry Shannon guided the data analysis and interpretation. All authors reviewed the draft manuscript and approved the final version.

Competing interests

None declared

This article has been peer reviewed.

Cet article a fait l’objet d’une révision par des pairs.

Copyright © 2025 the College of Family Physicians of Canada