A new "research impact ranking" provides a more objective approach to assessing and comparing research productivity at US orthopedic departments, according to a study in the November Journal of Orthopaedic Trauma (JOT), published by Wolters Kluwer.

Partly motivated by the desire to track the success of their own program's research effort, Dr. Louis B. Jones and colleagues of University of Mississippi Medical Center, Jackson, developed an alternative approach to ranking orthopedic surgery residency training programs regarding research productivity.

The researchers write, "We have created a reproducible and meaningful residency ranking system that will allow programs to evaluate their progress both against their peers and within their systems over time."

System Ranks Quality as Well as Quantity of Research by Orthopaedic Surgery Trainees

Research, along with clinical care and academic performance, is a key part of the mission of orthopedic surgery training programs. But research productivity is difficult to measure objectively. Simply counting the number of studies published accounts for the quantity but not necessarily the quality of research produced.

Dr. Jones and colleagues sought to develop a benchmarking system, or "h-index," for residency programs. From a list of 72 orthopedic surgery journals, they identified 48 "relevant, impactful" journals based on the Impact Factor, a traditional measure of the importance of medical journals; and a newer alternative metric called the Eigenfactor. 

Dr. Jones and colleagues then created a database of nearly 103,000 articles published in these journals from 2005 to 2015. These articles were then attributed to the 157 US orthopedic surgery residency programs. The researchers put their model through several iterations to identify the most effective search process.

The results provided a ranking of the orthopedic training programs with the most productive research efforts, based on the number of citations in high-quality journals. The article in Journal of Orthopedic Trauma presents the full list of programs and their rankings. The data can also be accessed at the University of Mississippi Medical Center Department of Orthopaedic Surgery and Rehabilitation research page.

Because the ranking system accounted for the number of articles published, larger programs–based on some trainees and number of faculty–tended to have higher rankings. Each of the six top-ranked programs had more than 1,000 research publications and more than 20,000 citations over the ten-year period studied.

The rankings regarding number of citations were closely correlated with the number of articles published in the select orthopedic surgery journals. The rankings were "moderately" correlated with other systems for ranking training programs.

Dr. Jones and colleagues note some limitations of their ranking system–notably including the fact that it's sometimes difficult to attribute articles to a specific residency program.

Nevertheless, they believe their ranking system, based on "clear and reproducible" methods, provides a useful approach to assessing and comparing the research productivity of orthopedic residency programs. The authors conclude, "It is relevant and based on measurable data that is the cornerstone of an academic surgeon's practice: publishing articles to contribute to the orthopedic knowledge base."