Utilization of Computer Adaptive Tests in Patients with Knee Conditions: A Systematic Review

Tanner Holden, SPT, CSCS

Background

I have been highly interested in orthopedic physical therapy since I began shadowing physical therapists as an undergraduate. Throughout my graduate education I have been fortunate enough to have incredible mentors and clinical experiences that exposed me to a variety of patient populations, settings, and clinical skills. Conducting research was one aspect of my professional education that I had not experienced prior to this semester. As such, I was excited when I learned that Dr. Louise Thoma, PT, PhD was joining our faculty and was offering orthopedic research experience as part of a research elective course. I, and two other, third year students (Gabe Dimock, SPT and Emma Shirley, SPT) spent the fall semester learning how to conduct a systematic review under the guidance of Dr. Thoma. Our plan was to conduct a systematic review on the recovery of impairments after anterior cruciate ligament reconstruction in the early postoperative period. However, given the scope of the project and our limited resources we determined the project was not feasible for our deadlines.

 

Thus, we shifted gears and embarked on 3 separate systematic reviews. Under the guidance and direction of Dr. Thoma, I decided to pursue a systematic review examining the use of computer adaptive tests (CATs) to collect patient-reported outcomes in patients with knee conditions. The importance of patient-reported outcome measures (PROMs) for measuring the value of healthcare is well-established.¹ There are multiple patient-reported outcome measures that are used in populations of patients with knee conditions. This can make selecting the most appropriate tool difficult. Additionally, fixed-length PROMs are often lengthy and can take a considerable amount of time for the patient to complete and the clinician to interpret. In busy clinics, patients may be rushed to complete multiple outcome measures or they may not be able to provide complete responses due to clinic flow. These limitations of fixed-length measures can make collecting accurate and clinically useful data unnecessarily difficult.

CATs have the potential to mitigate the aforementioned shortcomings of fixed-length PROMs and improve the overall efficiency of data collection. In contrast to fixed-length measures, CATs are flexible, meaning that items can be added or deleted to the item bank to improve the coverage for a wide range of patient abilities.¹ CATs can capture patient-reported data with fewer questions as well. This is because CATs use complex computer algorithms and item response theory (IRT) to tailor question delivery based on responses to each subsequent question.¹ Check out my infographic for a visual of how CATs work.  While you may need to answer all 20 questions on a fixed-length measure, CAT tools can determine a score in only 4-6 questions without sacrificing precision, accuracy, or reliability. Additionally, these CATs are capable of being integrated into electronic health records to allow for instantaneous data to be used for clinical decision-making. A CAT tool utilized at UNC Hospitals Therapy Services called Focus On Therapeutic Outcomes (FOTO) was my first exposure to CAT in patient-reported outcomes. I wanted to look at the psychometric properties and utilization of FOTO in more detail as well as identify other CATs that have been used in patients with knee conditions.

It is my hope that the results of this project will inform clinicians about the clinical feasibility, psychometric properties, and positive impact that CATs can have on orthopedic practice. Furthermore, the results of this project can be used to fill gaps in the literature in an effort to move towards more efficient methods of collecting patient-reported outcomes.

Statement of Need and Purpose

Patient-reported outcome measures (PROMs) are widely recognized in the field of orthopedics as important tools to evaluate the value of care and the effectiveness of treatment from the perspective of the patient.^1,2 As healthcare in the United States continues to shift towards value-based reimbursement, data from PROMs increasingly influence patterns of care delivery and clinical decision-making.² An abundance of validated PROMs for orthopedic knee conditions are historically used in clinical practice such as the International Knee Documentation Committee (IKDC) subjective knee form, the Knee Injury and Osteoarthritis Outcomes Score (KOOS), and the Lysholm Knee score, among others.^1,3,4 These fixed scale PROMs typically carry a trade-off between test length and score precision and they require responses to most, if not all, of the questions to produce a valid score.⁵ Administering these tests burden patients and clinicians by increasing time required for completion, especially when multiple PROMs are indicated.^5,6 User fatigue and respondent burden may preclude the collection of accurate data and render the results less meaningful.

 

For a PROM to be implemented successfully, it should possess sound psychometric properties, minimize respondent fatigue, and improve the feasibility of scoring and interpretation.^1,3 Computerized adaptive testing (CAT) is a measurement methodology that meets these metrics. In contrast to fixed-length PROMs, CATs measure patient-reported outcomes using algorithms based on item response theory (IRT). IRT tailors item delivery to the patient by selecting each successive item from an item bank based on the patient’s response to the previous item, in order to generate an estimation of the construct being measured.⁵ This allows for precise and accurate data collection with fewer questions.⁵ CATs reduce administrative and respondent burden, as well as improve the efficiency of patient-reported data collection and interpretation.^3,5,7 With development of CATs such as the Patient Reported Outcomes Measurement Information System (PROMIS CAT)⁸ and Focus On Therapeutic Outcomes (FOTO)⁹, there has been an increase in published literature evaluating the utility of these tests in research and practice. Recent studies have validated the use of CATs in spine surgery^10–12 and upper limb trauma.¹³ They reported precision and efficiency of CATs, strong correlations with fixed scales, and adequate psychometric properties compared to fixed length PROMs.^10–14

 

For people with orthopedic knee conditions, there are several fixed-length legacy PROMs used in research and clinical practice.^1,3,4  However, it is unclear how and to what extent CATs are used in this population, and the utility of CATs compared to legacy PROMs. Therefore, the purpose of this study was to systematically review the use of CATs to measure patient-reported data in people with knee conditions. Specifically, our aim was to (1) assess how and for what purposes CATs have been used, (2) evaluate the psychometric properties of CATs, and (3) determine correlations between legacy PROMs and CATs for people with knee conditions.

Overview

This systematic review was performed using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines and the review protocol was prospectively registered in the PROSPERO database (currently awaiting approval). Systematic searches were conducted on December 3, 2019 in PubMed, Sport Discus, EMBASE, CINAHL, and Web of Science. There were 346 studies included for the title/abstract screening. Sixty-seven studies were included for full text review. Of these, 27 studies were excluded for reporting only fixed length short form versions of CATs (n = 17), failing to report data from CATs (n = 6), full text unavailable (n = 3), and less than 50% of the patient sample had a knee condition (n = 1). Forty studies met the inclusion criteria and were included in this review. The most frequently implicated domains at risk for bias included selection bias, failure to adjust for confounding variables, and doubtful methodologies used to estimate of reliability and measurement error. Three unique CAT tools were identified from the search: PROMIS CAT, FOTO Knee FS, and OA-CAT. The PROMIS CAT scales demonstrated moderate to strong correlations to a variety of fixed length measures used to evaluate patients with knee conditions. CATs were administered to multiple patient populations and used for a variety of purposes. All 3 CATs demonstrated sound psychometric properties and were more efficient than traditional fixed length PROMs without sacrificing measurement precision. Studies on the PROMIS CAT scales and FOTO Knee FS measure have identified clinically relevant values, such as the MCID which can assist clinicians and patients in shared decision-making about treatment. As such, the PROMIS CAT measures and FOTO Knee FS measure are recommended for clinical use in patients with knee conditions. This review had several limitations including selection bias and selective reporting bias of included studies as well as publication bias. Publication trends suggest PROMIS CAT literature is increasing year after year. This trend was not observed with the FOTO Knee FS measure or OA-CAT scales. Future research should continue to validate PROMIS CAT scales in specific knee populations and young, high functioning patient cohorts.

Products

I began conducting a systematic review of the literature on the utilization of CATs in patients with knee conditions on December 3, 2019. I constructed a manuscript based on my findings. The second linked document contains the tables and figures referenced in the manuscript. Tables are frequently referenced in the manuscript. As such, it is recommended that both the manuscript and accompanying tables and figures be viewed, simultaneously. The results of this review are intended to be clinically useful for healthcare professionals and informative for health outcomes research. I also created an infographic to address health literacy. The infographic is intended to inform patients and clinicians of the purpose and potential benefits of using CAT tools to collect patient-reported data. The final products are linked below.

• CAT Manuscript
• CAT Tables, Figures, and Appendices
• Computer Adaptive Testing Infographic

Next Steps

The ultimate goal of this project was to prepare a manuscript for journal submission. While I am ultimately proud of the quality of the current products, further revision and peer review is necessary before formal submission to a journal is considered. I have also discussed the possibility of preparing an abstract poster presentation for the 2021 Combined Sections Meeting. I will continue to consult with my advisor and committee members while I consider these options.

Evaluation

Evaluation of this capstone project came in the form of discussion and formal editing by my advisor (Dr. Thoma), peers (Emma Shirley and Gabe Dimock), and my capstone committee members (Dr. Debby Givens, PT, PhD and Dr. Carla Hill, PT, OCS). The manuscript was subject to several revisions throughout the course of the semester. After completing each section, the manuscript was independently reviewed twice, once by my advisor and once by a peer. At midterm, the drafts of the Introduction, Methods, and Results sections were sent to my advisor and my committee members for review. The full manuscript draft and infographic were sent to my advisor, my peer review group, and my capstone committee before final submission.

Self-Reflection

I have learned a great deal throughout the development of this project. I have gained practical experience conducting a systematic review and have developed a deeper appreciation for the rigor and level of detail that goes into research. This project was a massive undertaking that spanned almost 2 full semesters. I’ll be the first to admit that I was terrified when our original plan (see background above) got derailed and we decided to conduct 3 separate reviews. I didn’t think myself capable of being the lead author on a research project and certainly not so soon in my career. However, I stepped out of my comfort zone, tried something new and challenging, and ultimately created products that I’m really proud of. I’ve greatly improved my academic writing skills as a result of extensive peer reviewing and reading research. In the same vein, I’ve learned to accept (and appreciate) that my first draft will be drastically different than my final draft; I am indebted to the revision process. Working with an extremely intelligent group of people was a key to my success.

This project reinforced the importance of planning, preparation, and organization. As a result of reading similar reviews and taking bits and pieces from their data extraction methods, I was able to formulate a clearly organized plan for my own data extraction. I have a better understanding and appreciation for the complexity of psychometric properties—though I still have much to learn. I feel more confident in my ability to critically analyze and synthesize relatively large bodies of evidence and identify gaps in the literature. Conducting research is difficult, time-consuming, and requires thinking about the big picture and small details, concurrently. I’m looking forward to any opportunities I have to conduct research in the future.

Countless hours, over 17,000 words, 71 pages, 64 references, 40 included studies, 2 semesters, and 1 infographic later, I’ve finally finished one of my biggest endeavors to date. Some final takeaways I had are to create a timeline and stick to it, put in work every day, click the save icon frequently, be as organized as possible, and use a reference manager.

Acknowledgements

This project would not have been possible without the contribution of time and effort from an outstanding group of faculty, clinicians, and colleagues. First, I would like to thank my outstanding advisor, Dr. Louise Thoma for the opportunity to gain practical experience conducting research. Dr. Thoma has been extremely patient with me throughout each stage of the project and has been pivotal in providing me feedback, direction, and clarity. This project would not have come to fruition without her guidance and expertise.

I would like to acknowledge my all-star capstone committee members, Dr. Debby Givens and Dr. Carla Hill. Both Dr. Givens and Dr. Hill have provided me with timely and valuable feedback amidst fulfilling countless other obligations. Dr. Givens has been our fearless leader and biggest advocate over the past 3 years and through a pandemic. Dr. Hill has an impressive clinical skillset and has been an excellent clinical mentor to me.

I would also like to recognize my impressive peer reviewers and fellow classmates, Gabe Dimock and Emma Shirley. Their constructive feedback, encouragement, and comradery throughout each stage of the project motivated me to put my best foot forward and create quality products.

I would also like to thank my amazing wife, Rachael, and my dog Sage, who have provided endless hugs, encouragement, and support throughout this project and my Doctorate education.

Last but certainly not least, I would like to thank the outstanding UNC DPT faculty and my 2020 cohort-family for everything over the past 3 years. The UNC DPT faculty are a world class group of professionals who have poured their hearts and souls into developing us as physical therapists. To the class of 2020, we have laughed together, we invented a muscle as DPT1s, and we repeatedly clapped for each other for no reason at all, over the past 3 years. You all are incredible human beings and extremely capable healthcare professionals. I’m honored to have been in the trenches with you and I know that there will be more chapters in our book.

 

Best regards,

Tanner Holden

References

1. Beaton DE, Schemitsch E. Measures of health-related quality of life and physical function. Clin. Orthop. Relat. Res. 2003;(413):90-105. doi:10.1097/01.blo.0000079772.06654.c8.
2. Ayers DC. Implementation of Patient-reported Outcome Measures in Total Knee Arthroplasty. J Am Acad Orthop Surg 2017;25 Suppl 1:S48-S50. doi:10.5435/JAAOS-D-16-00631.
3. Fidai MS, Saltzman BM, Meta F, et al. Patient-Reported Outcomes Measurement Information System and Legacy Patient-Reported Outcome Measures in the Field of Orthopaedics: A Systematic Review. Arthroscopy 2018;34(2):605-614. doi:10.1016/j.arthro.2017.07.030.
4. Wang D, Jones MH, Khair MM, Miniaci A. Patient-reported outcome measures for the knee. J. Knee Surg. 2010;23(3):137-151. doi:10.1055/s-0030-1268691.
5. Brodke DJ, Saltzman CL, Brodke DS. PROMIS for orthopaedic outcomes measurement. J Am Acad Orthop Surg 2016;24(11):744-749. doi:10.5435/JAAOS-D-15-00404.
6. Bodart S, Byrom B, Crescioni M, Eremenco S, Flood E. Perceived Burden of Completion of Patient-Reported Outcome Measures in Clinical Trials:: Results of a Preliminary Study. Ther. Innov. Regul. Sci. 2019;53(3):318-323. doi:10.1177/2168479018788053.
7. Scott EJ, Westermann R, Glass NA, Hettrich C, Wolf BR, Bollier MJ. Performance of the PROMIS in patients after anterior cruciate ligament reconstruction. Orthop. J. Sports Med. 2018;6(5):2325967118774509. doi:10.1177/2325967118774509.
8. Cella D, Yount S, Rothrock N, et al. The patient-reported outcomes measurement information system (PROMIS): Progress of an NIH roadmap cooperative group during its first two years. Med. Care 2007;45(5 Suppl 1):S3-S11. doi:10.1097/01.mlr.0000258615.42478.55.
9. Hart DL, Wang Y-C, Stratford PW, Mioduski JE. Computerized adaptive test for patients with knee impairments produced valid and responsive measures of function. J. Clin. Epidemiol. 2008;61(11):1113-1124. doi:10.1016/j.jclinepi.2008.01.005.
10. Bhatt S, Boody BS, Savage JW, Hsu WK, Rothrock NE, Patel AA. Validation of Patient-reported Outcomes Measurement Information System Computer Adaptive Tests in Lumbar Disk Herniation Surgery. J Am Acad Orthop Surg 2019;27(3):95-103. doi:10.5435/JAAOS-D-17-00300.
11. Boody BS, Bhatt S, Mazmudar AS, Hsu WK, Rothrock NE, Patel AA. Validation of Patient-Reported Outcomes Measurement Information System (PROMIS) computerized adaptive tests in cervical spine surgery. J Neurosurg Spine 2018;28(3):268-279. doi:10.3171/2017.7.SPINE17661.
12. Haws BE, Khechen B, Bawa MS, et al. The Patient-Reported Outcomes Measurement Information System in spine surgery: a systematic review. J Neurosurg Spine 2019;30(3):405-413. doi:10.3171/2018.8.SPINE18608.
13. Jayakumar P, Overbeek C, Vranceanu AM, et al. The use of computer adaptive tests in outcome assessments following upper limb trauma: a systematic review. Bone Joint J. 2018;100-B(6):693-702. doi:10.1302/0301-620X.100B6.BJJ-2017-1349.R1.
14. Gulledge CM, Lizzio VA, Smith DG, Guo E, Makhni EC. What Are the Floor and Ceiling Effects of Patient-Reported Outcomes Measurement Information System Computer Adaptive Test Domains in Orthopaedic Patients? A Systematic Review. Arthroscopy 2020. doi:10.1016/j.arthro.2019.09.022.

Image references:

1. EHR 2.0: How New Clinical Decision Support Tools Are Helping Doctors Make Smarter Decisions at the Point of Care | Mercy Technology Services. https://www.mercytechnology.net/insights/blog/clinical-decision-support. Accessed April 16, 2020.
2. Knee Osteoarthritis: Symptoms, Diagnosis & Stages. https://www.pthealth.ca/blog/knee-osteoarthritis-my-knees-hurt-do-i-have-oa/. Accessed April 15, 2020.
3. Waiting Room Ipad Stock Photos, Pictures & Royalty-Free Images – iStock. https://www.istockphoto.com/photos/waiting-room-ipad?mediatype=photography&phrase=waiting%20room%20ipad&sort=mostpopular. Accessed April 16, 2020.
4. The PROMIS in Putting Patients’ Waiting Room Time to Good Use | PCORI. https://www.pcori.org/research-results/pcori-stories/promis-putting-patients-waiting-room-time-good-use. Accessed April 15, 2020.