Utilization of Computer Adaptive Tests in Patients with Knee Conditions: A Systematic Review
Tanner Holden, SPT, CSCS
Background
I have been highly interested in orthopedic physical therapy since I began shadowing physical therapists as an undergraduate. Throughout my graduate education I have been fortunate enough to have incredible mentors and clinical experiences that exposed me to a variety of patient populations, settings, and clinical skills. Conducting research was one aspect of my professional education that I had not experienced prior to this semester. As such, I was excited when I learned that Dr. Louise Thoma, PT, PhD was joining our faculty and was offering orthopedic research experience as part of a research elective course. I, and two other, third year students (Gabe Dimock, SPT and Emma Shirley, SPT) spent the fall semester learning how to conduct a systematic review under the guidance of Dr. Thoma. Our plan was to conduct a systematic review on the recovery of impairments after anterior cruciate ligament reconstruction in the early postoperative period. However, given the scope of the project and our limited resources we determined the project was not feasible for our deadlines.
Thus, we shifted gears and embarked on 3 separate systematic reviews. Under the guidance and direction of Dr. Thoma, I decided to pursue a systematic review examining the use of computer adaptive tests (CATs) to collect patient-reported outcomes in patients with knee conditions. The importance of patient-reported outcome measures (PROMs) for measuring the value of healthcare is well-established.1 There are multiple patient-reported outcome measures that are used in populations of patients with knee conditions. This can make selecting the most appropriate tool difficult. Additionally, fixed-length PROMs are often lengthy and can take a considerable amount of time for the patient to complete and the clinician to interpret. In busy clinics, patients may be rushed to complete multiple outcome measures or they may not be able to provide complete responses due to clinic flow. These limitations of fixed-length measures can make collecting accurate and clinically useful data unnecessarily difficult.
CATs have the potential to mitigate the aforementioned shortcomings of fixed-length PROMs and improve the overall efficiency of data collection. In contrast to fixed-length measures, CATs are flexible, meaning that items can be added or deleted to the item bank to improve the coverage for a wide range of patient abilities.1 CATs can capture patient-reported data with fewer questions as well. This is because CATs use complex computer algorithms and item response theory (IRT) to tailor question delivery based on responses to each subsequent question.1 Check out my infographic for a visual of how CATs work. While you may need to answer all 20 questions on a fixed-length measure, CAT tools can determine a score in only 4-6 questions without sacrificing precision, accuracy, or reliability. Additionally, these CATs are capable of being integrated into electronic health records to allow for instantaneous data to be used for clinical decision-making. A CAT tool utilized at UNC Hospitals Therapy Services called Focus On Therapeutic Outcomes (FOTO) was my first exposure to CAT in patient-reported outcomes. I wanted to look at the psychometric properties and utilization of FOTO in more detail as well as identify other CATs that have been used in patients with knee conditions.
It is my hope that the results of this project will inform clinicians about the clinical feasibility, psychometric properties, and positive impact that CATs can have on orthopedic practice. Furthermore, the results of this project can be used to fill gaps in the literature in an effort to move towards more efficient methods of collecting patient-reported outcomes.
Statement of Need and Purpose
Patient-reported outcome measures (PROMs) are widely recognized in the field of orthopedics as important tools to evaluate the value of care and the effectiveness of treatment from the perspective of the patient.1,2 As healthcare in the United States continues to shift towards value-based reimbursement, data from PROMs increasingly influence patterns of care delivery and clinical decision-making.2 An abundance of validated PROMs for orthopedic knee conditions are historically used in clinical practice such as the International Knee Documentation Committee (IKDC) subjective knee form, the Knee Injury and Osteoarthritis Outcomes Score (KOOS), and the Lysholm Knee score, among others.1,3,4 These fixed scale PROMs typically carry a trade-off between test length and score precision and they require responses to most, if not all, of the questions to produce a valid score.5 Administering these tests burden patients and clinicians by increasing time required for completion, especially when multiple PROMs are indicated.5,6 User fatigue and respondent burden may preclude the collection of accurate data and render the results less meaningful.
For a PROM to be implemented successfully, it should possess sound psychometric properties, minimize respondent fatigue, and improve the feasibility of scoring and interpretation.1,3 Computerized adaptive testing (CAT) is a measurement methodology that meets these metrics. In contrast to fixed-length PROMs, CATs measure patient-reported outcomes using algorithms based on item response theory (IRT). IRT tailors item delivery to the patient by selecting each successive item from an item bank based on the patient’s response to the previous item, in order to generate an estimation of the construct being measured.5 This allows for precise and accurate data collection with fewer questions.5 CATs reduce administrative and respondent burden, as well as improve the efficiency of patient-reported data collection and interpretation.3,5,7 With development of CATs such as the Patient Reported Outcomes Measurement Information System (PROMIS CAT)8 and Focus On Therapeutic Outcomes (FOTO)9, there has been an increase in published literature evaluating the utility of these tests in research and practice. Recent studies have validated the use of CATs in spine surgery10–12 and upper limb trauma.13 They reported precision and efficiency of CATs, strong correlations with fixed scales, and adequate psychometric properties compared to fixed length PROMs.10–14
For people with orthopedic knee conditions, there are several fixed-length legacy PROMs used in research and clinical practice.1,3,4 However, it is unclear how and to what extent CATs are used in this population, and the utility of CATs compared to legacy PROMs. Therefore, the purpose of this study was to systematically review the use of CATs to measure patient-reported data in people with knee conditions. Specifically, our aim was to (1) assess how and for what purposes CATs have been used, (2) evaluate the psychometric properties of CATs, and (3) determine correlations between legacy PROMs and CATs for people with knee conditions.
Overview
This systematic review was performed using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines and the review protocol was prospectively registered in the PROSPERO database (currently awaiting approval). Systematic searches were conducted on December 3, 2019 in PubMed, Sport Discus, EMBASE, CINAHL, and Web of Science. There were 346 studies included for the title/abstract screening. Sixty-seven studies were included for full text review. Of these, 27 studies were excluded for reporting only fixed length short form versions of CATs (n = 17), failing to report data from CATs (n = 6), full text unavailable (n = 3), and less than 50% of the patient sample had a knee condition (n = 1). Forty studies met the inclusion criteria and were included in this review. The most frequently implicated domains at risk for bias included selection bias, failure to adjust for confounding variables, and doubtful methodologies used to estimate of reliability and measurement error. Three unique CAT tools were identified from the search: PROMIS CAT, FOTO Knee FS, and OA-CAT. The PROMIS CAT scales demonstrated moderate to strong correlations to a variety of fixed length measures used to evaluate patients with knee conditions. CATs were administered to multiple patient populations and used for a variety of purposes. All 3 CATs demonstrated sound psychometric properties and were more efficient than traditional fixed length PROMs without sacrificing measurement precision. Studies on the PROMIS CAT scales and FOTO Knee FS measure have identified clinically relevant values, such as the MCID which can assist clinicians and patients in shared decision-making about treatment. As such, the PROMIS CAT measures and FOTO Knee FS measure are recommended for clinical use in patients with knee conditions. This review had several limitations including selection bias and selective reporting bias of included studies as well as publication bias. Publication trends suggest PROMIS CAT literature is increasing year after year. This trend was not observed with the FOTO Knee FS measure or OA-CAT scales. Future research should continue to validate PROMIS CAT scales in specific knee populations and young, high functioning patient cohorts.
Products
I began conducting a systematic review of the literature on the utilization of CATs in patients with knee conditions on December 3, 2019. I constructed a manuscript based on my findings. The second linked document contains the tables and figures referenced in the manuscript. Tables are frequently referenced in the manuscript. As such, it is recommended that both the manuscript and accompanying tables and figures be viewed, simultaneously. The results of this review are intended to be clinically useful for healthcare professionals and informative for health outcomes research. I also created an infographic to address health literacy. The infographic is intended to inform patients and clinicians of the purpose and potential benefits of using CAT tools to collect patient-reported data. The final products are linked below.
Next Steps
The ultimate goal of this project was to prepare a manuscript for journal submission. While I am ultimately proud of the quality of the current products, further revision and peer review is necessary before formal submission to a journal is considered. I have also discussed the possibility of preparing an abstract poster presentation for the 2021 Combined Sections Meeting. I will continue to consult with my advisor and committee members while I consider these options.
Evaluation
Evaluation of this capstone project came in the form of discussion and formal editing by my advisor (Dr. Thoma), peers (Emma Shirley and Gabe Dimock), and my capstone committee members (Dr. Debby Givens, PT, PhD and Dr. Carla Hill, PT, OCS). The manuscript was subject to several revisions throughout the course of the semester. After completing each section, the manuscript was independently reviewed twice, once by my advisor and once by a peer. At midterm, the drafts of the Introduction, Methods, and Results sections were sent to my advisor and my committee members for review. The full manuscript draft and infographic were sent to my advisor, my peer review group, and my capstone committee before final submission.
Self-Reflection
I have learned a great deal throughout the development of this project. I have gained practical experience conducting a systematic review and have developed a deeper appreciation for the rigor and level of detail that goes into research. This project was a massive undertaking that spanned almost 2 full semesters. I’ll be the first to admit that I was terrified when our original plan (see background above) got derailed and we decided to conduct 3 separate reviews. I didn’t think myself capable of being the lead author on a research project and certainly not so soon in my career. However, I stepped out of my comfort zone, tried something new and challenging, and ultimately created products that I’m really proud of. I’ve greatly improved my academic writing skills as a result of extensive peer reviewing and reading research. In the same vein, I’ve learned to accept (and appreciate) that my first draft will be drastically different than my final draft; I am indebted to the revision process. Working with an extremely intelligent group of people was a key to my success.
This project reinforced the importance of planning, preparation, and organization. As a result of reading similar reviews and taking bits and pieces from their data extraction methods, I was able to formulate a clearly organized plan for my own data extraction. I have a better understanding and appreciation for the complexity of psychometric properties—though I still have much to learn. I feel more confident in my ability to critically analyze and synthesize relatively large bodies of evidence and identify gaps in the literature. Conducting research is difficult, time-consuming, and requires thinking about the big picture and small details, concurrently. I’m looking forward to any opportunities I have to conduct research in the future.
Countless hours, over 17,000 words, 71 pages, 64 references, 40 included studies, 2 semesters, and 1 infographic later, I’ve finally finished one of my biggest endeavors to date. Some final takeaways I had are to create a timeline and stick to it, put in work every day, click the save icon frequently, be as organized as possible, and use a reference manager.
Acknowledgements
This project would not have been possible without the contribution of time and effort from an outstanding group of faculty, clinicians, and colleagues. First, I would like to thank my outstanding advisor, Dr. Louise Thoma for the opportunity to gain practical experience conducting research. Dr. Thoma has been extremely patient with me throughout each stage of the project and has been pivotal in providing me feedback, direction, and clarity. This project would not have come to fruition without her guidance and expertise.
I would like to acknowledge my all-star capstone committee members, Dr. Debby Givens and Dr. Carla Hill. Both Dr. Givens and Dr. Hill have provided me with timely and valuable feedback amidst fulfilling countless other obligations. Dr. Givens has been our fearless leader and biggest advocate over the past 3 years and through a pandemic. Dr. Hill has an impressive clinical skillset and has been an excellent clinical mentor to me.
I would also like to recognize my impressive peer reviewers and fellow classmates, Gabe Dimock and Emma Shirley. Their constructive feedback, encouragement, and comradery throughout each stage of the project motivated me to put my best foot forward and create quality products.
I would also like to thank my amazing wife, Rachael, and my dog Sage, who have provided endless hugs, encouragement, and support throughout this project and my Doctorate education.
Last but certainly not least, I would like to thank the outstanding UNC DPT faculty and my 2020 cohort-family for everything over the past 3 years. The UNC DPT faculty are a world class group of professionals who have poured their hearts and souls into developing us as physical therapists. To the class of 2020, we have laughed together, we invented a muscle as DPT1s, and we repeatedly clapped for each other for no reason at all, over the past 3 years. You all are incredible human beings and extremely capable healthcare professionals. I’m honored to have been in the trenches with you and I know that there will be more chapters in our book.
Best regards,
Tanner Holden
References
- Beaton DE, Schemitsch E. Measures of health-related quality of life and physical function. Clin. Orthop. Relat. Res. 2003;(413):90-105. doi:10.1097/01.blo.0000079772.06654.c8.
- Ayers DC. Implementation of Patient-reported Outcome Measures in Total Knee Arthroplasty. J Am Acad Orthop Surg 2017;25 Suppl 1:S48-S50. doi:10.5435/JAAOS-D-16-00631.
- Fidai MS, Saltzman BM, Meta F, et al. Patient-Reported Outcomes Measurement Information System and Legacy Patient-Reported Outcome Measures in the Field of Orthopaedics: A Systematic Review. Arthroscopy 2018;34(2):605-614. doi:10.1016/j.arthro.2017.07.030.
- Wang D, Jones MH, Khair MM, Miniaci A. Patient-reported outcome measures for the knee. J. Knee Surg. 2010;23(3):137-151. doi:10.1055/s-0030-1268691.
- Brodke DJ, Saltzman CL, Brodke DS. PROMIS for orthopaedic outcomes measurement. J Am Acad Orthop Surg 2016;24(11):744-749. doi:10.5435/JAAOS-D-15-00404.
- Bodart S, Byrom B, Crescioni M, Eremenco S, Flood E. Perceived Burden of Completion of Patient-Reported Outcome Measures in Clinical Trials:: Results of a Preliminary Study. Ther. Innov. Regul. Sci. 2019;53(3):318-323. doi:10.1177/2168479018788053.
- Scott EJ, Westermann R, Glass NA, Hettrich C, Wolf BR, Bollier MJ. Performance of the PROMIS in patients after anterior cruciate ligament reconstruction. Orthop. J. Sports Med. 2018;6(5):2325967118774509. doi:10.1177/2325967118774509.
- Cella D, Yount S, Rothrock N, et al. The patient-reported outcomes measurement information system (PROMIS): Progress of an NIH roadmap cooperative group during its first two years. Med. Care 2007;45(5 Suppl 1):S3-S11. doi:10.1097/01.mlr.0000258615.42478.55.
- Hart DL, Wang Y-C, Stratford PW, Mioduski JE. Computerized adaptive test for patients with knee impairments produced valid and responsive measures of function. J. Clin. Epidemiol. 2008;61(11):1113-1124. doi:10.1016/j.jclinepi.2008.01.005.
- Bhatt S, Boody BS, Savage JW, Hsu WK, Rothrock NE, Patel AA. Validation of Patient-reported Outcomes Measurement Information System Computer Adaptive Tests in Lumbar Disk Herniation Surgery. J Am Acad Orthop Surg 2019;27(3):95-103. doi:10.5435/JAAOS-D-17-00300.
- Boody BS, Bhatt S, Mazmudar AS, Hsu WK, Rothrock NE, Patel AA. Validation of Patient-Reported Outcomes Measurement Information System (PROMIS) computerized adaptive tests in cervical spine surgery. J Neurosurg Spine 2018;28(3):268-279. doi:10.3171/2017.7.SPINE17661.
- Haws BE, Khechen B, Bawa MS, et al. The Patient-Reported Outcomes Measurement Information System in spine surgery: a systematic review. J Neurosurg Spine 2019;30(3):405-413. doi:10.3171/2018.8.SPINE18608.
- Jayakumar P, Overbeek C, Vranceanu AM, et al. The use of computer adaptive tests in outcome assessments following upper limb trauma: a systematic review. Bone Joint J. 2018;100-B(6):693-702. doi:10.1302/0301-620X.100B6.BJJ-2017-1349.R1.
- Gulledge CM, Lizzio VA, Smith DG, Guo E, Makhni EC. What Are the Floor and Ceiling Effects of Patient-Reported Outcomes Measurement Information System Computer Adaptive Test Domains in Orthopaedic Patients? A Systematic Review. Arthroscopy 2020. doi:10.1016/j.arthro.2019.09.022.
Image references:
- EHR 2.0: How New Clinical Decision Support Tools Are Helping Doctors Make Smarter Decisions at the Point of Care | Mercy Technology Services. https://www.mercytechnology.net/insights/blog/clinical-decision-support. Accessed April 16, 2020.
- Knee Osteoarthritis: Symptoms, Diagnosis & Stages. https://www.pthealth.ca/blog/knee-osteoarthritis-my-knees-hurt-do-i-have-oa/. Accessed April 15, 2020.
- Waiting Room Ipad Stock Photos, Pictures & Royalty-Free Images – iStock. https://www.istockphoto.com/photos/waiting-room-ipad?mediatype=photography&phrase=waiting%20room%20ipad&sort=mostpopular. Accessed April 16, 2020.
- The PROMIS in Putting Patients’ Waiting Room Time to Good Use | PCORI. https://www.pcori.org/research-results/pcori-stories/promis-putting-patients-waiting-room-time-good-use. Accessed April 15, 2020.
4 Responses to “Utilization of Computer Adaptive Tests in Patients with Knee Conditions: A Systematic Review”
Tanner
Conor,
Thank you for taking the time to write such a thoughtful message. I appreciate the encouragement you gave along the way, reminding me that I was capable of completing the project and that things would turn out well. Comments from you and others certainly motivated me to keep pushing through.
Regarding the topic, I do think they have a valuable place in both clinical and research settings. The IRT-based methodologies and dynamic nature of the item banks are characteristics that give these instruments seemingly unlimited potential in capturing a spectrum of patient abilities.
Regarding the infographic, I fully agree that an additional infographic or presentation targeted specifically for orthopedic clinicians or healthcare entities would be a valuable addition to this project. This addition would likely be pretty easy to put together and could explain in more detail the mechanisms behind item response theory, item bank development, and the promising psychometric properties. I could also include logistics such as the cost to implement and maintain access to data management services. Thank you for the extremely valuable feedback.
Thank you for your high praise. It means a great deal to me.
Tanner
Debbie,
Thank you for your extremely kind words. I will be meeting with my advisor, Dr. Thoma on Friday (via Zoom, of course) to discuss the next steps regarding further revision and journal submission.
Conor McClure
Tanner,
I can speak from experience sitting next to you in our teaching assistant elective and in the library that you put in a lot of blood, sweat, and tears (mostly tears) into this project. No one can question your work ethic and dedication to putting out a good product. I remember talking you off the ledge a couple times with how deep in the weeds you were with this review. I also remember confronting your pathological imposter syndrome a couple times as you worried about your project not being good enough—as you often do, with no good reason—but look at how far you’ve come. This product is not only well-written and absurdly and unbelievably thorough, it’s basically publishable! That’s more than I can say about my own project.
I know you were a little disappointed, as we all were, that this project didn’t follow the original ACL-focused path. That said, I think the outcome of your product is still very useful for the orthopedic physical therapy population. Despite priding myself on being a reasonably-competent orthopedic-minded student, I learned quite a bit reading through your manuscript (an arduous and harrowing task, for sure.) I had to constantly mentally correct my tendency to translate CAT into “Critically-Appraised Topic” and PROMs into “Passive Range of Motion.” Going into this project, I think the only CAT I was familiar with was FOTO, which was similar to your experience. I’ve never actually used any of these in the clinic, but having read through your project, I’m reasonably convinced that they need to be more widely implemented.
Your infographic was useful and pretty, though an argument could be made that it was almost too simple. I realize this was created as part of the health literacy component of the project, so I can understand the need to use patient-centered language and consider health and scientific literacy of the average patient population. I wonder if a separate infographic could be created and tailored more specific to orthopedic clinicians? I also agree that the creation of an abstract-style poster presentation for use at a national conference would be warranted further down the road. Finding an engaging method to summarize the findings of your massive manuscript would be great.
To sum up, I really can’t say this enough: you did an amazing job! I’m frankly astounded by the depth, length, and thoroughness of your final product(s), to the point where I think I’m acquiring some inadequacy syndrome with every word I write. Your self-reflection is refreshingly honest and inquisitive. I’m glad to see you confronting some of those insidious feelings that convince us we’re not good enough or capable enough to take on a project like this. You should be proud, buddy.
– Conor
Debbie Thorpe
Tanner
Wow…what an undertaking! Your systematic review reads really well and the figures and tables you provided support the manuscript well. Your CAT Infographic will be a valuable resource for clinicians and patients. I hope that you plan on submitting this systematic review for publication. You should be very proud of this accomplishment.
Best Wishes….