Could crowdsourcing end logjam of surgical skills evaluations?
“I’ve never played pro basketball, but I can tell the best player on the court by watching their performance.”
This notion led Dr. Thomas Lendvay and other researchers to explore whether a crowdsourcing marketplace at Amazon could adequately assess surgeons’ technical skills.
Their findings, published in the Journal of Surgical Research and the Journal of Endourology, show remarkable agreement between the evaluations of laypeople and those of surgical experts upon seeing videos of surgeons’ hands performing practice procedures.
What’s more, in one of the studies, it took just 10 hours to collect feedback from 1,500 laypeople versus more than three weeks from three busy surgeons. Further, 90% of laypersons’ evaluations included contextual comments to explain ratings, versus 20% of surgeons’ evaluations.
Lendvay, an associate professor of urology at the University of Washington, seeks to relieve a bottleneck that affects the development of surgeons-in-training and the improvement of those with established practices.
Surgeons acquire technical acumen through repeated practice procedures and during training apprenticeships. Surgeons must be certified to practice, but certifications may be based on subjective, inconsistent and potentially biased judgment of mentors and peers.
Assessment of technical skills via video, with the reviewer blinded to the performer’s identity, makes an evaluation more subjective but it doesn’t create more time in the day for reviewing surgeons whose other tasks, namely patient care, take priority.
“We generate all these videos of surgical performances, but experts don’t have enough time to sit and watch and score them,” Lendvay said. “The assessment tools have been validated as an objective way to rate a surgeon’s skills, but they are so onerous to fill out.”
The reviewers scored the videos on a 1-5 scale in six dimensions such as “respect for tissue,” “instrument handling,” and “flow of operation.”
The difference between the two groups’ scores of videos was not statistically significant.
Outsourcing surgical skills evaluations would appear to benefit not only patients and surgeons, but hospitals, too, given recent insurance mandates in which patients’ outcomes chiefly determine the reimbursements to doctors and hospitals.
In parallel, urban hospitals have quickly moved to deploy robotic-assisted instruments that allow more surgeries to be done via tiny-incision, minimally invasive approaches rather than large open incisions. These instruments extend surgeons’ deft handiwork into tight anatomic spaces, but they require additional training to master because the surgeon operates via a console several feet from the patient.
Because the minimally invasive surgeries are associated with quicker patient recoveries and better outcomes, competency with the instruments among young residents and experienced surgeons will be increasingly prized – thus the value of timely training-video evaluations and feedback.
In terms of cost, Lendvay said, paying prequalified crowdworkers a nominal, per-review fee makes sense, given the costs of an expert surgeon’s time and the prospective savings from “reduced patient complications from surgeons whose evaluations indicate an opportunity to improve.”
Lendvay’s model has promise, but limitations, too, said Dr. Carlos Pellegrini, professor and chair of UW's Department of Surgery.
“This is a measure of technical skill, but does not account for a surgeon’s judgment. Novices watching videos would have no awareness of whether the correct action is being performed, however skillfully it is done. Technique comprises only part of a surgeon’s competence,” Pellegrini said.
Two other studies have buttressed Lendvay’s findings:
- The New England Journal of Medicine published a seminal report in 2013 demonstrating that, among bariatric surgeons, better technical skill – as determined by scored reviews of videotaped procedures – correlates with fewer post-operative complications for patients.
- At the European Urology Association annual meeting, University of Michigan researchers presented findings indicating that experts and crowds agreed on the rankings of the bottom five surgeons among 12 practicing urologists whose robotic prostatectomy videos were reviewed.
This cluster of research makes a strong case for a faster, entirely unbiased model to screen surgeons, Lendvay said, which could bolster skills-coaching initiatives and ultimately improve patient care.
Lendvay has a financial stake in the outsourcing model’s success. He is working with CoMotion at UW, which has provided funding and incubation space to commercialize the video-evaluation system. Lendvay is a co-founder of the resulting UW startup C-SATS (Crowd-Sourced Assessment of Technical Skills), which recently raised $2.5 million to build out its cloud-based technology.