Could crowdsourcing end logjam of surgical skills evaluations?

News Archive

Could crowdsourcing end logjam of surgical skills evaluations?

Studies lend rationale to letting Amazon marketplace score doctors on technical prowess
Brian Donohue

“I’ve never played pro basketball, but I can tell the best player on the court by watching their performance.”

This notion led Dr. Thomas Lendvay and other researchers to explore whether a crowdsourcing marketplace at Amazon could adequately assess surgeons’ technical skills.

Their findings, published in the Journal of Surgical Research and the Journal of Endourology, show remarkable agreement between the evaluations of laypeople and those of surgical experts upon seeing videos of surgeons’ hands performing practice procedures. 

What’s more, in one of the studies, it took just 10 hours to collect feedback from 1,500 laypeople versus more than three weeks from three busy surgeons. Further, 90% of laypersons’ evaluations included contextual comments to explain ratings, versus 20% of surgeons’ evaluations.

Clare McLean
Urologist Dr. Tom Lendvay demonstrates a surgical robotic system. He is advocating to change the process of technical-skills evaluations for surgeons.
picture of Dr. Tom Lendvay using a surgical robotic system

 Lendvay, an associate professor of urology at the University of Washington, seeks to relieve a bottleneck that affects the development of surgeons-in-training and the improvement of those with established practices.

Surgeons acquire technical acumen through repeated practice procedures and during training apprenticeships. Surgeons must be certified to practice, but certifications may be based on subjective, inconsistent and potentially biased judgment of mentors and peers.

Assessment of technical skills via video, with the reviewer blinded to the performer’s identity, makes an evaluation more subjective but it doesn’t create more time in the day for reviewing surgeons whose other tasks, namely patient care, take priority.

“We generate all these videos of surgical performances, but experts don’t have enough time to sit and watch and score them,” Lendvay said. “The assessment tools have been validated as an objective way to rate a surgeon’s skills, but they are so onerous to fill out.” 

The reviewers scored the videos on a 1-5 scale in six dimensions such as “respect for tissue,” “instrument handling,” and “flow of operation.”

The difference between the two groups’ scores of videos was not statistically significant.

Outsourcing surgical skills evaluations would appear to benefit not only patients and surgeons, but hospitals, too, given recent insurance mandates in which patients’ outcomes chiefly determine the reimbursements to doctors and hospitals. 

Clare McLean
Gynecologic surgeons perform a procedure via robotic-assisted device at UW Medical Center. The instrument enables minimally-invasive approaches to surgery, a growing trend at hospitals.
picture of two surgeons operating via robotic-assisted device

In parallel, urban hospitals have quickly moved to deploy robotic-assisted instruments that allow more surgeries to be done via tiny-incision, minimally invasive approaches rather than large open incisions. These instruments extend surgeons’ deft handiwork into tight anatomic spaces, but they require additional training to master because the surgeon operates via a console several feet from the patient.

Because the minimally invasive surgeries are associated with quicker patient recoveries and better outcomes, competency with the instruments among young residents and experienced surgeons will be increasingly prized – thus the value of timely training-video evaluations and feedback.

In terms of cost, Lendvay said, paying prequalified crowdworkers a nominal, per-review fee makes sense, given the costs of an expert surgeon’s time and the prospective savings from “reduced patient complications from surgeons whose evaluations indicate an opportunity to improve.” 

Lendvay’s model has promise, but limitations, too, said Dr. Carlos Pellegrini, professor and chair of UW's Department of Surgery. 

Courtesy of Dr. Tom Lendvay
(Click to enlarge.) Surgical-skills reviewers see this interface of a video and a series of questions.
picture of surgical-skills evaluation user interface

“This is a measure of technical skill, but does not account for a surgeon’s judgment. Novices watching videos would have no awareness of whether the correct action is being performed, however skillfully it is done. Technique comprises only part of a surgeon’s competence,” Pellegrini said.

Two other studies have buttressed Lendvay’s findings:

  • The New England Journal of Medicine published a seminal report in 2013 demonstrating that, among bariatric surgeons, better technical skill – as determined by scored reviews of videotaped procedures – correlates with fewer post-operative complications for patients.
  • At the European Urology Association annual meeting, University of Michigan researchers presented findings indicating that experts and crowds agreed on the rankings of the bottom five surgeons among 12 practicing urologists whose robotic prostatectomy videos were reviewed. 

This cluster of research makes a strong case for a faster, entirely unbiased model to screen surgeons, Lendvay said, which could bolster skills-coaching initiatives and ultimately improve patient care.

Lendvay has a financial stake in the outsourcing model’s success. He is working with CoMotion at UW, which has provided funding and incubation space to commercialize the video-evaluation system. Lendvay is a co-founder of the resulting UW startup C-SATS (Crowd-Sourced Assessment of Technical Skills), which recently raised $2.5 million to build out its cloud-based technology.