Voice-Controlled Electrosurgical Generator

  Watch the Demo Video    Download the Project Poster

Project Sponsor:Medtronic

The Voice-controlled Electro-surgical Generator (VEG) solves a two-faceted problem:

The first facet is collecting large amounts of unbiased, annotated data related to their Electro-Surgical Generators (ESGs). Medtronic currently collects data in two ways, but neither method results in data that can be used in a meaningful way. The first method occurs at the Medtronic site. They collect small amounts of well-annotated data by using their tools on tissue samples and other materials. This process yields small amounts of data that cannot be considered unbiased because it only includes a few types of tissue and is performed in a very controlled environment. The second method occurs in the Operating Room (OR), where the Medtronic ESGs collect RF data while the tool is in use. This process generates large amounts of unbiased data, but the data is not annotated. Without other information, such as video and audio, there is very little context with which to annotate it. This leaves Medtronic in an unfortunate position because despite collecting large amounts of data using multiple methods, none of it can be used in a meaningful way. To solve this issue, the VEG includes a Data Acquisition board which synchronously collects and stores the ESG RF data along with video, audio, and other analog sensor data during surgeries. The context provided by the video and audio streams allows humans to annotate the unbiased data being collected by ESGs in the OR. This enables Medtronic to design artificial intelligence to improve the capabilities of its ESGs. The training process is enhanced by a GUI which overlays sensor data on the video and audio streams, allowing the user to examine audiovisual data to locate noteworthy events and inspect analog sensor data and RF data at and before moments of interest.

The second facet is the inconvenience of changing settings on the ESG. Oftentimes surgeons require power changes for different conditions encountered during surgery. As the surgeon is actively engaged in surgery, he/she cannot touch anything outside of the sterile field, which includes the ESG. Thus, a surgeon must ask a nurse outside the sterile field to change the settings. The voice-control aspect of this project streamlines communication by giving the surgeon direct control over the ESG without interrupting the surgery. The PC-application provides voice control using off the shelf voice recognition software to allow the doctor to alter the generator settings directly rather than having a nurse operate as a middleman.

Doctors and patients in an operating room benefit from both more effective settings configuration and increased safety introduced by AI running on the ESGs. 

The VEG introduces many features that are not available in the current generation of electrosurgical generators. Time-series RF data is currently available to Medtronic, but without contextual labeling and other data streams an AI model cannot accurately fit the available data. The VEG provides this capability to analysts by synchronously collecting other streams of data and allowing ad-hoc, online, voice-labeling of the data. The VEG is the first electrosurgical generator capable of utilizing AI to understand the RF data it collects, thus allowing it to prevent issues that could occur as a result of improper use of equipment in real-time. Additionally, surgeons are able to change ESG settings mid-operation without assistance, reducing the potential for miscommunications and delays during time-sensitive operations and reducing the number of staff needed in the OR.