Title: Automatic detection of potentially ineffective verbal communication for training through simulation

Abstract

Objectives: Artificial Intelligence models can extract human-factor relatedinformation from audio recordings. We present an automatic workflow detecting dialogue segments with potentially ineffective communicationbetween team membersduring neonatal simulation sessions. Materials and Methods: 10 cases from historical audio recordings of neonatalsimulation training sessions at Centro NINA, Maternal-Neonatal Department,AziendaOspedaliero-UniversitariaPisana (Pisa, Italy) were selected. The workflow analysed syllabic-scale (100-200 ms)spoken dialogue energy and intonation, using cluster analysis based on the K-means algorithm1.Tone units were detected, their audio segmentsextracted and, through cluster analysis of energy and pitch, labelled as either potentially ineffective or viable verbal communication. The audio of potentially ineffective unitswas transcribed through an automatic speech recogniser, and keywords extracted to produce a word cloud.Performance was measured against a gold standard containing annotations of 79 minutes of audio recordings from neonatal simulations, in Italian, under different noise conditions (from 4.63 to 14.17 SNR), compiled by two researchers with complementary expertise in the field. Results: Our workflow achieved a detection accuracy of 64% against a commercial automatic speech recogniser sentence accuracy of 9.37%. Detected keyword viability - the percentage of gold-standard words contained in the word cloud - was 59%.Potentially ineffective communication keywords includedrepeated items in the first person plural, and expressions of uncertainty, which may point toissues around leadership, self-confidence, and/or instruction clarity. There was no reference to time- or equipment/setting issues. Conclusion: Our workflow successfully identifiedeffective/ineffective communication during neonatal simulation sessions. It can be applied to other languages than Italian and can help trainersrefine feedback and measure learning improvement.

Biography

Dr. NicolettaFossatiwith a medical degree and a PhD, both from Pisa University and S. Anna School of Advanced Studies, complemented by a specialty degree in Anaesthesia and Resuscitation, She has held a consultant level job in Italy for 10 years before moving to London in 2004 to take on a Consultant Anaesthetist post at St George’s Hospital. In addition to anaesthesia, she has a long-standing interest in medical education. An Honorary Reader in Clinical Education and Anaesthesia at St George’s, University of London, she holds a Master’s degree in Higher and Professional Education (Institute of Education/UCL). In July 2022 she was awarded the Senior Fellowship of Advance HE/Higher Education Academy (SFHEA).