publications

2024

1. IVA

   Keeping Users Engaged During Repeated Interviews by a Virtual Agent: Using Large Language Models to Reliably Diversify Questions

   Hye Sun Yun, Mehdi Arjmand, Phillip Sherlock, and 3 more authors

   In Proceedings of the 24th ACM International Conference on Intelligent Virtual Agents, 2024

   PDF
2. IVA

   Investigating User Perceptions of Collaborative Agenda Setting in a Virtual Health Counseling Session

   Mina Fallah, Farnaz Nouraei, Hye Sun Yun, and 1 more author

   In Proceedings of the 24th ACM International Conference on Intelligent Virtual Agents, 2024

   PDF
3. MLHC

   Automatically Extracting Numerical Results from Randomized Controlled Trials with Large Language Models

   Hye Sun Yun, David Pogrebitskiy, Iain J Marshall, and 1 more author

   In Machine Learning for Healthcare Conference, 2024

   PDF

2023

1. JCE

   In a pilot study, automated real-time systematic review updates were feasible, accurate, and work-saving

   Iain J. Marshall, Thomas A. Trikalinos, Frank Soboczenski, and 4 more authors

   Journal of Clinical Epidemiology, 2023

   Abs HTML

   Objectives The aim of this study is to describe and pilot a novel method for continuously identifying newly published trials relevant to a systematic review, enabled by combining artificial intelligence (AI) with human expertise. Study Design and Setting We used RobotReviewer LIVE to keep a review of COVID-19 vaccination trials updated from February to August 2021. We compared the papers identified by the system with those found by the conventional manual process by the review team. Results The manual update searches (last search date July 2021) retrieved 135 abstracts, of which 31 were included after screening (23% precision, 100% recall). By the same date, the automated system retrieved 56 abstracts, of which 31 were included after manual screening (55% precision, 100% recall). Key limitations of the system include that it is limited to searches of PubMed/MEDLINE, and considers only randomized controlled trial reports. We aim to address these limitations in future. The system is available as open-source software for further piloting and evaluation. Conclusion Our system identified all relevant studies, reduced manual screening work, and enabled rolling updates on publication of new primary research.

2. CHI

   Improving Multiparty Interactions with a Robot Using Large Language Models

   Prasanth Murali, Ian Steenstra, Hye Sun Yun, and 2 more authors

   In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, 2023

   Abs PDF

   Speaker diarization is a key component of systems that support multiparty interactions of co-located users, such as meeting facilitation robots. The goal is to identify who spoke what, often to provide feedback, moderate participation, and personalize responses by the robot. Current systems use a combination of acoustic (e.g. pitch differences) and visual features (e.g. gaze) to perform diarization, but involve the use of additional sensors or require overhead signal processing efforts. Alternatively, automatic speech recognition (ASR) is a necessary step in the diarization pipeline, and utilizing the transcribed text to directly identify speaker labels in the conversation can eliminate such challenges. With that motivation, we leverage large language models (LLMs) to identify speaker labels from transcribed text and observe an exact match of 77% and a word level accuracy of 90%. We discuss our findings and the potential use of LLMs as a diarization tool for future systems.

3. EMNLP

   Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviews

   Hye Sun Yun, Iain J Marshall, Thomas Trikalinos, and 1 more author

   arXiv preprint arXiv:2305.11828, 2023

   PDF

2022

1. IUI HEALTHI

   Techno-Spiritual Engagement: Mechanisms for Improving Uptake of mHealth Apps Designed for Church Members

   Hye Sun Yun, Shou Zhou, Everlyne Kimani, and 7 more authors

   In Joint Proceedings of the IUI 2022 Workshops: APEx-UI, HAI-GEN, HEALTHI, HUMANIZE, TExSS, SOCIALIZE co-located with the ACM International Conference on Intelligent User Interfaces (IUI 2022), Virtual Event, Helsinki, Finland, March 21-22, 2022, 2022

   Abs PDF Slides

   Keeping users engaged with mHealth applications is important but difficult to achieve. We describe the development of a smartphone-based application designed to promote health and wellness in church communities, along with mechanisms explicitly designed to maintain engagement. We evaluated religiously tailored techno-spiritual engagement mechanisms, including a prayer posting wall, pastor announcements, an embodied conversational agent for dialogue-based scriptural reflections and health coaching, and tailored push notifications. We conducted a four-week pilot study with 25 participants from two churches, measuring high levels of participant acceptance and satisfaction with all features of the application. Engagement with the app was higher for users considered to be more religious and correlated with the number of notifications received. Our findings demonstrate that our tailored mechanisms can increase engagement with an mHealth app.

2. Church after Sunday: supporting everyday well-being through techno-spiritual health interventions

   Teresa K O’Leary, Elizabeth Stowell, Darley Sackitey, and 5 more authors

   Interactions, 2022

   PDF
3. IVA

   Motivating Health Behavior Change with a Storytelling Virtual Agent

   Hye Sun Yun, Matias Volonte, and Timothy Bickmore

   In Proceedings of the 22nd ACM International Conference on Intelligent Virtual Agents, 2022

   Abs PDF

   We developed a virtual agent that motivates church-going users to change their health behavior by telling existing cultural narratives that have high relevance with the counseling topic in an engaging way. We evaluated this agent in a between-subjects experiment where participants interacted with an agent that counseled them on nutrition either without a story, with a story but told in a neutral speech style, or with a story using dramatic delivery inspired by church sermons. We found that interaction with either one of the storytelling agents leads to a significantly greater change in confidence to engage in the target behavior of healthy eating than interacting with a non-storytelling agent, demonstrating the efficacy of stories in health counseling by virtual agents.