Design and Evaluation Challenges of Conversational Agents in Health Care and Well-being: Selective Review Study

Ahmet Baki Kocaballi; Emre Sezgin; Leigh Clark; John M Carroll; Yungui Huang; Jina Huh-Yoo; Junhan Kim; Rafal Kocielnik; Yi-Chieh Lee; Lena Mamykina; Elliot G Mitchell; Robert J Moore; Prasanth Murali; Elizabeth D Mynatt; Sun Young Park; Alessandro Pasta; Deborah Richards; Lucas M Silva; Diva Smriti; Brendan Spillane; Zhan Zhang; Tamara Zubatiy

doi:10.2196/38525

Design and Evaluation Challenges of Conversational Agents in Health Care and Well-being: Selective Review Study

Ahmet Baki Kocaballi, Emre Sezgin, Leigh Clark, John M Carroll, Yungui Huang, Jina Huh-Yoo, Junhan Kim, Rafal Kocielnik, Yi-Chieh Lee, Lena Mamykina, Elliot G Mitchell, Robert J Moore, Prasanth Murali, Elizabeth D Mynatt, Sun Young Park, Alessandro Pasta, Deborah Richards, Lucas M Silva, Diva Smriti, Brendan SpillaneZhan Zhang, Tamara Zubatiy

Department of Applied Mathematics and Computer Science

Research output: Contribution to journal › Review › peer-review

62 Downloads (Pure)

Abstract

BACKGROUND: Health care and well-being are 2 main interconnected application areas of conversational agents (CAs). There is a significant increase in research, development, and commercial implementations in this area. In parallel to the increasing interest, new challenges in designing and evaluating CAs have emerged.

OBJECTIVE: This study aims to identify key design, development, and evaluation challenges of CAs in health care and well-being research. The focus is on the very recent projects with their emerging challenges.

METHODS: A review study was conducted with 17 invited studies, most of which were presented at the ACM (Association for Computing Machinery) CHI 2020 conference workshop on CAs for health and well-being. Eligibility criteria required the studies to involve a CA applied to a health or well-being project (ongoing or recently finished). The participating studies were asked to report on their projects' design and evaluation challenges. We used thematic analysis to review the studies.

RESULTS: The findings include a range of topics from primary care to caring for older adults to health coaching. We identified 4 major themes: (1) Domain Information and Integration, (2) User-System Interaction and Partnership, (3) Evaluation, and (4) Conversational Competence.

CONCLUSIONS: CAs proved their worth during the pandemic as health screening tools, and are expected to stay to further support various health care domains, especially personal health care. Growth in investment in CAs also shows the value as a personal assistant. Our study shows that while some challenges are shared with other CA application areas, safety and privacy remain the major challenges in the health care and well-being domains. An increased level of collaboration across different institutions and entities may be a promising direction to address some of the major challenges that otherwise would be too complex to be addressed by the projects with their limited scope and budget.

Original language	English
Article number	e38525
Journal	Journal of Medical Internet Research
Volume	24
Issue number	11
Number of pages	15
ISSN	1439-4456
DOIs	https://doi.org/10.2196/38525
Publication status	Published - 15 Nov 2022

Keywords

Humans
Aged
Communication
Delivery of Health Care
Health Personnel

Access to Document

10.2196/38525

FulltextFinal published version, 209 KB

OpenUrl availability

Full text

Cite this

Kocaballi, A. B., Sezgin, E., Clark, L., Carroll, J. M., Huang, Y., Huh-Yoo, J., Kim, J., Kocielnik, R., Lee, Y.-C., Mamykina, L., Mitchell, E. G., Moore, R. J., Murali, P., Mynatt, E. D., Park, S. Y., Pasta, A., Richards, D., Silva, L. M., Smriti, D., ... Zubatiy, T. (2022). Design and Evaluation Challenges of Conversational Agents in Health Care and Well-being: Selective Review Study. Journal of Medical Internet Research, 24(11), Article e38525. https://doi.org/10.2196/38525

@article{572e36ad95d74344ae1c18c2859007bc,

title = "Design and Evaluation Challenges of Conversational Agents in Health Care and Well-being: Selective Review Study",

abstract = "BACKGROUND: Health care and well-being are 2 main interconnected application areas of conversational agents (CAs). There is a significant increase in research, development, and commercial implementations in this area. In parallel to the increasing interest, new challenges in designing and evaluating CAs have emerged.OBJECTIVE: This study aims to identify key design, development, and evaluation challenges of CAs in health care and well-being research. The focus is on the very recent projects with their emerging challenges.METHODS: A review study was conducted with 17 invited studies, most of which were presented at the ACM (Association for Computing Machinery) CHI 2020 conference workshop on CAs for health and well-being. Eligibility criteria required the studies to involve a CA applied to a health or well-being project (ongoing or recently finished). The participating studies were asked to report on their projects' design and evaluation challenges. We used thematic analysis to review the studies.RESULTS: The findings include a range of topics from primary care to caring for older adults to health coaching. We identified 4 major themes: (1) Domain Information and Integration, (2) User-System Interaction and Partnership, (3) Evaluation, and (4) Conversational Competence.CONCLUSIONS: CAs proved their worth during the pandemic as health screening tools, and are expected to stay to further support various health care domains, especially personal health care. Growth in investment in CAs also shows the value as a personal assistant. Our study shows that while some challenges are shared with other CA application areas, safety and privacy remain the major challenges in the health care and well-being domains. An increased level of collaboration across different institutions and entities may be a promising direction to address some of the major challenges that otherwise would be too complex to be addressed by the projects with their limited scope and budget.",

keywords = "Humans, Aged, Communication, Delivery of Health Care, Health Personnel",

author = "Kocaballi, {Ahmet Baki} and Emre Sezgin and Leigh Clark and Carroll, {John M} and Yungui Huang and Jina Huh-Yoo and Junhan Kim and Rafal Kocielnik and Yi-Chieh Lee and Lena Mamykina and Mitchell, {Elliot G} and Moore, {Robert J} and Prasanth Murali and Mynatt, {Elizabeth D} and Park, {Sun Young} and Alessandro Pasta and Deborah Richards and Silva, {Lucas M} and Diva Smriti and Brendan Spillane and Zhan Zhang and Tamara Zubatiy",

note = "{\textcopyright}Ahmet Baki Kocaballi, Emre Sezgin, Leigh Clark, John M Carroll, Yungui Huang, Jina Huh-Yoo, Junhan Kim, Rafal Kocielnik, Yi-Chieh Lee, Lena Mamykina, Elliot G Mitchell, Robert J Moore, Prasanth Murali, Elizabeth D Mynatt, Sun Young Park, Alessandro Pasta, Deborah Richards, Lucas M Silva, Diva Smriti, Brendan Spillane, Zhan Zhang, Tamara Zubatiy. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 15.11.2022.",

year = "2022",

month = nov,

day = "15",

doi = "10.2196/38525",

language = "English",

volume = "24",

journal = "Journal of Medical Internet Research",

issn = "1439-4456",

publisher = "JMIR Publications",

number = "11",

}

Kocaballi, AB, Sezgin, E, Clark, L, Carroll, JM, Huang, Y, Huh-Yoo, J, Kim, J, Kocielnik, R, Lee, Y-C, Mamykina, L, Mitchell, EG, Moore, RJ, Murali, P, Mynatt, ED, Park, SY, Pasta, A, Richards, D, Silva, LM, Smriti, D, Spillane, B, Zhang, Z & Zubatiy, T 2022, 'Design and Evaluation Challenges of Conversational Agents in Health Care and Well-being: Selective Review Study', Journal of Medical Internet Research, vol. 24, no. 11, e38525. https://doi.org/10.2196/38525

TY - JOUR

T1 - Design and Evaluation Challenges of Conversational Agents in Health Care and Well-being

T2 - Selective Review Study

AU - Kocaballi, Ahmet Baki

AU - Sezgin, Emre

AU - Clark, Leigh

AU - Carroll, John M

AU - Huang, Yungui

AU - Huh-Yoo, Jina

AU - Kim, Junhan

AU - Kocielnik, Rafal

AU - Lee, Yi-Chieh

AU - Mamykina, Lena

AU - Mitchell, Elliot G

AU - Moore, Robert J

AU - Murali, Prasanth

AU - Mynatt, Elizabeth D

AU - Park, Sun Young

AU - Pasta, Alessandro

AU - Richards, Deborah

AU - Silva, Lucas M

AU - Smriti, Diva

AU - Spillane, Brendan

AU - Zhang, Zhan

AU - Zubatiy, Tamara

N1 - ©Ahmet Baki Kocaballi, Emre Sezgin, Leigh Clark, John M Carroll, Yungui Huang, Jina Huh-Yoo, Junhan Kim, Rafal Kocielnik, Yi-Chieh Lee, Lena Mamykina, Elliot G Mitchell, Robert J Moore, Prasanth Murali, Elizabeth D Mynatt, Sun Young Park, Alessandro Pasta, Deborah Richards, Lucas M Silva, Diva Smriti, Brendan Spillane, Zhan Zhang, Tamara Zubatiy. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 15.11.2022.

PY - 2022/11/15

Y1 - 2022/11/15

N2 - BACKGROUND: Health care and well-being are 2 main interconnected application areas of conversational agents (CAs). There is a significant increase in research, development, and commercial implementations in this area. In parallel to the increasing interest, new challenges in designing and evaluating CAs have emerged.OBJECTIVE: This study aims to identify key design, development, and evaluation challenges of CAs in health care and well-being research. The focus is on the very recent projects with their emerging challenges.METHODS: A review study was conducted with 17 invited studies, most of which were presented at the ACM (Association for Computing Machinery) CHI 2020 conference workshop on CAs for health and well-being. Eligibility criteria required the studies to involve a CA applied to a health or well-being project (ongoing or recently finished). The participating studies were asked to report on their projects' design and evaluation challenges. We used thematic analysis to review the studies.RESULTS: The findings include a range of topics from primary care to caring for older adults to health coaching. We identified 4 major themes: (1) Domain Information and Integration, (2) User-System Interaction and Partnership, (3) Evaluation, and (4) Conversational Competence.CONCLUSIONS: CAs proved their worth during the pandemic as health screening tools, and are expected to stay to further support various health care domains, especially personal health care. Growth in investment in CAs also shows the value as a personal assistant. Our study shows that while some challenges are shared with other CA application areas, safety and privacy remain the major challenges in the health care and well-being domains. An increased level of collaboration across different institutions and entities may be a promising direction to address some of the major challenges that otherwise would be too complex to be addressed by the projects with their limited scope and budget.

AB - BACKGROUND: Health care and well-being are 2 main interconnected application areas of conversational agents (CAs). There is a significant increase in research, development, and commercial implementations in this area. In parallel to the increasing interest, new challenges in designing and evaluating CAs have emerged.OBJECTIVE: This study aims to identify key design, development, and evaluation challenges of CAs in health care and well-being research. The focus is on the very recent projects with their emerging challenges.METHODS: A review study was conducted with 17 invited studies, most of which were presented at the ACM (Association for Computing Machinery) CHI 2020 conference workshop on CAs for health and well-being. Eligibility criteria required the studies to involve a CA applied to a health or well-being project (ongoing or recently finished). The participating studies were asked to report on their projects' design and evaluation challenges. We used thematic analysis to review the studies.RESULTS: The findings include a range of topics from primary care to caring for older adults to health coaching. We identified 4 major themes: (1) Domain Information and Integration, (2) User-System Interaction and Partnership, (3) Evaluation, and (4) Conversational Competence.CONCLUSIONS: CAs proved their worth during the pandemic as health screening tools, and are expected to stay to further support various health care domains, especially personal health care. Growth in investment in CAs also shows the value as a personal assistant. Our study shows that while some challenges are shared with other CA application areas, safety and privacy remain the major challenges in the health care and well-being domains. An increased level of collaboration across different institutions and entities may be a promising direction to address some of the major challenges that otherwise would be too complex to be addressed by the projects with their limited scope and budget.

KW - Humans

KW - Aged

KW - Communication

KW - Delivery of Health Care

KW - Health Personnel

U2 - 10.2196/38525

DO - 10.2196/38525

M3 - Review

C2 - 36378515

SN - 1439-4456

VL - 24

JO - Journal of Medical Internet Research

JF - Journal of Medical Internet Research

IS - 11

M1 - e38525

ER -

Design and Evaluation Challenges of Conversational Agents in Health Care and Well-being: Selective Review Study

Abstract

Keywords

Access to Document

OpenUrl availability

Fingerprint

Cite this