Beyond Face Value: Visual and Auditory Signals in Human and Machine Trust Judgments

Natalia Tyulina; Yuwen Yu; Tatiana Aloi Emmanouil; Sarah Ita Levitan

doi:10.1145/3719160.3737630

Beyond Face Value: Visual and Auditory Signals in Human and Machine Trust Judgments

Natalia Tyulina
, Yuwen Yu
, Tatiana Aloi Emmanouil
, Sarah Ita Levitan

City University of New York

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

As conversational agents become increasingly multimodal, they invite human-like evaluations—especially in trust-sensitive contexts. Building on the human tendency to form rapid judgments from subtle visual and auditory cues, we explore how trust is constructed from faces and voices. In a behavioral experiment, 150 participants rated bimodal stimuli across four trust congruence conditions. We then trained a multimodal model using HuBERT and ResNet-50 with late fusion to predict trust scores. To examine alignment between human and model judgments, we applied Permutation Feature Importance (PFI) to compare the most influential features. Our results highlight the dominance of auditory cues in both human and model trust evaluations, while revealing subtle but meaningful differences in feature weighting across modalities and conditions.

Original language	English
Title of host publication	CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces
Publisher	Association for Computing Machinery, Inc
ISBN (Electronic)	9798400715273
DOIs	https://doi.org/10.1145/3719160.3737630
State	Published - 7 Jul 2025
Externally published	Yes
Event	7th Conference on Conversational User Interfaces, CUI 2025 - Waterloo, Canada Duration: 8 Jul 2025 → 10 Jul 2025

Publication series

Name	CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces

Conference

Conference	7th Conference on Conversational User Interfaces, CUI 2025
Country/Territory	Canada
City	Waterloo
Period	8/07/25 → 10/07/25

Bibliographical note

Publisher Copyright:
© 2025 Copyright held by the owner/author(s).

Keywords

Behavioral Experiment
Feature Importance
Human-AI Alignment
Multimodal Trust Perception

Access to Document

10.1145/3719160.3737630

Cite this

Tyulina, N., Yu, Y., Emmanouil, T. A., & Levitan, S. I. (2025). Beyond Face Value: Visual and Auditory Signals in Human and Machine Trust Judgments. In CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces Article 26 (CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces). Association for Computing Machinery, Inc. https://doi.org/10.1145/3719160.3737630

@inproceedings{d986b26ea198453192c1d50c505935a5,

title = "Beyond Face Value: Visual and Auditory Signals in Human and Machine Trust Judgments",

abstract = "As conversational agents become increasingly multimodal, they invite human-like evaluations—especially in trust-sensitive contexts. Building on the human tendency to form rapid judgments from subtle visual and auditory cues, we explore how trust is constructed from faces and voices. In a behavioral experiment, 150 participants rated bimodal stimuli across four trust congruence conditions. We then trained a multimodal model using HuBERT and ResNet-50 with late fusion to predict trust scores. To examine alignment between human and model judgments, we applied Permutation Feature Importance (PFI) to compare the most influential features. Our results highlight the dominance of auditory cues in both human and model trust evaluations, while revealing subtle but meaningful differences in feature weighting across modalities and conditions.",

keywords = "Behavioral Experiment, Feature Importance, Human-AI Alignment, Multimodal Trust Perception",

author = "Natalia Tyulina and Yuwen Yu and Emmanouil, \{Tatiana Aloi\} and Levitan, \{Sarah Ita\}",

note = "Publisher Copyright: {\textcopyright} 2025 Copyright held by the owner/author(s).; 7th Conference on Conversational User Interfaces, CUI 2025 ; Conference date: 08-07-2025 Through 10-07-2025",

year = "2025",

month = jul,

day = "7",

doi = "10.1145/3719160.3737630",

language = "אנגלית",

series = "CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces",

publisher = "Association for Computing Machinery, Inc",

booktitle = "CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces",

}

Tyulina, N, Yu, Y, Emmanouil, TA & Levitan, SI 2025, Beyond Face Value: Visual and Auditory Signals in Human and Machine Trust Judgments. in CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces., 26, CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces, Association for Computing Machinery, Inc, 7th Conference on Conversational User Interfaces, CUI 2025, Waterloo, Canada, 8/07/25. https://doi.org/10.1145/3719160.3737630

Beyond Face Value: Visual and Auditory Signals in Human and Machine Trust Judgments. / Tyulina, Natalia; Yu, Yuwen; Emmanouil, Tatiana Aloi et al.
CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces. Association for Computing Machinery, Inc, 2025. 26 (CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Beyond Face Value

T2 - 7th Conference on Conversational User Interfaces, CUI 2025

AU - Tyulina, Natalia

AU - Yu, Yuwen

AU - Emmanouil, Tatiana Aloi

AU - Levitan, Sarah Ita

PY - 2025/7/7

Y1 - 2025/7/7

N2 - As conversational agents become increasingly multimodal, they invite human-like evaluations—especially in trust-sensitive contexts. Building on the human tendency to form rapid judgments from subtle visual and auditory cues, we explore how trust is constructed from faces and voices. In a behavioral experiment, 150 participants rated bimodal stimuli across four trust congruence conditions. We then trained a multimodal model using HuBERT and ResNet-50 with late fusion to predict trust scores. To examine alignment between human and model judgments, we applied Permutation Feature Importance (PFI) to compare the most influential features. Our results highlight the dominance of auditory cues in both human and model trust evaluations, while revealing subtle but meaningful differences in feature weighting across modalities and conditions.

AB - As conversational agents become increasingly multimodal, they invite human-like evaluations—especially in trust-sensitive contexts. Building on the human tendency to form rapid judgments from subtle visual and auditory cues, we explore how trust is constructed from faces and voices. In a behavioral experiment, 150 participants rated bimodal stimuli across four trust congruence conditions. We then trained a multimodal model using HuBERT and ResNet-50 with late fusion to predict trust scores. To examine alignment between human and model judgments, we applied Permutation Feature Importance (PFI) to compare the most influential features. Our results highlight the dominance of auditory cues in both human and model trust evaluations, while revealing subtle but meaningful differences in feature weighting across modalities and conditions.

KW - Behavioral Experiment

KW - Feature Importance

KW - Human-AI Alignment

KW - Multimodal Trust Perception

UR - https://www.scopus.com/pages/publications/105011593448

U2 - 10.1145/3719160.3737630

DO - 10.1145/3719160.3737630

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:105011593448

T3 - CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces

BT - CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces

PB - Association for Computing Machinery, Inc

Y2 - 8 July 2025 through 10 July 2025

ER -

Tyulina N, Yu Y, Emmanouil TA, Levitan SI. Beyond Face Value: Visual and Auditory Signals in Human and Machine Trust Judgments. In CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces. Association for Computing Machinery, Inc. 2025. 26. (CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces). doi: 10.1145/3719160.3737630

Beyond Face Value: Visual and Auditory Signals in Human and Machine Trust Judgments

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this