Tzovaras Multimodal User Interfaces

From Signals to Interaction
1. Auflage 2008
ISBN: 978-3-540-78345-9
Verlag: Springer Berlin Heidelberg
Format: PDF
Kopierschutz: 1 - PDF Watermark

E-Book, Englisch, 315 Seiten

Reihe: Signals and Communication Technology

tionship indicates how multimodal medical image processing can be unified to a large extent, e. g. multi-channel segmentation and image registration, and extend information theoretic registration to other features than image intensities. The framework is not at all restricted to medical images though and this is illustrated by applying it to multimedia sequences as well. In Chapter 4, the main results from the developments in plastic UIs and mul- modal UIs are brought together using a theoretic and conceptual perspective as a unifying approach. It is aimed at defining models useful to support UI plasticity by relying on multimodality, at introducing and discussing basic principles that can drive the development of such UIs, and at describing some techniques as proof-of-concept of the aforementioned models and principles. In Chapter 4, the authors introduce running examples that serve as illustration throughout the d- cussion of the use of multimodality to support plasticity.

Autoren/Hrsg.

Tzovaras, Dimitrios

Tzovaras, Dimitros

Inhaltsverzeichnis

1;Content;6
2;Introduction;12
3;Multimodality Theory;16
3.1;2.1 Introduction;16
3.2;2.2 What is a Multimodal System?;16
3.2.1;2.2.1 An Enigma;16
3.2.2;2.2.2 A Solution;17
3.3;2.3 Which Modalities are There?;21
3.3.1;2.3.1 Deriving a Taxonomy of Input/Output Modalities;21
3.3.2;2.3.2 Basic Concepts;23
3.3.3;2.3.3 Modality Taxonomy;26
3.3.4;2.3.4 Information Channels;30
3.3.5;2.3.5 Interaction Devices;31
3.3.6;2.3.6 Practical Uses of the Theory;31
3.4;2.4 Multimodal Information Representation;35
3.4.1;2.4.1 Advantages of Multimodality;35
3.4.2;2.4.2 Constructing Multimodality from Unimodal Modalities;36
3.4.3;2.4.3 Linear Modality Addition and Replacement;36
3.4.4;2.4.4 Non-linear Effects, Users, Design Detail, Purpose;39
4;Information-Theoretic Framework for Multimodal Signal Processing;42
4.1;3.1 Introduction;42
4.2;3.2 Some Information Theoretic Concepts;43
4.2.1;3.2.1 Stochastic Process and Error Probability;43
4.2.2;3.2.2 Fano’s Inequality and the Data Processing Inequality;44
4.2.3;3.2.3 Information Theoretic Feature Extraction.;46
4.3;3.3 From Error Probability to Multimodal Signal Processing;47
4.3.1;3.3.1 Multimodal Stochastic Processes;47
4.3.2;3.3.2 Objective Functions for Multimodal Signal Processing;50
4.4;3.4 Optimization;59
4.5;3.5 Results;61
4.5.1;3.5.1 Multimodal Medical Images;61
4.5.2;3.5.2 Speech-Video Sequences;66
5;Multimodality for Plastic User Interfaces: Models, Methods, and Principles;72
5.1;4.1 Introduction;72
5.2;4.2 Running Examples;73
5.2.1;4.2.1 The Assisted Neuro-surgery System;73
5.2.2;4.2.2 The Sedan-Bouillon Web Site;75
5.3;4.3 Modality and Multimodality;77
5.3.1;4.3.1 Definitions;77
5.3.2;4.3.2 The CARE Properties;78
5.4;4.4 The Problem Space of Plastic Multimodal UIs;81
5.4.1;4.4.1 Two Adaptation Means: UI Re-molding and UI Re- distribution;82
5.4.2;4.4.2 UI Components Granularity;83
5.4.3;4.4.3 State Recovery Granularity;83
5.4.4;4.4.4 UI Deployment;84
5.4.5;4.4.5 Coverage of the Context of Use;84
5.4.6;4.4.6 Coverage of Technological Spaces;85
5.4.7;4.4.7 Existence of a Meta-UI;85
5.4.8;4.4.8 UI Re-molding and Modalities;86
5.4.9;4.4.9 UI Re-molding and Levels of Abstraction;87
5.4.10;4.4.10 Summary;87
5.5;4.5 Domain of Plasticity of a User Interface;88
5.6;4.6 Three Principles for the Development of Plastic Multimodal UIs;92
5.6.1;4.6.1 Blurring the Distinction between Design-time and Run-time;92
5.6.2;4.6.2 Mixing Close and Open Adaptiveness;93
5.6.3;4.6.3 Keeping Humans in the Loop;93
6;Face and Speech Interaction;96
6.1;5.1 Face and Facial Feature Detection;96
6.1.1;5.1.1 Face Detection;97
6.1.2;5.1.2 Facial Feature Detection;108
6.2;5.2 Interaction;109
6.2.1;5.2.1 Multimodal Speaker Localization;110
6.2.2;5.2.2 Audio-Visual Speech Recognition;115
7;Recognition of Emotional States in Natural Human- Computer Interaction;130
7.1;6.1 Introduction;130
7.2;6.2 Fundamentals;132
7.2.1;6.2.1 Emotion Representation;132
7.2.2;6.2.2 Methodology Outline;134
7.2.3;6.2.3 Running Example;135
7.3;6.3 Feature Extraction;136
7.3.1;6.3.1 Visual Modality;136
7.3.2;6.3.2 Auditory Modality;149
7.4;6.4 Multimodal Expression Classification;151
7.4.1;6.4.1 The Elman Net;151
7.4.2;6.4.2 Classification;154
7.5;6.5 Experimental Results;157
7.5.1;6.5.1 The Case for Naturalistic Data;157
7.5.2;6.5.2 Statistical Results;158
7.5.3;6.5.3 Quantitative Comparative Study;160
7.5.4;6.5.4 Qualitative Comparative Study;162
8;Two SIMILAR Different Speech and Gestures Multimodal Interfaces;166
8.1;7.1 Introduction and State-of-the-art;166
8.2;7.2 ICANDO Multimodal Interface;171
8.2.1;7.2.1 Objectives;171
8.2.2;7.2.2 System’s Description;172
8.3;7.3 Experimental Results;179
8.4;7.4 MOWGLI Multimodal Interface;180
8.4.1;7.4.1 Objectives;181
8.4.2;7.4.2 System’s Description;181
9;Multimodal User Interfaces in Ubiquitous Environments;196
9.1;8.1 Introduction;196
9.2;8.2 Related Work;197
9.3;8.3 Migratory User Interfaces;199
9.4;8.4 The Dimensions of Migration;199
9.5;8.5 An Architecture for a Migration Platform;203
9.6;8.6 Example Applications;207
9.6.1;8.6.1 Domotic Scenario ( mobile vocal+ graphical->graphical desktop);207
9.6.2;8.6.2 Museum Scenario (pda->digital tv);208
10;Software Engineering for Multimodal Interactive Systems;212
10.1;9.1 Introduction;212
10.2;9.2 PAC-Amodeus: a Conceptual Architectural Solution;214
10.2.1;9.2.1 Concurrent Processing of Data;215
10.2.2;9.2.2 Data Fusion;215
10.3;9.3 Software Tools for Multimodality;218
10.3.1;9.3.1 Existing Tools;218
10.3.2;9.3.2 ICARE Platform;220
10.3.3;9.3.3 ICARE Conceptual Model;221
10.3.4;9.3.4 ICARE Graphical Editor;222
10.3.5;9.3.5 OpenInterface Platform;224
11;Gestural Interfaces for Hearing- Impaired Communication;230
11.1;10.1 Introduction;230
11.2;10.2 Modality Processing and Analysis;232
11.2.1;10.2.1 Preprocessing;233
11.2.2;10.2.2 Hand Shape;235
11.2.3;10.2.3 Hand Location;240
11.2.4;10.2.4 Hand Motion;241
11.2.5;10.2.5 Facial Movements;244
11.2.6;10.2.6 Lip Reading;245
11.2.7;10.2.7 Facial Expressions;246
11.3;10.3 Temporal Analysis;247
11.3.1;10.3.1 Sign Language;247
11.3.2;10.3.2 Cued Speech;248
11.4;10.4 Multimodal Fusion;248
11.4.1;10.4.1 Temporal Modelling;249
11.4.2;10.4.2 Heterogenic Multiplexing;251
11.5;10.5 Applications;257
11.5.1;10.5.1 Sign Language Tutoring Tool;257
11.5.2;10.5.2 Cued Speech Manual Gesture Interpreter;259
12;Modality Replacement Framework for Applications for the Disabled;262
12.1;11.1 Introduction;262
12.2;11.2 The Modality Replacement Concept;265
12.3;11.3 Cued Speech;266
12.4;11.4 Feature Extraction and Representation for the Cued Speech Language;268
12.4.1;11.4.1 Audio Feature Extraction;268
12.4.2;11.4.2 Lip Shape Feature Extraction;268
12.4.3;11.4.3 Gesture Feature Extraction;269
12.5;11.5 Coupled Hidden Markov Models;270
12.6;11.6 Modality Reliability;271
12.7;11.7 Modified Coupled Hidden Markov Model;272
12.7.1;11.7.1 Training;272
12.8;11.8 Evaluation of the Cued Speech Recognition System;273
12.9;11.9.1 Multimodal Collaborative Game;276
13;A medical Component-based Framework for Image Guided Surgery;282
13.1;12.1 Introduction;282
13.2;12.2 MedicalStudio Framework;283
13.2.1;12.2.1 Architecture;283
13.2.2;12.2.2 Framework Implementation;284
13.3;12.3 General Purpose Components;285
13.3.1;12.3.1 Multimodal Registration;286
13.3.2;12.3.2 Segmentation and 3D Reconstruction;287
13.4;12.4 Applications;288
13.4.1;12.4.1 3D Medical Assistant for Orthognatic Computer Surgery;288
13.4.2;12.4.2 ACROGuide;289
14;Multimodal Interfaces for Laparoscopic Training;292
14.1;13.1 Introduction;292
14.2;13.2 Functionality;293
14.3;13.3 Technical Issues;293
14.3.1;13.3.1 Simulator Architecture;294
14.3.2;13.3.2 Collision Detection and Handling;296
14.3.3;13.3.3 Design of Multimodal Interface Scenarios and Surgical Simulation Tasks;298
14.4;13.4 Research Issues;299
14.4.1;13.4.1 Study of Laparoscopic Sensory Interaction;299
14.4.2;13.4.2 Laparoscopic Simulation Conceptual Framework;300
15;References;302