Share
VIDEOS 1 TO 50
Defining Multimodal Interactions: One Size Does Not Fit All (Google I/O
Defining Multimodal Interactions: One Size Does Not Fit All (Google I/O '17)
Published: 2017/05/18
Channel: Google Developers
Natural User Interfaces for Human Drone Multi-Modal Interaction
Natural User Interfaces for Human Drone Multi-Modal Interaction
Published: 2016/02/23
Channel: Vision4UAV
Multimodal interaction to simulate natural interactive walking
Multimodal interaction to simulate natural interactive walking
Published: 2013/04/06
Channel: Luca Turchet
Multimodal interaction with a washing machine
Multimodal interaction with a washing machine
Published: 2013/10/16
Channel: KAEMaRT Group
Multimodal Interactions & JS: The What, The Why and The How by Diego Paez, Despegar
Multimodal Interactions & JS: The What, The Why and The How by Diego Paez, Despegar
Published: 2016/12/15
Channel: node.js
Eyes on Multimodal Interaction
Eyes on Multimodal Interaction
Published: 2016/09/06
Channel: Microsoft Research
AR and Multimodal Interaction in Service and Maintenance
AR and Multimodal Interaction in Service and Maintenance
Published: 2014/03/06
Channel: SAPEnterpriseMobile
Multimodal interaction - video prototype
Multimodal interaction - video prototype
Published: 2011/07/12
Channel: gummbahla
Multi-modal interaction using 3D graphics and touch [HD]
Multi-modal interaction using 3D graphics and touch [HD]
Published: 2009/06/30
Channel: ikinamo
A multimodal interaction system for big displays
A multimodal interaction system for big displays
Published: 2017/04/07
Channel: UserAndroid GPDS
Multimodal interaction
Multimodal interaction
Published: 2017/07/20
Channel: Leibniz-Institut für Wissensmedien (IWM)
Chris Bennet on Multimodal Interaction
Chris Bennet on Multimodal Interaction
Published: 2011/03/14
Channel: The University of Maine
The MINT.tools for Multimodal Interaction Analysis
The MINT.tools for Multimodal Interaction Analysis
Published: 2013/06/02
Channel: David Schlangen
Multimodal Interactions with HoloLens
Multimodal Interactions with HoloLens
Published: 2017/04/30
Channel: mcrlab
Integrated Multimodal Interaction Using Normal Maps
Integrated Multimodal Interaction Using Normal Maps
Published: 2015/03/17
Channel: Auston Sterling
Multimodal Interaction with W3C Standards Toward Natural User Interfaces to Everything
Multimodal Interaction with W3C Standards Toward Natural User Interfaces to Everything
Published: 2017/05/11
Channel: tery tia
Multimodal interaction with virtual environments
Multimodal interaction with virtual environments
Published: 2007/07/17
Channel: mightbesheep
Integrated Multimodal Interaction Using Texture Representations
Integrated Multimodal Interaction Using Texture Representations
Published: 2016/01/27
Channel: Auston Sterling
Multimodal interaction with a fridge
Multimodal interaction with a fridge
Published: 2013/10/16
Channel: KAEMaRT Group
NeoBank Multimodal Interaction
NeoBank Multimodal Interaction
Published: 2013/01/28
Channel: Gunnar Bylund
Effects of Errors on Multimodal Interaction in WoZ experiment
Effects of Errors on Multimodal Interaction in WoZ experiment
Published: 2016/05/27
Channel: Michal Kapinus
Wearable multi-modal interfaces for human multi-robot interaction
Wearable multi-modal interfaces for human multi-robot interaction
Published: 2016/09/21
Channel: Gianni Di Caro
Supporting model-based multimodal interaction in the web.
Supporting model-based multimodal interaction in the web.
Published: 2015/04/21
Channel: MusicNet istLab - Tei Crete
Voice Browsing And Multimodal Interaction In 2009
Voice Browsing And Multimodal Interaction In 2009
Published: 2009/03/09
Channel: GoogleTechTalks
Semantics architecture of multimodal interaction for ambient intelligence applied to Nao robot
Semantics architecture of multimodal interaction for ambient intelligence applied to Nao robot
Published: 2012/07/18
Channel: SD
Multimodal Interaction - Leap Motion, Philips Hue and VLC Player
Multimodal Interaction - Leap Motion, Philips Hue and VLC Player
Published: 2014/01/17
Channel: Rocco Musolino
Multimodal Interaction - Interactive tabletop video
Multimodal Interaction - Interactive tabletop video
Published: 2010/04/21
Channel: willem willemsen
Raffaella Bernardi - Language & Multimodal Interaction track (LMI) ... - Rovereto, 10 March 2017
Raffaella Bernardi - Language & Multimodal Interaction track (LMI) ... - Rovereto, 10 March 2017
Published: 2017/03/13
Channel: ICTs polorovereto UniTrento
Multimodal interaction idea for digital pet
Multimodal interaction idea for digital pet
Published: 2007/03/25
Channel: Ralph Zoontjens
Multimodal interaction with virtual environments 2
Multimodal interaction with virtual environments 2
Published: 2007/07/17
Channel: mightbesheep
Motivating Multimodal Interaction Around Digital Tabletops
Motivating Multimodal Interaction Around Digital Tabletops
Published: 2009/07/24
Channel: UofCInteractionsLab1
SCRIPT Spinning Cube: website with multi-modal interaction
SCRIPT Spinning Cube: website with multi-modal interaction
Published: 2015/01/19
Channel: uidgmbh
A control architecture for multiple drones operated via multimodal interaction
A control architecture for multiple drones operated via multimodal interaction
Published: 2016/07/11
Channel: PRISMA Lab
Multi-Modal Human Computer Interface
Multi-Modal Human Computer Interface
Published: 2016/04/29
Channel: Stretch1414
Mutual Dissambiguation of 3D Multimodal Interaction in Augmented and Virtual Reality
Mutual Dissambiguation of 3D Multimodal Interaction in Augmented and Virtual Reality
Published: 2016/12/06
Channel: Microsoft Research
Language & Multimodal Interaction track (LMI) and European Program in ... (LCT) – Raffaella Bernardi
Language & Multimodal Interaction track (LMI) and European Program in ... (LCT) – Raffaella Bernardi
Published: 2016/03/01
Channel: ICTs polorovereto UniTrento
HCI Project - multimodal interface design strider review
HCI Project - multimodal interface design strider review
Published: 2014/02/25
Channel: Dizax
Multi-modal interaction: We Vote, supporting decision making in meetings
Multi-modal interaction: We Vote, supporting decision making in meetings
Published: 2010/06/23
Channel: Niko Vegt
VersaPen: Exploring Multimodal Interactions with a Programmable Modular Pen
VersaPen: Exploring Multimodal Interactions with a Programmable Modular Pen
Published: 2017/05/03
Channel: ACM SIGCHI
Voice and Multimodal Solutions by T-Labs - Intuitive user interfaces
Voice and Multimodal Solutions by T-Labs - Intuitive user interfaces
Published: 2011/11/07
Channel: T-Labs - Telekom Innovation Laboratories
[GT] Internal Project - Multimodal Interaction with GeoSpatial Data
[GT] Internal Project - Multimodal Interaction with GeoSpatial Data
Published: 2015/03/28
Channel: Fondazione GraphiTech
Multimodal Interaction (DG 303, 2009), Presentation 1
Multimodal Interaction (DG 303, 2009), Presentation 1
Published: 2009/03/23
Channel: Zeljko Obrenovic
MULCE LETEC: Multimodal interaction ; when audio is ignored
MULCE LETEC: Multimodal interaction ; when audio is ignored
Published: 2013/09/15
Channel: Thierry Chanier
Towards Multimodal Affective Feedback - Interaction between Visual and Haptic Modalities
Towards Multimodal Affective Feedback - Interaction between Visual and Haptic Modalities
Published: 2015/04/18
Channel: Association for Computing Machinery (ACM)
Multimodal mobile interaction - blending speech and GUI input - iphone demo
Multimodal mobile interaction - blending speech and GUI input - iphone demo
Published: 2010/10/14
Channel: Manolis Perakakis
Multimodal Interaction WS 2015/16 - Kleinbeck, Reemts, Unruh
Multimodal Interaction WS 2015/16 - Kleinbeck, Reemts, Unruh
Published: 2016/03/24
Channel: HCI Group Würzburg
Multimodal Communication
Multimodal Communication
Published: 2016/09/28
Channel: Cerebral Palsy Foundation
iPhone multimodal interaction example
iPhone multimodal interaction example
Published: 2009/02/12
Channel: solaiemes
Big Bang Demo - Multimodal Interactions in VR (HTC Vive, Unreal Engine and SimX)
Big Bang Demo - Multimodal Interactions in VR (HTC Vive, Unreal Engine and SimX)
Published: 2016/10/28
Channel: HCI Group Würzburg
Multimodal Interaction (DG 303, 2009), Presentation 2
Multimodal Interaction (DG 303, 2009), Presentation 2
Published: 2009/03/23
Channel: Zeljko Obrenovic
NEXT
GO TO RESULTS [51 .. 100]

WIKIPEDIA ARTICLE

From Wikipedia, the free encyclopedia
Jump to: navigation, search

Multimodal interaction provides the user with multiple modes of interacting with a system. A multimodal interface provides several distinct tools for input and output of data. For example, a multimodal question answering system employs multiple modalities (such as text and photo) at both question (input) and answer (output) level.[1]

Introduction[edit]

Multimodal human-computer interaction refers to the "interaction with the virtual and physical environment through natural modes of communication",[2] i. e. the modes involving the five human senses.[3] This implies that multimodal interaction enables a more free and natural communication, interfacing users with automated systems in both input and output.[4] Specifically, multimodal systems can offer a flexible, efficient and usable environment allowing users to interact through input modalities, such as speech, handwriting, hand gesture and gaze, and to receive information by the system through output modalities, such as speech synthesis, smart graphics and others modalities, opportunely combined. Then a multimodal system has to recognize the inputs from the different modalities combining them according to temporal and contextual constraints[5] in order to allow their interpretation. This process is known as multimodal fusion, and it is the object of several research works from nineties to now.[6][7][8][9][10][11][12][13] The fused inputs are interpreted by the system. Naturalness and flexibility can produce more than one interpretation for each different modality (channel) and for their simultaneous use, and they consequently can produce multimodal ambiguity[14] generally due to imprecision, noises or other similar factors. For solving ambiguities, several methods have been proposed.[15][16][17][18][19][20][21] Finally the system returns to the user outputs through the various modal channels (disaggregated) arranged according to a consistent feedback (fission).[22] The pervasive use of mobile devices, sensors and web technologies can offer adequate computational resources to manage the complexity implied by the multimodal interaction. "Using cloud for involving shared computational resources in managing the complexity of multimodal interaction represents an opportunity. In fact, cloud computing allows delivering shared scalable, configurable computing resources that can be dynamically and automatically provisioned and released".[23]

Multimodal input[edit]

Two major groups of multimodal interfaces have merged, one concerned in alternate input methods and the other in combined input/output. The first group of interfaces combined various user input modes beyond the traditional keyboard and mouse input/output, such as speech, pen, touch, manual gestures,[24] gaze and head and body movements.[25] The most common such interface combines a visual modality (e.g. a display, keyboard, and mouse) with a voice modality (speech recognition for input, speech synthesis and recorded audio for output). However other modalities, such as pen-based input or haptic input/output may be used. Multimodal user interfaces are a research area in human-computer interaction (HCI).

The advantage of multiple input modalities is increased usability: the weaknesses of one modality are offset by the strengths of another. On a mobile device with a small visual interface and keypad, a word may be quite difficult to type but very easy to say (e.g. Poughkeepsie). Consider how you would access and search through digital media catalogs from these same devices or set top boxes. And in one real-world example, patient information in an operating room environment is accessed verbally by members of the surgical team to maintain an antiseptic environment, and presented in near realtime aurally and visually to maximize comprehension.

Multimodal input user interfaces have implications for accessibility.[26] A well-designed multimodal application can be used by people with a wide variety of impairments. Visually impaired users rely on the voice modality with some keypad input. Hearing-impaired users rely on the visual modality with some speech input. Other users will be "situationally impaired" (e.g. wearing gloves in a very noisy environment, driving, or needing to enter a credit card number in a public place) and will simply use the appropriate modalities as desired. On the other hand, a multimodal application that requires users to be able to operate all modalities is very poorly designed.

The most common form of input multimodality in the market makes use of the XHTML+Voice (aka X+V) Web markup language, an open specification developed by IBM, Motorola, and Opera Software. X+V is currently under consideration by the W3C and combines several W3C Recommendations including XHTML for visual markup, VoiceXML for voice markup, and XML Events, a standard for integrating XML languages. Multimodal browsers supporting X+V include IBM WebSphere Everyplace Multimodal Environment, Opera for Embedded Linux and Windows, and ACCESS Systems NetFront for Windows Mobile. To develop multimodal applications, software developers may use a software development kit, such as IBM WebSphere Multimodal Toolkit, based on the open source Eclipse framework, which includes an X+V debugger, editor, and simulator.

Multimodal input and output[edit]

The second group of multimodal systems presents users with multimedia displays and multimodal output, primarily in the form of visual and auditory cues. Interface designers have also started to make use of other modalities, such as touch and olfaction. Proposed benefits of multimodal output system include synergy and redundancy. The information that is presented via several modalities is merged and refers to various aspects of the same process. The use of several modalities for processing exactly the same information provides an increased bandwidth of information transfer .[27][28][29] Currently, multimodal output is used mainly for improving the mapping between communication medium and content and to support attention management in data-rich environment where operators face considerable visual attention demands.[30]

An important step in multimodal interface design is the creation of natural mappings between modalities and the information and tasks. The auditory channel differs from vision in several aspects. It is omnidirection, transient and is always reserved.[30] Speech output, one form of auditory information, received considerable attention. Several guidelines have been developed for the use of speech. Michaelis and Wiggins (1982) suggested that speech output should be used for simple short messages that will not be referred to later. It was also recommended that speech should be generated in time and require an immediate response.

The sense of touch was first utilized as a medium for communication in the late 1950s.[31] It is not only a promising but also a unique communication channel. In contrast to vision and hearing, the two traditional senses employed in HCI, the sense of touch is proximal: it senses objects that are in contact with the body, and it is bidirectonal in that it supports both perception and acting on the environment.

Examples of auditory feedback include auditory icons in computer operating systems indicating users' actions (e.g. deleting a file, open a folder, error), speech output for presenting navigational guidance in vehicles, and speech output for warning pilots on modern airplane cockpits. Examples of tactile signals include vibrations of the turn-signal lever to warn drivers of a car in their blind spot, the vibration of auto seat as a warning to drivers, and the stick shaker on modern aircraft alerting pilots to an impending stall.[30]

Invisible interface spaces became available using sensor technology. Infrared, ultrasound and cameras are all now commonly used.[32] Transparency of interfacing with content is enhanced providing an immediate and direct link via meaningful mapping is in place, thus the user has direct and immediate feedback to input and content response becomes interface affordance (Gibson 1979).

Multimodal fusion[edit]

The process of integrating information from various input modalities and combining them into a complete command is referred as multimodal fusion.[7] In literature, three main approaches to the fusion process have been proposed, according to the main architectural levels (recognition and decision) at which the fusion of the input signals can be performed: recognition-based,[11][12][33] decision-based,[9][10][13][34][35][36][37] and hybrid multi-level fusion.[6][8][38][39][40][41][42][43]

The recognition-based fusion (also known as early fusion) consists in merging the outcomes of each modal recognizer by using integration mechanisms, such as, for example, statistical integration techniques, agent theory, hidden Markov models, artificial neural networks, etc. Examples of recognition-based fusion strategies are action frame,[33] input vectors[11] and slots.[12]

The decision-based fusion (also known as late fusion) merges the semantic information that are extracted by using specific dialogue-driven fusion procedures to yield the complete interpretation. Examples of decision-based fusion strategies are typed feature structures,[34][39] melting pots,[36][37] semantic frames,[9][13] and time-stamped lattices.[10]

The potential applications for multimodal fusion include learning environments, consumer relations, security/surveillance, computer animation, etc. Individually, modes are easily defined, but difficulty arises in having technology consider them a combined fusion.[44] It's difficult for the algorithms to factor in dimensionality; there exist variables outside of current computation abilities. For example, semantic meaning: two sentences could have the same lexical meaning but different emotional information.[44]

In the hybrid multi-level fusion, the integration of input modalities is distributed among the recognition and decision levels. The hybrid multi-level fusion includes the following three methodologies: finite-state transducers,[39] multimodal grammars[8][38][40][41][42][43][45] and dialogue moves.[46]

Multimodal interpretation and ambiguity[edit]

User's actions or commands produce multimodal inputs (multimodal message[5]), which have to be interpreted by the system. The multimodal message is the medium that enables communication between users and multimodal systems. It is obtained by merging information that are conveyed via several modalities by considering the different types of cooperation between several modalities,[47] the time relationships[48] among the involved modalities and the relationships between chunks of information connected with these modalities.[49]

The natural mapping between the multimodal input, which is provided by several interaction modalities (visual and auditory channel and sense of touch), and information and tasks imply to manage the typical problems of human-human communication, such as ambiguity. An ambiguity arises when more than one interpretation of input is possible. A multimodal ambiguity[14] arises both, if an element, which is provided by one modality, has more than one interpretation (i.e. ambiguities are propagated at the multimodal level), and/or if elements, connected with each modality, are univocally interpreted, but information referred to different modalities are incoherent at the syntactic or the semantic level (i.e. a multimodal sentence having different meanings or different syntactic structure).

In "The Management of Ambiguities",[16] the methods for solving ambiguities and for providing the correct interpretation of the user's input are organized in three main classes: prevention, a-posterior resolution and approximation resolution methods.[15][17]

Prevention methods impose users to follow predefined interaction behaviour according to a set of transitions between different allowed states of the interaction process. Example of prevention methods are: procedural method,[50] reduction of the expressive power of the language grammar,[51] improvement of the expressive power of the language grammar.[52]

The a-posterior resolution of ambiguities uses mediation approach.[18] Examples of mediation techniques are: repetition, e.g. repetition by modality,[18] granularity of repair[53] and undo,[20] and choice.[21]

The approximation resolution methods do not require any user involvement in the disambiguation process. They can all require the use of some theories, such as fuzzy logic, Markov random field, Bayesian networks and hidden Markov models.[15][17]

See also[edit]

References[edit]

  1. ^ Mittal et al. (2011). "Versatile question answering systems: seeing in synthesis", Int. J. of Intelligent Information Database Systems, 5(2), 119-142.
  2. ^ Bourguet, M.L.( 2003). "Designing and Prototyping Multimodal Commands". Proceedings of Human-Computer Interaction (INTERACT'03), pp. 717-720.
  3. ^ Ferri, F., & Paolozzi, S. (2009). Analyzing Multimodal Interaction. In P. Grifoni (Ed.), Multimodal Human Computer Interaction and Pervasive Services (pp. 19-33). Hershey, PA: Information Science Reference. doi:10.4018/978-1-60566-386-9.ch002
  4. ^ Stivers, T., Sidnell, J. Introduction: Multimodal interaction. Semiotica, 156(1/4), pp. 1-20. 2005.
  5. ^ a b Caschera M. C., Ferri F., Grifoni P. (2007). "Multimodal interaction systems: information and time features". International Journal of Web and Grid Services (IJWGS), Vol. 3 - Issue 1, pp 82-99.
  6. ^ a b D'Ulizia, A., Ferri, F. and Grifoni, P. (2010). "Generating Multimodal Grammars for Multimodal Dialogue Processing". IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans, Vol 40, no 6, pp. 1130 – 1145.
  7. ^ a b D'Ulizia , A. (2009). "Exploring Multimodal Input Fusion Strategies". In: Grifoni P (ed) Handbook of Research on Multimodal Human Computer Interaction and Pervasive Services: Evolutionary Techniques for Improving Accessibility. IGI Publishing, pp. 34-57.
  8. ^ a b c Sun, Y., Shi, Y., Chen, F. and Chung , V.(2007). "An Efficient Multimodal Language Processor for Parallel Input Strings in Multimodal Input Fusion," in Proc. of the international Conference on Semantic Computing, pp. 389-396.
  9. ^ a b c Russ, G., Sallans, B., Hareter, H. (2005). "Semantic Based Information Fusion in a Multimodal Interface". International Conference on Human-Computer Interaction (HCI'05), Las Vegas, Nevada, USA, 20–23 June, pp 94-100.
  10. ^ a b c Corradini, A., Mehta M., Bernsen, N.O., Martin, J.-C. (2003). "Multimodal Input Fusion in Human-Computer Interaction on the Example of the on-going NICE Project". In Proceedings of the NATO-ASI conference on Data Fusion for Situation Monitoring, Incident Detection, Alert and Response Management, Yerevan, Armenia.
  11. ^ a b c Pavlovic, V.I., Berry, G.A., Huang, T.S. (1997). "Integration of audio/visual information for use in human-computer intelligent interaction". Proceedings of the 1997 International Conference on Image Processing (ICIP '97), Volume 1, pp. 121-124.
  12. ^ a b c Andre, M., Popescu, V.G., Shaikh, A., Medl, A., Marsic, I., Kulikowski, C., Flanagan J.L. (1998). "Integration of Speech and Gesture for Multimodal Human-Computer Interaction". In Second International Conference on Cooperative Multimodal Communication. 28–30 January, Tilburg, The Netherlands.
  13. ^ a b c Vo, M.T., Wood, C. (1996). "Building an application framework for speech and pen input integration in multimodal learning interfaces". In Proceedings of the Acoustics, Speech, and Signal Processing (ICASSP'96), May 7–10, IEEE Computer Society, Volume 06, pp. 3545-3548.
  14. ^ a b Caschera, M.C. , Ferri, F. , Grifoni, P. (2013). "From Modal to Multimodal Ambiguities: a Classification Approach", Journal of Next Generation Information Technology (JNIT), Vol. 4, No. 5, pp. 87 -109.
  15. ^ a b c Caschera, M.C. , Ferri, F. , Grifoni, P. (2013). InteSe: An Integrated Model for Resolving Ambiguities in Multimodal Sentences". IEEE Transactions on Systems, Man, and Cybernetics: Systems, Volume: 43, Issue: 4, pp. 911 - 931.18. Spilker, J., Klarner, M., Görz, G. (2000). "Processing Self Corrections in a speech to speech system". COLING 2000. pp. 1116-1120.
  16. ^ a b Caschera M.C., Ferri F., Grifoni P., (2007). "The Management of ambiguities". In Visual Languages for Interactive Computing: Definitions and Formalizations. IGI Publishing. pp.129-140.
  17. ^ a b c J. Chai, P. Hong, and M. X. Zhou, (2004 )."A probabilistic approach to reference resolution in multimodal user interface" in Proc. 9th Int. Conf. Intell. User Interf., Madeira, Portugal, Jan. 2004, pp. 70–77.
  18. ^ a b c Dey, A. K. Mankoff , J., (2005). "Designing mediation for context-aware applications". ACM Trans. Comput.-Hum. Interact. 12(1), pp. 53-80.
  19. ^ Caschera, M. C. (2009). Interpretation Methods and Ambiguity Management in Multimodal Systems. In P. Grifoni (Ed.), Multimodal Human Computer Interaction and Pervasive Services (pp. 87-102). Hershey, PA: Information Science Reference. doi:10.4018/978-1-60566-386-9.ch005
  20. ^ a b Spilker, J., Klarner, M., Görz, G. (2000). "Processing Self Corrections in a speech to speech system". COLING 2000. pp. 1116-1120.
  21. ^ a b Mankoff, J., Hudson, S.E., Abowd, G.D. (2000). "Providing integrated toolkit-level support for ambiguity in recognition-based interfaces". Proceedings of ACM CHI'00 Conference on Human Factors in Computing Systems. pp. 368 – 375.
  22. ^ Grifoni P (2009) Multimodal fission. In: Multimodal human computer interaction and pervasive services. IGI Global, pp 103–120
  23. ^ Patrizia Grifoni, Fernando Ferri, Maria Chiara Caschera, Arianna D'Ulizia, Mauro Mazzei, "MIS: Multimodal Interaction Services in a cloud perspective", JNIT: Journal of Next Generation Information Technology, Vol. 5, No. 4, pp. 01 ~ 10, 2014
  24. ^ Kettebekov, Sanshzar, and Rajeev Sharma (2001). "Toward Natural Gesture/Speech Control of a Large Display." ProceedingsEHCI '01 Proceedings of the 8th IFIP International Conference on Engineering for Human-Computer Interaction Pages 221-234
  25. ^ Marius Vassiliou, V. Sundareswaran, S. Chen, R. Behringer, C. Tam, M. Chan, P. Bangayan, and J. McGee (2000), "Integrated Multimodal Human-Computer Interface and Augmented Reality for Interactive Display Applications," in Darrel G. Hopper (ed.) Cockpit Displays VII: Displays for Defense Applications (Proc. SPIE . 4022), 106-115. ISBN 0-8194-3648-8
  26. ^ Vitense, H.S.; Jacko, J.A.; Emery, V.K. (2002). "Multimodal feedback: establishing a performance baseline for improved access by individuals with visual impairments". ACM Conf. on Assistive Technologies. 
  27. ^ Oviatt, S. (2002), "Multimodal interfaces", in Jacko, J.; Sears, A, The Human-Computer Interaction Handbook (PDF), Lawrence Erlbaum 
  28. ^ Bauckhage, C.; Fritsch, J.; Rohlfing, K.J.; Wachsmuth, S.; Sagerer, G. (2002). "Evaluating integrated speech-and image understanding". Int. Conf. on Multimodal Interfaces. 
  29. ^ Ismail, N.A.; O'Brien, E.A. (2008). "Enabling Multimodal Interaction in Web-Based Personal Digital Photo Browsing" (PDF). Int. Conf. on Computer and Communication Engineering. 
  30. ^ a b c Sarter, N.B. (2006). "Multimodal information presentation: Design guidance and research challenges". Int. J. of Industrial Ergonomics. 36 (5). pp. 439–445. 
  31. ^ Geldar, F.A (1957). "Adventures in tactile literacy". American Psychologist. 12 (3). pp. 115–124. 
  32. ^ Brooks, A.; Petersson, E. (2007). "SoundScapes: non-formal learning potentials from interactive VEs". SIGGRAPH. 
  33. ^ a b Vo, M.T. (1998). "A framework and Toolkit for the Construction of Multimodal Learning Interfaces", PhD. Thesis, Carnegie Mellon University, Pittsburgh, USA.
  34. ^ a b Cohen, P.R.; Johnston, M.; McGee, D.; Oviatt, S.L.; Pittman, J.; Smith, I.A.; Chen, L.; Clow, J. (1997). "Quickset: Multimodal interaction for distributed applications", ACM Multimedia, pp. 31-40.
  35. ^ Johnston, M. (1998). "Unification-based Multimodal Parsing". Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics (COLING-ACL '98), August 10–14, Université de Montréal, Montreal, Quebec, Canada. pp. 624-630.
  36. ^ a b Nigay, L.; Coutaz, J. (1995). "A generic platform for addressing the multimodal challenge". Proceedings of the Conference on Human Factors in Computing Systems, ACM Press.
  37. ^ a b Bouchet, J.; Nigay, L.; Ganille, T. (2004). "Icare software components for rapidly developing multimodal interfaces". ICMI '04: Proceedings of the 6th international conference on Multimodal interfaces (New York, NY, USA), ACM, pp. 251-258.
  38. ^ a b D'Ulizia, A.; Ferri, F.; Grifoni P. (2007). "A Hybrid Grammar-Based Approach to Multimodal Languages Specification", OTM 2007 Workshop Proceedings, 25–30 November 2007, Vilamoura, Portugal, Springer-Verlag, Lecture Notes in Computer Science 4805, pp. 367-376.
  39. ^ a b c Johnston, M.; Bangalore, S. (2000). "Finite-state Multimodal Parsing and Understanding", In Proceedings of the International Conference on Computational Linguistics, Saarbruecken, Germany.
  40. ^ a b Sun, Y.; Chen, F.; Shi, Y.D.; Chung, V. (2006). "A novel method for multi-sensory data fusion in multimodal human computer interaction". In Proceedings of the 20th conference of the computer-human interaction special interest group (CHISIG) of Australia on Computer-human interaction: design: activities, artefacts and environments, Sydney, Australia, pp. 401-404
  41. ^ a b Shimazu, H.; Takashima, Y. (1995). "Multimodal Definite Clause Grammar," Systems and Computers in Japan, vol. 26, no 3, pp. 93-102.
  42. ^ a b Johnston, M.; Bangalore, S. (2005). "Finite-state multimodal integration and understanding," Nat. Lang. Eng, Vol. 11, no. 2, pp. 159-187.
  43. ^ a b Reitter, D.; Panttaja, E. M.; Cummins, F. (2004). "UI on the fly: Generating a multimodal user interface," in Proc. of HLT-NAACL-2004, Boston, Massachusetts, USA.
  44. ^ a b Guan, Ling. "Methods and Techniques for MultiModal Information Fusion" (PDF). Circuits & Systems Society. 
  45. ^ D'Ulizia, A.; Ferri, F.; Grifoni P. (2011). "A Learning Algorithm for Multimodal Grammar Inference", IEEE Transactions on Systems, Man, and Cybernetics - Part B: Cybernetics, Vol. 41 (6), pp. 1495 - 1510.
  46. ^ Pérez, G.; Amores, G.; Manchón, P. (2005). "Two strategies for multimodal fusion". In Proceedings of Multimodal Interaction for the Visualization and Exploration of Scientific Data, Trento, Italy, 26–32.
  47. ^ Martin, J.C. (1997). "Toward intelligent cooperation between modalities: the example of a system enabling multimodal interaction with a map", Proceedings of International Joint Conference on Artificial Intelligence (IJCAI'97) Workshop on 'Intelligent Multimodal Systems', Nagoya, Japan
  48. ^ Allen, J.F.; Ferguson, G. (1994). "Actions and events in interval temporal logic", Journal of Logic and Computation, Vol. 4, No. 5, pp.531–579
  49. ^ Bellik, Y. (2001). "Technical requirements for a successful multimodal interaction", International Workshop on Information Presentation and Natural Multimodal Dialogue, Verona, Italy, 14–15 December
  50. ^ Lee, Y.C.; Chin, F. (1995). "An Iconic Query Language for Topological Relationship in GIS". International Journal of geographical Information Systems 9(1). pp. 25-46
  51. ^ Calcinelli, D.; Mainguenaud, M. (1994). "Cigales, a visual language for geographic information system: the user interface". Journal of Visual Languages and Computing 5(2). pp. 113-132
  52. ^ Ferri, F.; Rafanelli, M. (2005). "GeoPQL: A Geographical Pictorial Query Language That Resolves Ambiguities in Query Interpretation". J. Data Semantics III. pp.50-80
  53. ^ Suhm, B., Myers, B. and Waibel, A. (1999). "Model-based and empirical evaluation of multimodal interactive error correction". In Proc. Of CHI'99, May, 1999, pp. 584-591

External links[edit]

Disclaimer

None of the audio/visual content is hosted on this site. All media is embedded from other sites such as GoogleVideo, Wikipedia, YouTube etc. Therefore, this site has no control over the copyright issues of the streaming media.

All issues concerning copyright violations should be aimed at the sites hosting the material. This site does not host any of the streaming media and the owner has not uploaded any of the material to the video hosting servers. Anyone can find the same content on Google Video or YouTube by themselves.

The owner of this site cannot know which documentaries are in public domain, which has been uploaded to e.g. YouTube by the owner and which has been uploaded without permission. The copyright owner must contact the source if he wants his material off the Internet completely.

Powered by YouTube
Wikipedia content is licensed under the GFDL and (CC) license