NASA SISM Intelligent Systems Project Human-Centered Computing Multimodal Interfaces

Approach

The HCC Multimodal Interfaces (MI) technical area is about integrating humans, computational devices, and environments in a seamless manner, leveraging the unique capabilities of each to satisfy system-level requirements. MI technologies include speech and natural language interfaces, multimedia systems, adaptive and intelligent interfaces and displays, ubiquitous computing, and mobile and wearable computing. Investments are selective, guided by NASA requirements. NASA's planned missions will require dramatic advances in the functional bandwidth and reliability of operator-system interfaces. We must find ways to couple humans and computers into cooperative, efficient systems. Team interfaces that reflect mixed-initiative models of content, activity, and dialogue will require advances in spoken and visual languages, flexible movement among levels of abstraction, and expert coordination of multiple representations. Multimodal interfaces go far beyond simple displays and voice-response systems. They help decision makers accomplish high-bandwidth, heterogeneous, real-time data integration and interpretation. Applications include remote science, model-based (predictive) control, onboard system management and procedure execution, and information management for ground operations. A representative example might be an intelligent maneuvering system for the International Space Station. The ISS will offer serious challenges to humans performing intricate docking and assembly maneuvers. An intelligent maneuvering system could enhance the performance of astronauts by integrating telerobotic systems with software for monitoring, scheduling, and decision support. Such an intelligent system would reduce the probability of mishaps, while freeing operators from routine monitoring and manipulation tasks. Another broad area of application is enhancement of air traffic safety and capacity. A new generation of human-centered performance aids is needed both on the ground and in the air. In the cockpit, a new generation of displays and interfaces could mitigate information overload and improve situation awareness. NASA is also developing human-centered systems that can assist pilots in flying severely damaged aircraft by reconfiguring the flight-control computer system. Interfaces must be useful even under difficult conditions, and their safety and reliability must be established in advance through rigorous design and testing. Engineering design of such flexible, robust performance-enhancement systems is beyond the state of the art. MI tasks are developing better models of attention, memory, conceptual structure, decision-making, learning, and higher-level perception needed for principled design and testing of novel human-computer systems. These will permit early, reliable design choices (minimizing life cycle cost) and reduce system risk.

Research Tasks

Multimodal Interfaces  Research Tasks
Principal Investigator	Lead Org.	Task Title
Robert O. Ambrose	JSC	Robonaut Intelligent Software
Christopher Assad	JPL	Multimedia Human-Computer Interfaces for Mission-Critical Systems
Durand R. Begault	ARC	Human-Centered Design of Multimodal Virtual Environment Displays
Theodore W. Berger	USC	Robust Speech Recognition using Dynamic Synapse Neural Networks
Alberto J. Cañas	UWF	Concept Maps for Mars Exploration
James L. Hieronymus	ARC	Dialogue Systems
Beth A. Hockey	RIACS	Advanced Capabilities for Spoken Dialogue-Natural Language Interface Systems
Richard M. Keller	ARC	A Testbed for Agent-Assisted Collaborative Scientific Experimentation
Patrick Langley	ISLE	Filtering Information in Complex Temporal Domains
David B. Leake	IU	Integrated Intelligent Support for Knowledge Capture, Refinement, and Sharing
Elizabeth E. Shriberg	SRI	Harnessing Speech Prosody for Robust Human-Computer Interaction
Steven A. Sloman	Brown	Causal Reasoning
Leonard J. Trejo	ARC	Multimodal Neuroelectric Human-Computer Interface Development

NASA | SISM | IS | Human-Centered Computing

IS Tasks | HCC Tasks