Graphical Communication in Human-Computer Interaction

GRACE - 6296

Work Area: Models and Theory for Human Computer Interaction

Keywords human-computer interaction, natural language, graphical representation, animation, multi-modal systems

Start Date: 1 August 1992 / Duration: 36 months / Status: finished

[ participants / contact ]

Abstract GRACE aims to develop a theory of the cognitive impact of modality choice in communication between people and machines, especially the choice and combination of language, graphics and animation. The theory should serve as the basis for generalising interface taxonmomy, design and evaluation. Formal semantic theory will guide interface design, which undergoes psychological evaluation to study the impact of closely comparable information presented in different modality combinations.


A theory of the cognitive impact of choice of modalities in communication between people and machines will be developed, especially the choice and combination of language, graphics and animation: ie, the modality allocation problem. The theory is to serve as the basis for generalising interface taxonomy, design and evaluation. It will take a modality-independent characterisation of information and then explore the cognitive effects of translating the same information into different media and combinations of media. Hence semantic theory will guide interface design which undergoes psychological evaluation to study the impact of closely comparable information presented in different modality combinations.

Approach and Methods

Methods of semantic analysis developed for natural language discourse will be applied to the more general case of multi-media dialogue between user and computer, thus bridging the gap between "top-down" and "bottom-up" approaches to human-computer interaction (HCI). GRACE's central theoretical claim is that the key to understanding the graphical modality is its specificity: graphics forces the over-determination of information in a message. The major aspects of the work include: characterisation of information-modality mappings in terms of relevant parameters; utilisation of the analysis offered by the SITUE functions described by top-down HCI as input to the semantic analysis of interactive multi-modal communications; development of a taxonomy of existing multi-modal interfaces based on this general theory; development of cognitive characterisations of the operation of specificity and of abstractive conventions of interpretation; extension of existing applications of the theory in static graphics to animation; exploration of the relation between graphical reasoning and model-building techniques from constraint logic programming and AI; development of a software tool to support experimentation with different interface parameters, and demonstration of optimal interface techniques.

Progress and Results

To date, we have focused on foundational issues in the semantics and pragmatics of graphical representations, and on gathering data on: graphically-based interactions between humans; interface design practice; and the effects on learning of exposure to graphics.

Progress has been made in the following areas. First, we have characterised more formally our key idea of specificity, and investigated criteria distinguishing multimodal and multimedia systems from traditional interfaces. Secondly, we have developed a preliminary methodology for selecting combinations of modalities in interface design, and developed a software workbench, for investigating the theoretical space of graphics. Thirdly, we have substantially developed an algebraic semantics for systems of graphically-based reasoning-support; we have also carried out empirical studies on the effects of graphics on the acquisition of abstract reasoning skills. Fourthly, we have developed an event-based logical characterisation of animations, and studied the effects on comprehension of dynamic display techniques. Fifthly, we have surveyed the applicability to multimodal interfaces of existing theories of linguistic discourse structure, and used a version of Gestalt theory to model implicature in network diagrams. Sixthly, we have carried out empirical studies of human-human graphical interactions. Finally, we have carried out interviews with designers to elicit design rationales.


GRACE is expected to contribute to the development of HCI, discourse and animation theory, and to practical improvements in computer interface design, design methodology, and, further downstream, to produce training material relating the project's results to interface design practice. Exploitation will be facilitated by the establishment and maintenance of a set of industrial "uncles", generally IT providers, who will be expected to monitor project progress and attend annual briefings and specific workshops. Each project partner will support at least two such uncles. We expect that the relationship between the project and its uncles will be mutual: uncles will gain early access to project results, while the project will gain access to uncles' experience with real-world problems.

Latest Publications

Information Dissemination Activies

Partners submitted a number of journal papers, and discussed preliminary results at international workshops. We demonstrated software at InterCHI93 (Amsterdam); and organised a workshop on graphics and multimodality at the 1992 European Summer School in Language, Logic and Information (Colchester).


University of Edinburgh - UK
Buccleuch Place 2


Roskilde Universitet - DK
Universiteit van Amsterdam - NL
Katholieke Universiteit Leuven - B


Prof. K. Stenning
tel +44/31 650 4445
fax +44/31 650 4587

LTR synopses home page LTR work area index LTR acronym index LTR number index LTR Projects index
All synopses home page all acronyms index all numbers index

GRACE - 6296, August 1994

please address enquiries to the ESPRIT Information Desk

html version of synopsis by Nick Cook