Designing Interactive Multimedia

Abstract

This paper presents contrasting metaphors and paradigms for designing interactive media interfaces. Multimedia interface designers and researchers with diverse backgrounds discuss their own design approaches and important design issues. Discussion of these issues is continued beyond this paper through a web site.

Introduction

Multimedia applications are finding their way into ever expanding corners of our lives: in business, education, public spaces, and private entertainment. Rapidly improving technology and tools enable multimedia developers to create complex information spaces and varied forms of interaction that were only dreamed of before. Yet our interface design methodologies do not appear to have kept pace with these improvements. Books providing guidance to multimedia designers tend to either reiterate two-dimensional graphic design principles [7] or promote one of a few metaphors: the hyper-linked textbook [4], the desktop [10], the stage[5]. The trouble with these approaches is that they do not consider the rich, multi-dimensional information space that multimedia is capable of representing. Studies increasingly show that time and space incorporated in the interface add interest and depth, and even enhance navigation [6]. Others point out the need to tailor interface techniques to the intended audience [1]. Unfortunately, many applications fall short of their potential because of poor design.

As multimedia developers from diverse backgrounds, we have gathered to compare and contrast our approaches to designing interactive multimedia. We also intend to raise, and possibly answer, some critical questions. Can we establish clear guidelines or methodologies for design, or must we resign ourselves to relying on intuition only? What methodologies are most effective, and for what applications? Are current metaphors adequate, or do we need new ones? How does the background of the designer influence the outcome of the interface design? We hope to stimulate a sorely needed intellectual discourse that we hope will continue beyond these pages.

To facilitate the continuing discussion, we have developed a web site that presents these issues and contrasting points of view. The web pages include the panelists' presentation slides. We have also included a form enabling others with experience in multimedia development to present their own methodologies, paradigms, and metaphors, thereby providing a richer set of opinions to visitors of the site. We invite interested parties to visit this site at http://www.brooklyn.cuny.edu/bc/ahp/LScarlatos.html.

Lori Scarlatos: The Art of Multimedia Design

Many designers of multimedia systems rightfully think of themselves as artists. A work of art communicates ideas to an intended audience by emphasizing important relationships and diminishing irrelevant information. Art defines a space that is rich in micro and macro information, inviting exploration. Art is meant to be viewed many times from many different angles, always leading to new discoveries: the more a viewer engages the art, the more that the viewer gets out of it. These are characteristics that we would like our interactive multimedia systems to have.

Although artists may appear to rely purely on intuition, they actually draw from a rich tradition of well established processes and techniques that have been developed over the centuries. Beyond their original inspirations, which are truly creative, artists (consciously or unconsciously) develop their work through a series of overlapping processes. These processes include:

1. Filtering the information being presented, so as not to clutter the piece with irrelevant details;
2. Selecting representational elements and styles that best convey the artist's ideas;
3. Creating layers of information/detail/meaning that will reward the attentive viewer with ever more new discoveries as the viewer's vantage point or focus changes; and
4. Composing the elements of the piece in such a way as to establish relationships and guide the viewer through the piece, so that even the quickest study reveals the essence of the work.

These processes, which correlate well to the multimedia designer's tasks, form the basis of a multimedia design methodology. In fact, I propose that this design methodology suggests a set of multimedia authoring tools. Although outside inspiration will still be required, these tools could provide a way to systematically create innovative multimedia applications, and help many developers to realize their creative potential.

Rudy Darken: Designing for Wayfinding in Virtual Environments

The field of psychology has endeavored over the past 100 years to describe how man relates to his environment. What we find are deep analytical discussions concerning the physical environment and man's place in it; how we extract information from the environment, how we process that information and eventually act on it. Now along comes the notion of a "virtual" environment - and all the rules changed. All the assumptions psychologists have made concerning the characteristics and features of the environment are now invalid, or at least demand reevaluation. Much of what we are doing here is reevaluating a body of research within the context of a new medium. How are virtual environments different from real environments and what behavioral repercussions does this difference have on human activity and the ability to do real tasks?

My work involves one small piece of this puzzle: the ability to form a mental representation of an environment - virtual (VE) or real (RE). While my earlier work was concerned only with wayfinding performance enhancement within the VE, it has lately been extended to include the transfer of spatial knowledge acquired in the VE back to the RE that it represents. Note that my primary interest here is in the cognitive elements of navigation rather than the actual activity of movement (locomotion or maneuvering) from one point to another. While they are certainly inter-related, at this early stage of research it seems appropriate to investigate them separately.

The problem of disorientation in VE's usually becomes apparent when a person is placed in any large space (i.e. an environment that is too large to be viewed from a single vantage point). However, this same level of disorientation is not apparent (at least to the same degree) in similar RE's. Accepting the fact that we can't support the same level of stimulus fidelity in the VE that would be present in the RE, we need to understand what stimuli are most useful for the presentation of spatial information. This will provide a priority scheme for the rendering of the environment (both visual, aural, and other) enabling the removal, when necessary, of non-essential stimuli. Furthermore, since all aspects of the VE are under our control, we can augment spatial information (e.g. dynamic maps, markers, etc.) in ways not possible in the RE.

It seems obvious that we can acquire spatial knowledge about a specific space from navigating a virtual representation of it. However, what we don't understand is exactly what knowledge is acquired and how it differs from that acquired from the actual environment. One thing we can always count on (at least in my lifetime) is that VE's will always be lesser representations of RE's, especially in terms of stimulus fidelity. That being the case, we now want to know what effect degraded stimuli have on spatial knowledge acquisition.

Komei Harada: Scenario Design Independent from Media Contents

What characterizes the design of an interactive multimedia application is that it is divided into creating the media contents and developing the scenario. While the media contents usually determine the formats of images, video, audio and text that are common to the platform, the format of the scenario depends on the authoring system used, and therefore may be represented by various metaphors. The quality of the interactive media relies upon the organization of the scenario as well as the quality of media contents.

With currently available authoring systems, multimedia developers cannot begin to design screens and user interaction until the media contents are created. Therefore, the scenario editing process highly depends on the creation of media contents. The scenario structure of interactive media is basically independent from the media contents. By separating the scenario design process from the media contents creation, authors can take advantage of these strategies:

1. Top-down design: creating the scenario of a large-scale application starting from the global design;
2. Collaboration: sharing the authoring processes among different designers; and
3. Reuse: using the same scenario structure in different application.

I propose a design methodology in which the scenario design and the media contents creation are independent from each other. In order to keep the consistency between the scenario and media contents when they are being edited separately, the concept of "surrogate media" is introduced. Surrogate media serves as the identifier of each media element while designing the scenario. It also specifies the actual media contents to be created.

In order to realize this scenario design methodology, we at NEC have developed the Anecdote Multimedia Storyboarding System. The user interface of Anecdote is based on the storyboarding technique where the "sketch" is mainly used as surrogate media. A sketch can be created much more easily than the actual media contents, and it visually represents the media contents to be created in the scenario.

Carrie Heeter: Creativity-Driven Software Design

At the Michigan State University Comm Tech Lab, we place creativity, intuition, and flexibility at the center of every phase of software design. No model can create exciting interface design with dull participants. Intuition and designer background are very important and can not be dismissed.

1) Defining the Problem Statement - 1 to 3 weeks

We start with a general content domain or problem space and begin a process of clarifying intended users and emotional and informational goals. We immerse ourselves in the topic, reading, seeking media examples, observing and interviewing people involved with the content, getting to know users, and brainstorming with content experts. From this base, the project director and other relevant stakeholders reach agreement on a problem statement that defines the intended users and goals of the software.

2) Finding a Vehicle - 1 to 3 weeks

Next is a period of brainstorming core defining elements of the software. Examples include a metaphor (i.e., a microbe zoo), or a function (i.e., combining text, digital video of ASL signs and speech synthesis for a communication device), or a point of view (focusing on personal stories of breast cancer survivors), or a style (rendered 3D models). The intent is to find a powerful mix of the potential of technology, art, and function. The whole project should not be mapped out too early because each stage should be allowed to introduce changes.

3) Prototyping - 1 to 2 months

Initially we develop one small chunk of the project, to test drive and revise the vehicle. At this stage we also experiment with artistic and technical methodologies before engaging in large scale implementation.

4) Implementation - 6 to 12 months

At this stage, we more specifically define the scope of the content and then replicate and expand upon the successful prototype, maintaining harmony and balance as it grows. Stages 1 through 3 are repeated on a much smaller scale during the implementation phase each time we encounter undefined elements of the design or find reason to redefine elements.

5) Questioning Assumptions and Finishing - 2 to 3 months

Implementation tends to be a long process of mostly doing more of what was planned. Toward the end, it is helpful to stop and question the entire interface. Significant changes in form and function are often made in the final two months of a multiyear project. The designers' and programmers' perspectives have changed. We have observed users with the software. The GUI interface is getting filled up and brittle. Methods of programming that made sense at the beginning can be much more elegant and flexible. We at least stop and consider changing the look and feel and function of the software, though usually not the content, in the final phase.

Richard Muller: Ownership of the Development Process

Interactive media development is an inherently multi-disciplinary process, involving graphic designers, programmers, human factors specialists, and content experts, among others. "Ownership" of such a project is often an ambiguous matter: members of the development group may have different individual and professional needs as well as widely divergent notions of what success or failure might mean in the design process. And in many situations, budgetary and deadline concerns overwhelm other factors which influence design and implementation decisions.

Nowhere is this situation more evident and potentially problematic than in the design of the visual interface. I intend to explore different models of communication and responsibility among the members of the design group, in an attempt to discover patterns of organization which satisfy the needs of creative, technical and content specialists.

Ben Shneiderman: Information Visualization with Dynamic Queries, Starfield Displays, and LifeLines

The future of user interfaces is in the direction of larger, higher resolution screens, that present perceptually-rich and information-abundant displays. With such designs, the worrisome flood of information can be turned into a productive river of knowledge. My experience during the past five years has been that visual query formulation and visual display of results can be combined with the successful strategies of direct manipulation. Human perceptual skills are quite remarkable and largely underutilized in current information and computing systems. Based on this insight, I have developed dynamic queries, starfield displays, treemaps, treebrowsers, and a variety of widgets to present, search, browse, filter, and compare rich information spaces.

Dynamic queries are animated user-controlled displays that show information in response to movements of sliders, buttons, maps, or other widgets. For example, in the HomeFinder the users see points of light on a map representing homes for sale. As they shift sliders for the price, number of bedrooms, etc. the points of light come and go within 100 milliseconds, offering a quick understanding of how many and where suitable homes are being sold. Clicking on a point of light produces a full description and, potentially, a picture of the house.

A starfield display was created for the FilmFinder, which provided visual access to a database of films. The films were arranged as color coded rectangles along the x-axis by the production year and along the y-axis by popularity. Recent popular films were in the upper right hand corner. Zoombars (a variant of scroll bars) enabled users to zoom-in in milliseconds on the desired region. When less than 25 films were on the screen, the film titles appeared and when the users clicked on a film's rectangle, a dialog box would appear giving full information and an image from the film. The commercial version of starfield displays became available late in 1996 from IVEE Development under the name Spot Fire.

In the LifeLines prototype, we applied multiple timeline representations to personal histories such as medical records. Horizontal and vertical zooming, focusing, and filtering enabled us to represent complex histories and support exploration by clicking on timelines to get detailed information.

There are many visual alternatives but the basic principle for browsing and searching might be summarized as the Visual Information Seeking Mantra:

Overview first, zoom and filter, then details-on-demand

In several projects I found myself rediscovering this principle and therefore wrote it down and highlighted it as a continuing reminder. If we can design systems with effective visual displays, direct manipulation interfaces, and dynamic queries then users will be able to responsibly and confidently take on even more ambitious tasks.

Biographies

Lori Scarlatos is an assistant professor at Brooklyn College where she teaches computer science and TV/radio courses, and leads the college's distance learning efforts. She also works on computer based training and other multimedia projects for Art & Sol in New York. With a B.F.A. in painting from Pratt Institute and a Ph.D. in computer science from SUNY at Stony Brook, Dr. Scarlatos approaches multimedia design as both an artist and a computer scientist. In the mid-1980's she led the development of real-time PC animation software at Lecht Sciences, Inc., where she was a vice president. In Grumman Data Systems' research department she worked on numerous cartographic and visualization contracts for the military, and developed her original research on terrain models for real time systems. Prior to joining Brooklyn College, Dr. Scarlatos taught multimedia as both a design discipline and a computer science at Hampshire College.

Rudy Darken is an Assistant Professor of Computer Science at the Naval Postgraduate School in Monterey, California. He joined the department in July of 1996, having been at the Naval Research Laboratory in Washington, D.C. since 1991 as director and co-founder of the Tactical Electronic Warfare Division's Virtual Environment Laboratory. His research has been primarily focused on human factors in virtual environments with emphasis on navigation and wayfinding in large-scale virtual worlds. His background includes experience in interface design, collaborative computing, computer augmented training systems, team training systems, real-time visual simulation, computer graphics, and computer animation. Recent research initiatives include spatial audio in aircraft training and operations and wireless mobile computing, or more to the point, virtual environment technology applied to real world tasks. He is a member of the editorial board of PRESENCE Journal. He received his B.S. in Computer Science Engineering from the University of Illinois at Chicago in 1990 and his M.S. and D.Sc. degrees in Computer Science from The George Washington University in 1993 and 1995, respectively.

Komei Harada is a research staff member in the C & C Research Laboratories (CCRL) of NEC USA, Inc. Since he obtained his Bachelor's degree in metallurgy from University of Tokyo and joined NEC Corp., Japan, he has been engaged in the research of multimedia authoring systems and the development of interactive multimedia applications. After joining CCRL in San Jose, California, in 1995, he has primarily focused on the research of early-stage multimedia authoring support and developed the Anecdote Multimedia Storyboarding System.

Dr. Carrie Heeter is founding Director of the Communication Technology Laboratory and Director of Creative Development for Michigan State University's virtual university initiative. Heeter is a professor of digital media arts and technology, teaching interface design and interactivity. She has produced 10 videodiscs, six commercial educational CD-ROMs, numerous web sites, and has four additional educational CD-ROM projects in progress. Comm Tech Lab designs have been exhibited nationally and internationally at peer reviewed galleries and have won awards, including Discover Magazine's Software Innovation of the Year Award, New Media's Envision Award and MacroMedia's People's Choice Award. Her work has been funded by the National Science Foundation, U.S. Department of Education, McGraw-Hill Publishers, William C. Brown Publishers, the American Cancer Society, NASA, CIESIN, the State of Michigan Department of Public Health and Legislative Service Bureau, and Ameritech. Heeter also publishes academic research on interactive multimedia and virtual reality.

Richard Muller is Associate Professor of Communication and Computer Studies at Hampshire College in Amherst, Massachusetts. He teaches courses in interactive media design and production, and has been involved in the design and realization of both Web- and CD-ROM-based interactive media. He is President of Tinker Hill Associates, a Massachusetts-based digital media production company.

Ben Shneiderman is a Professor in the Department of Computer Science, Head of the Human-Computer Interaction Laboratory, and Member of the Institute for Systems Research, all at the University of Maryland at College Park. He regularly teaches popular short courses and organizes an annual satellite television presentation on User Interface Strategies seen by thousands of professionals since 1987. Dr. Shneiderman is the author of Software Psychology: Human Factors in Computer and Information Systems (1980) and Designing the User Interface: Strategies for Effective Human-Computer Interaction (1987, third edition 1997. In addition he has co-authored two textbooks, edited three technical books, and published more than 190 technical papers and book chapters. Ben Shneiderman has been on the Editorial Advisory Boards of nine journals including the newly formed ACM Transactions on Computer- Human Interaction and the ACM Interactions. He edits the Ablex Publishing Co. book series on "Human-Computer Interaction." He has consulted and lectured for many organizations including Apple, AT&T, Citicorp, GE, Honeywell, IBM, Intel, Library of Congress, NASA, NCR, and university research groups.

References

[1] Brouwer-Janse, M.D., Suri, J.F., Yawitz, M., de Vries, G., Fozard, J.L. and Coleman, R. User Interfaces for Young and Old, Interactions, IV.2 (March/April 1997), 34-46.

[2] Darken, R.P. and Silbert, J.L. Wayfinding Strategies and Behaviors in Large Virtual Worlds, Proc. of ACM CHI96 Conference, (April 1996), 142-149.

[3] Harada, K. and Hara, Y. Anecdote: A Multimedia Storyboarding System with Seamless Authoring Support, Proc. of ACM Multimedia 96 Conference, (November 1996), 341-351.

[4] Kristoff, R. and Satran, A. Interactivity by design: creating and communicating with new media, Adobe Press, Mountain View, CA, 1995.

[5] Laurel, B. Computers as Theatre, Addison-Wesley, Reading, MA, 1993.

[6] Mead, P. and Pacione, C. Time and Space, Interactions, III.2 (March 1996), 68-77.

[7] Mullet, K. and Sano, D. Designing Visual Interfaces, SunSoft Press, Prentice-Hall, Mountain View, CA, 1995.

[8] Plaisant, C., Carr, D., and Shneiderman, B., Image-browser taxonomy and guidelines for designers, IEEE Software 12, 2 (March 1995), 21-32.

[9] Plaisant, C., Rose, A., Milash, B., Widoff, S., and Shneiderman, B., LifeLines: Visualizing personal histories, Proc. of ACM CHI96 Conference (April 1996), 221-227, 518.

[10] Tognazzini, B. Tog on Interface, Addison-Wesley, Reading, MA, 1992.