Improving Graphical User Interfaces for Computer Music Applications

Adrian Freed

Introduction

This note is a plea to the computer music community to aim higher in the development of new graphical tools and better graphical user interfaces (GUIs) for computer music applications.

It has been 10 years now since user interfaces such as that of Digidesign Inc's Sound Designer and my own MacMix were designed. It is distressing to see so how many recently built programs are simple permutations of the same graphical elements. The interactive techniques used in MacMix (direct manipulation with mouse of 2-D polygonal objects in overlapping rectangular windows) were obsolete when I used them (by at least a decade). My defense is that computing performance of the day did not afford more exciting techniques. Now the situation is completely turned around. RISC computing performance exceeds the requirements of the old bitmap 2-D graphics paradigm by a huge factor.

First we should dismantle roadblocks hindering more interesting use of graphical user interfaces: portability and compatibility. These requirements have lead to very conservative choices. Of course, as researchers, we have no excuse for favoring these requirements over innovation. Fortunately, the personal computer and workstation industry has just successfully rallyed around a strong, reliable and portable standard for 3D graphics: OpenGL (Neider 1993).

Features of OpenGL that are relevant to the computer music developer include:

• client/server architecture;
• strong integration with X and other windowing systems such as Microsoft Windows and the Apple Macintosh Toolbox;
• broad range of geometric primitives including points, lines, polygons, images and bitmaps;
• binding for C and C++; and
• only 120 basic functions in the library

As for device compatability, there is little point supporting small screens or monochrome screens. The price-leading monitors are now color. Most new computers have enough video RAM for 16-bit color standard. By the time new software matures, computers will have 24-bit, million pixel graphics support as standard features.

Proposals

Here is a list of suggestions for future graphical user interfaces for computer music.

1. Thoughtful use of color - not just "pastel desktop and menus" or color spectragrams. (Wolff and Yaeger 1993)--Tech Note 3, "Visualization of floating-point data" pp. 237 illustrates some of the pitfalls here. Color is a virtually unexplored and potentially powerful element in "music visualization."
2. 3-D object representations--with careful use of perspective, fog, depth cuing and transparency--aid in the manipulation and processing of large numbers of objects, see e.g., (Robertson et al 1991, 1993), (Card et al. 1991), (Mackinlay et al. 1991), and (Bier et al 1993). The multi-dimensionality of musical data begs for higher dimensions of control and representation impossible with the current "paper on a desktop" metaphor.
3. Abandonment of the "data in files" model in favor of the mature database and multidimensional search techniques used in geometric modelling, as described in (Nielson et al. 1994) and (Matousek 1994). Many of the interesting questions we are asking are about properties of very large collections of different objects, e.g. databases of analyzed sounds of musical instrument over many pitches, loudness, and playing styles. We need to move beyond limited, file-based, single-data-type applications towards models that support richer data types, visualization paradigms and distributed storage such as the model behind the rapidly evolving world-wide web.
4. Support for input devices with higher control bandwidth and dimensionality than the mouse, as in (Buxton 1993). We need to integrate new kinds of keyboards, a broader range of physical gestures, vocalization, and even non-human control sources.
5. Integration of real-time sound synthesis throughout the graphical interface, e.g. cursors on frequency axis that synthesize a continous reference tone as they are moved (Gaver et al.1990-1993).

Suggested Reading

3-D Interface Paradigms

These references describe some of the recent work (at Xerox PARC and elsewhere) on advanced software interfaces for the presentation and manipulation of complex and very large information structures.

Robertson, G.G.; Card, S.K.; Mackinlay, J.D. Information Visualization using 3-D Interactive Animation. Communications of the ACM, April 1993, vol.36, (no.4):56-7.

Mackinlay, J.D.; Robertson, G.G.; Card, S.K. The Perspective Wall: Detail and Context Smoothly Integrated. IN: Human Factors in Computing Systems. Reaching Through Technology. CHI '91. Conference Proceedings. (Human Factors in Computing Systems. Reaching Through Technology. CHI '91. Conference Proceedings, New Orleans, LA, USA, 27 April-2 May 1991). Edited by: Robertson, S.P.; Olson, G.M.; Olson, J.S. New York, NY, USA: ACM, 1991. p. 173-9.

Card, S.K.; Robertson, G.G.; Mackinlay, J.D. The Information Visualizer, an Information Workspace. IN: Human Factors in Computing Systems. Reaching Through Technology. CHI '91. Conference Proceedings. (Human Factors in Computing Systems. Reaching Through Technology. CHI '91. Conference Proceedings, New Orleans, LA, USA, 27 April-2 May 1991). Edited by: Robertson, S.P.; Olson, G.M.; Olson, J.S. New York, NY, USA: ACM, 1991. p. 181-8.

Robertson, G.G.; Mackinlay, J.D.; Card, S.K. Cone Trees: Animated 3-D Visualizations of Hierarchical Information. IN: Human Factors in Computing Systems. Reaching Through Technology. CHI '91. Conference Proceedings. (Human Factors in Computing Systems. Reaching Through Technology. CHI '91. Conference Proceedings, New Orleans, LA, USA, 27 April-2 May 1991). Edited by: Robertson, S.P.; Olson, G.M.; Olson, J.S. New York, NY, USA: ACM, 1991. p. 189-94.

Bier, E.A.; Stone, M.C.; Pier, K.; Buxton, W.; and others. Toolglass and Magic Lenses: The See-through Interface. IN: Computer Graphics Proceedings. (Computer Graphics ProceedingsProceeding of SIGGRAPH 20th Annual International Conference on Computer Graphics and Interactive Techniques. The Eye of Technology, Anaheim, CA, USA, 1-6 Aug. 1993). New York, NY, USA: ACM, 1993. p. 73-80.

Auditory Icons

William Gaver has developed several software user interfaces to integrate non-speach audio into the Macintosh Finder and the Xerox PARC Alternate Reality Kit.

Gaver, W. and Smith, R. (1990). Auditory Icons in Large-scale Collaborative Environments. In D. Diaper et al. (Eds), Human-Computer Interaction - INTERACT '90, Elsevier Science Publishers B.V. (North-Holland), 735-740.

Gaver, W. (1993). Synthesizing Auditory Icons, Proceedings of INTERCHI'93, 228-235.

Gaver, W. and Smith, R. (1990). Auditory Icons in Large-scale Collaborative Environments. In D. Diaper et al. (Eds), Human-Computer Interaction - INTERACT '90, Elsevier Science Publishers B.V. (North-Holland), 735-740.

Gaver, W., Smith, R. and O'Shea, T. (1991). Effective Sounds in Complex Systems: The ARKola Simulation, Proceedings of CHI'91, 85-90.

Open GL

The OpenGL graphics interface library is stable and well-documented in the following references.

Neider, Jackie. OpenGL Programming Guide: The Official Guide to Learning OpenGL, Release 1. OpenGL Architecture Review Board, Jackie Neider, Tom Davis, Mason Woo. Reading, Mass. : Addison-Wesley, c1993.

OpenGL Reference Manual: The Official Reference Document for OpenGL, Release 1 / OpenGL Architecture Review Board. Reading, Mass. : Addison-Wesley, c1993.

Data Representation and Access

These problems, addressed in the scientific visualization community, are the same ones we face in timbre representation:

Nielson, G.M.; Brunet, P.; Gross, M.; Hagen, H.; and others. Research Issues in Data Modeling for Scientific Visualization. IEEE Computer Graphics and Applications, March 1994, vol.14, (no.2):70-3.

Matousek, J. Geometric Range Searching. ACM Computing Surveys, Dec. 1994, vol.26, (no.4):421-61.

Tips and Pitfalls

Graphics programming offers its own special challenges, but a muture literature on effective and efficient graphical programming techniques is now available:

Wolff and Yaeger Visualization of Natural Phenomena. Robert S. Wolff, Larry Yaeger. Santa Clara, Calif. : TELOS, c1993.

Graphics Gems, Andrew Glassner (ed.), Academic Press 1990, ISBN 0-12-286165-5

Graphics Gems II, James Arvo (ed.), Academic Press 1991, ISBN 0-12-064480-0

Graphics Gems III, David Kirk (ed.), Academic Press 1992, ISBN 0-12-409670-0 (with IBM disk) or 0-12-409671-9 (with Mac disk)

Graphics Gems IV, Paul Heckbert (ed.), Academic Press 1994, ISBN 0-12-336156-7 with MAC floppy, ISBN 0-12-336155-9 with PC floppy

Encouraging Beginnings

The following papers were part of a session at ICMC95 devoted to GUI's for computer music:

Robin Bargar, Bryan Holloway, Xavier Rodet, Chris Hartman Defining Spectral Surfaces

Heinrich Taube, Tobias Kunze Capella: A Graphical Interface for Algorithmic Composition

Richard Polfreman, John Sapsford-Francis A Human Factors Approach to Computer Music Systems User-Interface Design

Insook Choi, Robin Bargar, Camille Goudeseune A Manifold Interface for a High Dimensional Control Space

Conclusion

I hope this note spawns wide discussion of GUI design for computer music applications and hereby offer an electronic forum to track these discussions:www.cnmat.berkeley.edu:~adrian/GUIforCM.html. I welcome and look forward to your contributions.

Adrian Freed Berkeley, California, USA adrian [at] cnmat [dot] berkeley [dot] edu