Improving Graphical User Interfaces for Computer Music
Applications
Adrian Freed
Introduction
This note is a plea to the computer music community to aim
higher in the development of new graphical tools and better
graphical user interfaces (GUIs) for computer music
applications.
It has been 10 years now since user interfaces such as that of
Digidesign Inc's Sound Designer and my own MacMix were designed.
It is distressing to see so how many recently built programs are
simple permutations of the same graphical elements. The
interactive techniques used in MacMix (direct manipulation with
mouse of 2-D polygonal objects in overlapping rectangular
windows) were obsolete when I used them (by at least a decade).
My defense is that computing performance of the day did not
afford more exciting techniques. Now the situation is completely
turned around. RISC computing performance exceeds the
requirements of the old bitmap 2-D graphics paradigm by a huge
factor.
First we should dismantle roadblocks hindering more
interesting use of graphical user interfaces: portability and
compatibility. These requirements have lead to very conservative
choices. Of course, as researchers, we have no excuse for
favoring these requirements over innovation. Fortunately, the
personal computer and workstation industry has just successfully
rallyed around a strong, reliable and portable standard for 3D
graphics: OpenGL (Neider
1993).
Features of OpenGL that are relevant to the computer music
developer include:
- client/server architecture;
- strong integration with X and other windowing systems such
as Microsoft Windows and the Apple Macintosh Toolbox;
- broad range of geometric primitives including points,
lines, polygons, images and bitmaps;
- binding for C and C++; and
- only 120 basic functions in the library
As for device compatability, there is little point supporting
small screens or monochrome screens. The price-leading monitors
are now color. Most new computers have enough video RAM for
16-bit color standard. By the time new software matures,
computers will have 24-bit, million pixel graphics support as
standard features.
Proposals
Here is a list of suggestions for future graphical user
interfaces for computer music.
- Thoughtful use of color - not just "pastel desktop and
menus" or color spectragrams. (Wolff and Yaeger
1993)--Tech Note 3, "Visualization of floating-point data"
pp. 237 illustrates some of the pitfalls here. Color is a
virtually unexplored and potentially powerful element in "music
visualization."
- 3-D object representations--with careful use of
perspective, fog, depth cuing and transparency--aid in the
manipulation and processing of large numbers of objects, see
e.g., (Robertson
et al 1991, 1993), (Card et al. 1991), (Mackinlay et al.
1991), and (Bier
et al 1993). The multi-dimensionality of musical data begs
for higher dimensions of control and representation impossible
with the current "paper on a desktop" metaphor.
- Abandonment of the "data in files" model in favor of the
mature database and multidimensional search techniques used in
geometric modelling, as described in (Nielson et al. 1994) and
(Matousek 1994).
Many of the interesting questions we are asking are about
properties of very large collections of different objects, e.g.
databases of analyzed sounds of musical instrument over many
pitches, loudness, and playing styles. We need to move beyond
limited, file-based, single-data-type applications towards
models that support richer data types, visualization paradigms
and distributed storage such as the model behind the rapidly
evolving world-wide web.
- Support for input devices with higher control bandwidth and
dimensionality than the mouse, as in
(Buxton 1993). We need to integrate new kinds of keyboards,
a broader range of physical gestures, vocalization, and even
non-human control sources.
- Integration of real-time sound synthesis throughout the
graphical interface, e.g. cursors on frequency axis that
synthesize a continous reference tone as they are moved
(Gaver et
al.1990-1993).
Suggested Reading
3-D Interface Paradigms
These references describe some of the recent work (at Xerox
PARC and elsewhere) on advanced software interfaces for the
presentation and manipulation of complex and very large
information structures.
Robertson, G.G.;
Card, S.K.; Mackinlay, J.D. Information Visualization using
3-D Interactive Animation. Communications of the ACM,
April 1993, vol.36, (no.4):56-7.
Mackinlay, J.D.;
Robertson, G.G.; Card, S.K. The Perspective Wall: Detail
and Context Smoothly Integrated. IN: Human Factors in
Computing Systems. Reaching Through Technology. CHI '91.
Conference Proceedings. (Human Factors in Computing Systems.
Reaching Through Technology. CHI '91. Conference Proceedings, New
Orleans, LA, USA, 27 April-2 May 1991). Edited by: Robertson,
S.P.; Olson, G.M.; Olson, J.S. New York, NY, USA: ACM, 1991. p.
173-9.
Card, S.K.; Robertson, G.G.;
Mackinlay, J.D. The Information Visualizer, an Information
Workspace. IN: Human Factors in Computing Systems.
Reaching Through Technology. CHI '91. Conference Proceedings.
(Human Factors in Computing Systems. Reaching Through Technology.
CHI '91. Conference Proceedings, New Orleans, LA, USA, 27 April-2
May 1991). Edited by: Robertson, S.P.; Olson, G.M.; Olson, J.S.
New York, NY, USA: ACM, 1991. p. 181-8.
Robertson, G.G.;
Mackinlay, J.D.; Card, S.K. Cone Trees: Animated 3-D
Visualizations of Hierarchical Information. IN: Human
Factors in Computing Systems. Reaching Through Technology. CHI
'91. Conference Proceedings. (Human Factors in Computing Systems.
Reaching Through Technology. CHI '91. Conference Proceedings, New
Orleans, LA, USA, 27 April-2 May 1991). Edited by: Robertson,
S.P.; Olson, G.M.; Olson, J.S. New York, NY, USA: ACM, 1991. p.
189-94.
Bier, E.A.; Stone, M.C.;
Pier, K.; Buxton, W.; and others. Toolglass and Magic
Lenses: The See-through Interface. IN: Computer Graphics
Proceedings. (Computer Graphics ProceedingsProceeding of SIGGRAPH
20th Annual International Conference on Computer Graphics and
Interactive Techniques. The Eye of Technology, Anaheim, CA, USA,
1-6 Aug. 1993). New York, NY, USA: ACM, 1993. p. 73-80.
Auditory Icons
William Gaver has developed several software user interfaces
to integrate non-speach audio into the Macintosh Finder and the
Xerox PARC Alternate Reality Kit.
Gaver, W. and Smith, R. (1990).
Auditory Icons in Large-scale Collaborative
Environments. In D. Diaper et al. (Eds), Human-Computer
Interaction - INTERACT '90, Elsevier Science Publishers B.V.
(North-Holland), 735-740.
Gaver, W. (1993). Synthesizing Auditory Icons,
Proceedings of INTERCHI'93, 228-235.
Gaver, W. and Smith, R. (1990). Auditory Icons in
Large-scale Collaborative Environments. In D. Diaper et
al. (Eds), Human-Computer Interaction - INTERACT '90, Elsevier
Science Publishers B.V. (North-Holland), 735-740.
Gaver, W., Smith, R. and O'Shea, T. (1991). Effective
Sounds in Complex Systems: The ARKola Simulation,
Proceedings of CHI'91, 85-90.
Open GL
The OpenGL graphics interface library is stable and
well-documented in the following references.
Neider, Jackie. OpenGL
Programming Guide: The Official Guide to Learning OpenGL, Release
1. OpenGL Architecture Review Board, Jackie Neider, Tom
Davis, Mason Woo. Reading, Mass. : Addison-Wesley, c1993.
OpenGL Reference Manual: The Official Reference Document
for OpenGL, Release 1 / OpenGL Architecture Review Board.
Reading, Mass. : Addison-Wesley, c1993.
Data Representation and Access
These problems, addressed in the scientific visualization
community, are the same ones we face in timbre
representation:
Nielson, G.M.; Brunet, P.;
Gross, M.; Hagen, H.; and others. Research Issues in Data
Modeling for Scientific Visualization. IEEE Computer
Graphics and Applications, March 1994, vol.14,
(no.2):70-3.
Matousek, J. Geometric
Range Searching. ACM Computing Surveys, Dec. 1994, vol.26,
(no.4):421-61.
Tips and Pitfalls
Graphics programming offers its own special challenges, but a
muture literature on effective and efficient graphical
programming techniques is now available:
Wolff and Yaeger Visualization of
Natural Phenomena. Robert S. Wolff, Larry Yaeger.
Santa Clara, Calif. : TELOS, c1993.
Graphics Gems, Andrew Glassner (ed.), Academic
Press 1990, ISBN 0-12-286165-5
Graphics Gems II, James Arvo (ed.), Academic
Press 1991, ISBN 0-12-064480-0
Graphics Gems III, David Kirk (ed.), Academic
Press 1992, ISBN 0-12-409670-0 (with IBM disk) or 0-12-409671-9
(with Mac disk)
Graphics Gems IV, Paul Heckbert (ed.), Academic
Press 1994, ISBN 0-12-336156-7 with MAC floppy, ISBN
0-12-336155-9 with PC floppy
Encouraging Beginnings
The following papers were part of a session at ICMC95 devoted
to GUI's for computer music:
Robin Bargar, Bryan Holloway, Xavier Rodet, Chris Hartman
Defining Spectral Surfaces
Heinrich Taube, Tobias Kunze Capella: A Graphical
Interface for Algorithmic Composition
Richard Polfreman, John Sapsford-Francis A Human Factors
Approach to Computer Music Systems User-Interface
Design
Insook Choi, Robin Bargar, Camille Goudeseune A Manifold
Interface for a High Dimensional Control Space
Conclusion
I hope this note spawns wide discussion of GUI design for
computer music applications and hereby offer an electronic forum
to track these discussions:www.cnmat.berkeley.edu:~adrian/GUIforCM.html.
I welcome and look forward to your contributions.
Adrian Freed
Berkeley, California, USA
adrian [at] cnmat [dot] berkeley [dot] edu