Beyond 0-5V: Expanding Sensor Integration Architectures

Adrian Freed, Rimas Avizienis, Matt Wright {adrian, rimas,matt }@cnmat.berkeley.edu

 

Abstract:

A new sensor integration system and its first incarnation is described. As well as supporting existing analog sensor arrays a new architecture allows for easy integration of the new generation of low-cost digital sensors used in computer music performance instruments and installation art.

Keywords

Gesture, sensor, MEMS, FPGA, network, OSC, configurability

1.     INTRODUCTION

After years of experiments and live musical productions with several revisions of analog sensor input systems for an audio connectivity processor [2] we have learned that the basic multichannel 0-5v analog input standard is not sufficient for access to interesting new sensing technologies.

 

The trend for voltage output sensors is towards lower output levels, e.g. 3V. MEMS sensors and hall effect sensors use clocked sampling techniques internally and they currently convert their measurands into analog signals that are then resampled. More accuracy and lower costs are achieved as the switch to digital outputs is made.

A new, simple way to support a wide variety of these new sensors is needed especially the emerging diverse, low-cost MEMS sensors. Unfortunately, no single serial digital communication standard has curried favor with the sensor vendors themselves necessitating "bridging" hardware to hide these complexities from users.

Another major requirement in many musical instrument and art installation projects is for considerably more input channels than is usually provided in USB data acquisition systems or current gesture input boxes. Many sensing problems are most easily addressed by using linear and 2-dimensional arrays of networked sensors. We routinely find need for more than 32 channels.

In this paper we review several sensor integration projects, outlining their specific requirements and then we describe how these requirements are addressed by a new architecture for sensor integration based on FPGA's and sensor identification. Finally, we describe the first implementation of this as the latest refinement of our ongoing connectivity processor project.

Table 1. Sensor Technology Sample

Device	Measurand	Analog	Digital
ADXL213	Acceleration		PWM
ADXL320	Acceleration	0-5V
ADIS16006	Acceleration		SPI
ADIS16100	Yaw Rate		SPI
ADXRS150	Rotational Velocity	0-5V
QT401,501	Proximity		SPI
AS040	Absolute position (rotary)		SSI
GP2D12	Proximity (IR)	0-3V
GP2D02	Proximity (IR)		bit-serial
VP300	X,Y+pressure	Resistive array

 

2.     Applications

2.1     Proximity Detector Array

This array, developed by Michael Zbyszynski, satisfies the need for a sensing device that can be mounted into scenery and that can detect a large range of performance gestures from dancers. The prototype shown in Figure 1 is a 4' x 4' wooden box containing sixteen infrared range sensors. The sensors sense distance in a range of approximately 8" to 60", and are mounted beneath the top of the box, under 2" acrylic windows. This creates a floor that is strong enough for two people to dance on, and has no rough edges. It can also be tipped on edge for a different sensing perspective. In performance, a similar array was built into a curved wall eight feet high. By comparing the inputs from multiple sensors, motion and position can be deduced, as well as judgments about the shape and posture of the body. 

Furthermore, the dancers are free to move in a relatively wide area. Unlike video tracking, the array is dependable -- insensitive to the changing conditions of theatrical lighting.

Future realizations will need a denser and larger array of sensors requiring 64-128 channels of data acquisition.

 

Figure 1. Dance sensor Array

2.2     Tiled X/Y + Pressure Pads

This hand controller, conceived by David Wessel, consists of a central array of 24 FSR-based trackpads surrounded by long position- and pressure-sensing strips and IR proximity detectors. Five measurands are available from each 2-d track pad and two are available from the position sensing. The trackpads are 4-pin devices with internal FSR components. They require custom circuitry to tease out the x,y position and width and pressure parameters by steering controlled currents around the array and measuring the voltage induced in the resistors. This may be contrasted with one dimensional position sensors that are usually wired simply as voltage dividers for interface to voltage input sensing systems. When more complex circuitry is used the width of the object being sensed may be estimated by the bulk reduction of resistance across the sensor due to the varying width of the conductive "wiper" contact.

The fully tiled array of 24 3-dimensional sensors (x,y and pressure) provides 72 variables of output. To achieve a degree of control intimacy similar in character to an acoustic hand drum, the array must be sampled at rates near 8,000 Hertz.  This variable count and sample rate far exceeds the capacity of gesture capture systems currently available such as the AtomMic Pro, EoBody, and Infusion Systems' Digitizer [4]. Consequently an expanded view of sensor integration is required.

2.3     Touch display

This display has wide application in musical controllers. We are exploring the integration of commercially available 3" and 4" diagonal touch screens into an augmented cello [9] and the aforementioned hand controller. They are addressed by serial protocols such as RS-232 or SPI.

2.4     Harp Controller

This controller developed by Adrian Freed consists of 36 identical length nylon harp strings in a rectangular frame that is sensed with individual piezoelectric pickups. The audio signal from each string is used for trigger and timbral information so an 8kHz sample rate is used rather than standard audio sample rate.

 

Figure 2. Harp Controller

2.5     Compact Position sensor

David Wessel's SmartSticks project uses MEMS accelerometer- and gyro-equipped drum sticks sampled at a 4k Hz rate.  The goal is to accurately sense the trajectories of each stick and its position relative to an array of percussion instruments. Sensors are mounted on both ends of each stick allowing the stick orientation to be sensed. Wireless transmission of the stick sensor data is beneficial, providing the percussionist with freedom of movement. 

Another important option in this project is eventual integration with a high-speed video capture system.

This array is also used extensively on the flute by Roberto Morales [7], for example in performances of his piece "Cenzontle" which one first prize at the 2005 Bourges competition. The array uses two rotational MEMS gyros and a two-axis MEMS gyro. A low frequency filter on the accelerometer is used for tilt estimation. A high frequency is used to capture key click and other transients. Two rotation rates are estimated corresponding to pitch and yaw of the flute which are free dimensions of control for the performer.

Figure 3. MEMS position sensor

To increase the mobility of the performer and provide flexibility for installation in other instruments (and bows) we have developed an RF interface to the sensors.

 

3.     Solutions

One approach to these projects is to build entirely independent custom electronics for each. They have widely varying numbers of analog and digital sensors and standards and bandwidth requirements. We have found it more efficient to use hybrid architecture with a common motherboard fulfilling the many common requirements and customized "daughter" boards for custom needs.

Our connectivity processor has a motherboard with 8-channels of balanced audio D/A conversion, 2-channel headphone output, AES-3 I/O, ADAT optical I/O, multi-channel sync. I/O, MIDI I/O, a high speed (GIG) and 100BaseT Ethernet [1].

Figure 4. Connectivity Processor motherboard

Figure 5. Gesture inputs and Digital I/O

This motherboard supports two daughter cards: one is used usually for audio input, e.g., 8-channels of balanced audio; the other is a gestural input card.

The basic gestural input card supports two DB25 connectors carrying power and 16 channels of 0-5V inputs on each with sample rates up to 8kHz. Although 32-channels 0-5V channels has proven extremely valuable for many projects our current projects require more inputs and support for more diverse digital standards for sensor integration.

4.     Beyond 0-5V

To address the need for broader range of sensors and sensor output formats we have developed a lightweight standard that covers the connector, power and configuration aspects of a sensor network. By hiding the complexity of the communications issues in affordable and flexible FPGA (Field Programmable Gate Arrays) we avoid the difficulties and cost of previous sensor network standards which have failed to take hold such as IEEE 1451 [5] or BISS [3].

4.1     Dynamic configuration

We use a configuration pin to dynamically assign functions to 6 digital I/O pins on each connector. This allows for a wide variety of clock rates, channel assignment and signaling protocols. Controllers, implemented with FPGA's can readily support the serial protocols chosen by the sensor manufactures including: I2C, SPI, 1-wire, RS232, RS422, MIDI, TTL, USB etc. In most cases modules in VHDL or Verilog are already available to support the protocol and usually for free [6]. Two configuration methods are supported: a single resistor to GND identifies one of 32 commonly used devices, alternatively a Maxim/Dallas 1-wire EEPROM device may be connected using the same pin. This allows for thousands of different device types to be supported. Since 1-wire devices have programmable storage this allows configuration and calibration data to be stored with the sensor array itself allowing the sensors to be moved transparently from one controller to the next.

4.2     Connector

A DB-9 was selected for the connector type because it is a widely available, reliable, lockable connector, has sufficient power handling for most sensor/actuator applications and is large enough that most existing and future sensors can be adapted to the FPGA signaling options with small circuits built-into the connector itself. Many digital sensors can be interfaced without a circuit board at all. Analog sensors requiring a few channels, e.g. MEMS accelerometers and gyros can be connected to a 4-channel A/D built into the shroud. Sensors that require more electronics can be integrated using larger shrouds around the DB9 pins including readily available DB9->DB9, and DB9->DB25 shrouds and small "project" boxes available with built-in DB9 connectors.

4.3     Power

5V @500mA is provided allowing the use of cheap power protection devices developed for. Most sensors require a small fraction of this power but sufficient power is provided for other devices such as actuators, relays and the occasionally inherently power-hungry sensing technology, e.g., IR and ultrasound rangers.

5.     New Controller

Figure 6 shows the first implementation of our new approach to sensor integration, a daughter board for the connectivity processor. It implements 4 flexible ports, 4 legacy analog input ports, a wireless system for remote sensing and a large Spartan III FPGA which has sufficient memory, gates and high speed multipliers to support the many protocols required and perform DSP and sensor calibration functions.

Figure 6: Flexible Sensor Integration Daughterboard

6.     FPGA vs Microcontroller vs DSP

Most computer audio interface boxes and gesture interface boxes use a mix of FPGAs, a general purpose microcontroller and more recently some include a DSP chip. At CNMAT we have deliberately avoided this approach preferring to maximize development efficiency and flexibility by using only one development environment - for FPGA's. We also use simple, well-documented standards wherever possible, i.e. Ethernet. Firewire and USB are notoriously complex to develop reliable drivers for.

The problems in our applications with microcontrollers stem from the fact that they are optimized to trade performance for cost-efficiency and this usually means too little parallelism. For example, they may be fast enough to support software implementations ("bit-banging") for a few SPI serial ports and some even include built-in hardware buffering for one or two SPI ports. This doesn't scale well when dozens of sensors are employed. Current FPGA's on the other hand have plenty of available pins and all the parallel bit-manipulation capabilities to support 40 or more SPI ports.

 

We have found our systems to be cheaper to develop, cheaper in overall parts cost and more reliable than the conventional mixed device approach. The reliability comes from lower overall complexity and how little outside intellectual property has to be integrated.

 

We also note that since the Xilinx FPGA's we use are large enough to contain processors and DSP we can integrate conventional programming approaches and custom signal processing if necessary.

7.     Software Support: /dev/OSC

Since the sensor output from the new controller varies according to the sensors plugged in, a flexible and extensible protocol is needed to communicate the data to application programs. The OS/X device driver for the connectivity processor extracts audio and gestural data from incoming Ethernet packets and routes it through to Core Audio and Core MIDI. Gestural data is up-sampled into audio streams for reliable real-time delivery for Core Audio. This technique does not scale well to dynamic situations possible with our new daughterboard. In this case we build OSC [8] formatted data which is queued by a special driver accessible as a UNIX file /dev/OSC. This technique avoids priority inversion problems we have observed in both the OS/X and linux operating systems when we route gestural data through the TCP/IP stack. Performance and reliability of this system were confirmed in a concert in November 2005 featuring an augmented cello played by Frances-Marie Uitti [9].

8.     Conclusions and Future Work

Our new daughterboard gestural integration card controller is a solid starting point for our ongoing work to integrate a diverse range of sensors. We are extending our early favorable results in two directions: a more compact version for wearable wireless applications and a larger system combining our controller with 48-channels of analog data acquisition for big hybrid sensor applications.

9.     Acknowledgements

We gratefully acknowledge David Wessel and the Chambers Fund for supporting development of the new system. Xilinx sponsors our use of their FPGA development tools.

10.     REFERENCE

 

1.    Avizienis, R. and Freed, A., OSC and Gesture features of CNMAT's Connectivity Processor. in Open Sound Control, (Berkeley, California, 2004).

2.    Avizienis, R., Freed, A., Suzuki, T. and Wessel, D., Scalable Connectivity Processor for Computer Music Performance Systems. in International Computer Music Conference, (Berlin, Germany, 2000), ICMA, 523-526, http://cnmat.CNMAT.Berkeley.EDU/ICMC2000/pdf/connectivity-processor.pdf.

3.    BISS:. BISS, 2006, http://www.biss-interface.com/.

4.    Fléty, E., AtoMIC Pro: a Multiple Sensor Acquisition Device. in International Conference on New Interfaces for Musical Expression (NIME-02), (Dublin, Ireland, 2002), 96-101.

5.    IEEE. Draft Standard for a Smart Transducer Interface for Sensors and Actuators, 2006, http://ieee1451.nist.gov.

6.    Lampret, D. Open Cores, 2006, http://www.opencores.org.

7.    Morales, R.a.M., Eduardo, Combining Audio and Gestures for a Real-time Improviser. in International Computer Music Conference, (Barcelona, 2005), ICMA.

8.    Wright, M. Open Sound Control as an Enabling Technology for Networked Music. Organised Sound, 10 (3). 193-200.

9.    Freed, A, Uitti, F, Wessel, D., Zbyszynski, M, Augmenting the Cello, NIME 2006, Paris, France,