very rough notes!

Audible pictures, visual sounds

From sound to light and back

It's ART! Or, a medium with the potential for art.

I think it would be cool to have a way to print sounds onto paper, and then read them back again and hear them. You could do all sorts of neat things, like a exhibition of visual art, where each picture contains an additional level of artistry via a sound. Printed media like magazines, instead of having to include a CD or other physical audio medium like flexidisk, could simply embed the sound into a printed page.

Some rough figures

640 * 480 = 300000 * 3 = 900000 (1 MB = 1000K)

Telephone 8K/sec
CD 44K/sec

If low-quality audio is 10K/sec, then an image gives 90 seconds. But, this is an idealized case not achievable in reality. Things like color balance, light balance etc. introduce many significant bits of error into each R, G, and B. Also, a fully literal representation would probably be uninteresting visually; reducing the potential for aesthetic impact.

To convert from RGB to HSB may be a good idea, since it could smooth out the perceived colors, making brightness shifts into a single noise value, instead of all three RGB shifting. It could also help the human eye to see relationships between the sound and image, since HSB is more like the human visual conceptual model.

Printers operate on a CMYK color model, dealing with reflected rather than emitted light. This means there are colors possible with RGB that are not possible in CMYK, and these colors will not print accurately. We should avoid these colors, which limits the amount of information the medium can carry through the whole process. (To do: What colors are they? What is the RGB -> CMYK algorithm?)

A raw audio wave is single-valued, an amplitude wave. One very simple approach would be to use the raw HSB values, one of them or a combination, directly as amplitude. This algorithm (basically a codec) seems like a good candidate to implement first, to test the software framework.

There are really an infinite number of codecs possible. A modular software architecture would allow any programmer to write their own and plug it in. People could attempt to create codecs which give more aesthetically pretty results, while maintaining the fidelity of the audio encoding.

P0002184.jpg (133845 bytes)Ideas

These ideas first arose during conversation with sound artist Maryann Amacher in San Francisco in May 2001.