Sonic Abstractions is a cross between an interactive installation and a thought experiment, whose aesthetic and conceptual properties are still in development.
About
The theoretical framework for this project began with a relatively simple realization: that sounds and shapes are simply structures of data and that there’s nothing stopping us from converting between them. Sounds, as the field of acoustics will tell you, are the result of air pressure changing over time, while shapes can be represented by 2- or 3-dimensional arrays of points. What would happen if you extracted a series of 3D points from an object and arranged it in a format that can be interpreted as sound data? What would it sound like? If the object were to melt or otherwise change shape over time, how would the sound reflect that?
Hardware Considerations
The primary challenge to overcome at first was how to assemble the 3D data structure from a physical object, whose surfaces are not entirely visible from every angle, as would be the case with a 2-dimensional or flat object. Existing hardware solutions for this are rather expensive and use proprietary software. Cheaper solutions, such as tools that allow you to assemble a 3D mesh from a series of 2-dimensional pictures of the object from a different angle, create a very fragmented and asynchronous workflow.
My preliminary solution, as seen in the video above, uses a custom-built turntable and a RealSense 3D camera to scan objects from 360 degrees. It’s fairly accurate and offers a great compromise between functionality and affordability.
Preliminary Sonification
With the means of scanning objects into point clouds created, I set out to create a preliminary protocol for encoding the shape information of those meshes into sounds. My solution for sonifying 3D meshes was to treat each “slice” of the mesh as a separate 2D shape, essentially grouping the massive collection of points according to their y-value. I’d create a separate sound wave out of each slice, and then cycle back and forth through the sound waves to (hopefully) hear something cool. For each point in each 2D slice, I’d calculate its angle around the center of the slice and its distance from the center. The angle, in degrees, would be used as the sample number of the point within the sound wave, while the distance would be used as the amplitude.
The result, seen in the video above, is interesting but less than satisfying. For the first object, a small rectangular box, the sound doesn’t change over time since all the slices of the box are the same. For the tape and the roll of toilet paper, however, it does. My primary issues with the output are twofold. Firstly, the sound isn’t very interesting on its own; it’s mostly just variations on a buzzing sound. Secondly: the sound doesn’t convey any discernable information about the shape. I don’t just want to encode shape information into sounds, I want the sounds to somehow convey parts of that information to the listener, without requiring the use of an oscilloscope.
Iteration #2
My second solution set out to produce sounds based on more observational, rather than mathematical, data. I’m still treating the object as a stack of slices. This time, however, I’m creating a visual black-and-white image of each slice, where the white points are coordinates where a part of the object was detected. I’m then using OpenCV to perform blob detection on each slice, which gives me the central position of each blob, as well as a list of points relative to the center. I can then perform a similar operation to the points as I did to the slices in my previous solution, using their angle and distance relative to the center of the blob to create a sound wave for each blob. I’m using the angle of the blob around the center to calculate the pitch, and the relative size of the blob for the volume.
I find this solution significantly more satisfying. The presence of multiple blobs per slice introduces a harmonic aspect to the output, which makes it more interesting to listen to, and also seems to communicate something about the object to the listener. I’m particularly fond of the very end of the video, where 3 small knobs on the top of the object are sonified, creating a soft ending chord.