Sound and Vision: Digital technology in performance

A cartoon girl – a stick figure in an A-line dress – walks to a piano, pulls herself up on to the stool and begins playing the prelude from The Nutcracker. As snowflakes fall, the child transforms into the very real shape of Alexandra Dariescu (Piano 2011, Creative Entrepreneur 2017), and a ballerina begins to dance Tchaikovsky’s Christmas fantasy. All the characters she encounters, from the Prince to the Sugar Plum Fairy, exist only as digital images. It’s a fact soon forgotten as the magic takes hold.

Mixing live action and animation has long been a staple of the movies, all the way back to Max Fleischer’s silent cartoon shorts of the 1920s. But this is a live performance, with Dariescu and the dancer in perfect synchronisation with the projected video produced by London design studio Yeast Culture. It’s a feat that would have been beyond the technology that existed just 15 years ago.

Digital advances are shifting the frontiers of creativity, opening up new ways for artists to combine sound and vision in live performance. Many Guildhall alumni are taking advantage of these new possibilities – and according to Dan Shorten, the video lecturer who wrote the programme for Guildhall’s BA in Video Design for Live Performance, they’re doing so in increasingly sophisticated ways.

He says: “Maybe 10 or 15 years ago, a lot of digital work was really about the medium itself; the artistic aspect of it wasn’t quite as evolved. People develop the technology and want to show off what they can do with their new application or tool. It’s only when it starts to bed in that the art becomes more interesting.”

A good example is provided by one of Shorten’s areas of expertise, projection mapping – a technology that can turn irregular shapes, such as buildings, into a canvas for moving images. “There was a time when it was mostly about the fact that you could do it,” he says. “Then artists saw it, realised what was possible, and worked out what they wanted to do with it in a creative sense. And so the aesthetics of video mapping have evolved.”

Take Waddesdon Imaginarium at Waddesdon Manor, near Aylesbury. Using state-of-the-art video projection, Guildhall’s students brought the country house’s history to life. Their nocturnal installation turns the facade into a fantasia of roaming porcelain animals, dancing musical clocks and fluttering butterflies. The show was created by seven Video Design undergraduates, with a soundtrack produced by students on courses including the BMus in Electronic Music. More than 100,000 visitors witnessed the spectacle over Christmas 2017.

Above all, Shorten believes that technology is creating a new language through which people – particularly young people – can express themselves. “Art should be a reflection of the society it’s trying to engage,” he says. “Take the average 12-year-old. Some may go to dance or painting classes, but they’re a minority. However, every 12-year-old sits in front of a computer every day, so it’s a language in which they’re fluent.”

Engaging young people was Alexandra Dariescu’s prime motivation in creating The Nutcracker and I, which premiered at the Guildhall School’s Milton Court Concert Hall in December 2017. She says: “I wanted to challenge the concert format, because bringing the younger generation into piano recitals is the hardest thing ever. Having performed all around the UK and Europe, I could see that the average age in the audience was definitely not getting lower. I felt a responsibility as a young artist to do something that engaged with children and teenagers, as well as their parents and grandparents.”

The technical challenges were considerable. “Both the ballerina and I are very spontaneous,” she says. “So we have a lot of cue points for the visuals, and the live projection mapper has to be on the ball to keep everything synchronised. But that’s the beauty of live performance – it’s never the same twice.”

The Nutcracker and I condenses Tchaikovsky’s three-hour opera into a family-friendly 50-minute programme with no interval. But the animation technique chosen by Dariescu meant that only six seconds of animation could be produced each day. “We used rotoscoping, so everything was hand-drawn. We thought we’d be able to work with five animators. However, at 15 frames a second we needed 35,000 drawings in all, so in the end we had 32 animators working all over the world. I’ve not met them all in person.”

It’s a reminder that however clever the technology, realising an artistic vison can still demand a great deal of groundwork. This has certainly been the case for composer Oliver Leith (Composition 2014), whose recent multimedia works have been inspired by digital culture. His composition, 664 Love Songs Guaranteed to Cure Heartache, analysed the most popular words from all 664 number-one love songs between 1952 and 1990. In performance, these lyrics are flashed up on a screen with images synchronised to his music.

He says: “That piece was inspired by what you can do with a collection of digital data. It was only possible through exploiting the new means that we have now. But I still had to painstakingly copy all the lyrics, and feed this humongous file into word-analysis tools. I’d hoped to find that the syntax and vocabulary had changed over the years, but they hadn’t all that much. The same words came up, and even the same names, such as Mary and Joseph. I don’t think it’s a Biblical thing; I think they’re just nice words to sing!”

For Leith, a great advantage of the latest technology is that he can produce the visuals as well as the music, without the need to engage a videographer. “I have all the equipment of a recording studio in the space of one laptop, and I also have access to video software, so I can put that side together myself. Because it’s so important that the material in the video is synchronised with the music, I like to do it myself.” 

By contrast, collaboration is vital to the work of Joby Burgess (Percussion 2000), who founded the experimental trio, Powerplant, with visual artist Kathy Hinde and sound designer Matthew Fairclough. As well as playing traditional acoustic instruments in performance, he uses digital percussion controllers such as the Wernick Xylosynth. This is not only able to trigger any sound source via MIDI – granting him access to an unlimited palette of sounds – but allows him to control the visuals in real time during concerts.

He says: “Kathy, our video artist, is also a composer in her own right, and we’ve been trying to get a really organic connection between the video element of the performance and the sonics. Every piece works in a different way. Sometimes she’ll work almost as a video painter, reacting live to what’s happening in the auditorium. With other works, when I play a note on the Xylosynth, it’ll trigger some change in the visuals.

“For example, we did some arrangements of the American composer Conlon Nancarrow, using some of his early work from the 1950s. We use a bank of early images of a horse jumping by [Eadweard] Muybridge, and every time I strike a block, it advances the frame.”

Those images, photographed in 1878, are thought to have been used in the first-ever projection of a moving picture. Today, Dan Shorten believes that similarly profound advances in audiovisual experience may be around the corner. He says: “The concept of immersion is going to grow: lots of artists are talking about providing an immersive experience. Technologies like virtual reality are becoming extremely powerful but, for the moment, they’re solo or small-group activities. It’s not the same as going to the opera with hundreds of other people. So how do we take advantage of their aesthetics without eradicating the benefits of a shared experience?

“Sometimes, however wonderful a piece of art, the audience isn’t ready for it. There was a time not too long ago when using projection in a theatre was seen as disrupting the aesthetics. Now it’s far more embraced. These things take time. As the way we interact with technology in our own lives evolves, so will what we demand of artworks and how we are prepared to engage with them.”

This article first featured in the Spring/Summer 2018 edition of the Guildhall magazine, PLAY, and was written by YBM for the Guildhall School of Music & Drama.