Discover HKBU

Tracing the boundaries between humans and machines

30 Jul 2021

At a time when most, if not all, aspects of our lives are driven by algorithms and other digital technologies, generative art – work created by automated systems – may be the type of art that best represents our current hyperconnected world.

Over the past few years, machine learning algorithms have advanced to the point that artificial neural networks are capable of producing new and realistic images. While artificial intelligence (AI) has been successfully applied in the visual arts, the training of algorithms to generate sounds and music in the waveform domain has yet to be fully explored.

To find out more about this area of AI, an interdisciplinary research team from HKBU's Meta Creativity Lab, which is part of the Augmented Creativity Lab, has been using Generative Adversarial Networks (GANs) to further investigate the application of machine learning to different aspects of creative practice. Dr Roberto Alonso Trillo, one of the four principal investigators of the project, says a challenging but important goal, beyond the technical complexities of building and training the GANs, is to understand how these new technologies intersect with questions about artistic agency.

Exploring machine creativity

"This project involves the use of algorithmic architectures (GANs) to generate music," says Dr Alonso, Assistant Professor of the Department of Music. "One of most interesting conceptual dimensions of GANs is their ability to learn, and to address supervised and unsupervised learning problems."

GANs comprise two competing sub-models, respectively called the generator and the discriminator, which are trained on an input dataset. The generator tries to replicate the given data, while the discriminator attempts to determine whether its input is original (from the original database) or artificially generated (by the generator). This process can be likened to a competition between an art forger and an art appraiser. The forger creates a piece for the appraiser to determine its authenticity, and based on the results of the examination, he can create a more genuine-looking piece. The appraiser also enhances his level of expertise throughout the process. The content is then bounced back and forth between the two networks until the discriminator can no longer differentiate between the original data and the generated content. Through testing and learning, this process aims to create more morphologically realistic content.

Dr Alonso and his collaborators are using two GANs (MelGAN and UnaGAN) to generate sound outputs. "We feed the algorithms a large database of sounds, and we then train them and experiment with different modifications to the networks’ inner structures. At this stage, we are delving into the area of sound morphology: if we feed the sound of a violin to the GAN, can the GAN generate sounds that sound like a real violin?" he says, adding that both GANs employ unsupervised learning, that is, no conditions are imposed on the learning process.

Based on the results from the two GANs, the researchers are using a third neural network, a custom-designed descriptor/predictor model, to generate and predict sound sequences that the team can manipulate to influence the output. At this stage of the research, the team hopes to understand and diffuse the boundary between humans and machines in a live performance context. Dr Alonso says, "A lot of research projects concentrate on the technical aspects of machine learning in the arts, but we're trying to explore its artistic applications."

When art and technology intertwine

Translating the AI-based creative outputs into real performances is a complicated part of the project, says Dr Alonso, as it is essential to integrate the technology into the artistic narrative, and not use it as a gimmick. "Computer science has been evolving and new research into digital technologies has the potential to be applied in the arts. As artists, we need to do something meaningful with these new ideas and tools," he says.

The use of AI in the arts also raises questions about the nature of agency in an artistic context. To Dr Alonso, the development of new technologies has always dramatically changed human creative and artistic processes throughout history, and machine learning, as well as generative art, represent a similar paradigm shift.

While he sees machine learning as a set of computerised tools employed by the artist, he finds it difficult to trace the boundary between humans and machines in his project. "When you think about the tools we use in the arts, for example a musical instrument, the artist should experience a connection with the instrument when he plays. If this connection is well developed, the boundary between the artist and the tool eventually disappears," he says.

Pushing musical boundaries

Dr Alonso is no stranger to contemporary music and experimental performances. A Spanish native, he is a classically trained violinist with a Doctor of Philosophy in Musicology and Violin Performance from the Royal Birmingham Conservatoire, which is part of Birmingham City University in the UK, and he also has a Doctor of Musical Arts in Violin Performance from the Moores School of Music at the University of Houston in the US. He has performed both as a soloist and as a member of various chamber and new musical ensembles around the world. "For the past 20 years, I have worked with numerous composers and artists on cross-disciplinary projects. Some of them used AI-based technology, which ignited my interest in the application of new technological tools in artistic practice," he says.

He appreciates the opportunities at HKBU to conduct interdisciplinary research in music, particularly in areas related to AI creativity, while also fulfilling his roles as an educator and performer. As a professional violinist and musicologist, he is able to bridge the gap between practice-based research, i.e. gaining new knowledge through practice, and practice-led research, i.e. advancing knowledge about practice, in the creative arts. "I think it is interesting to be able to think about music and perform it from a musicological perspective. You can intellectualise but you can also inhabit," he says. "You're not talking about someone else's performance, you’re performing it."

The introduction of various restrictions and quarantine measures due to the ongoing pandemic has also made live performances more complicated, but for Dr Alonso, the artistic experience is well worth the inconvenience. He recently performed at the Resis Festival, an international contemporary music festival that took place in Spain this June. He says, "When you're performing, the transcendent moments when you connect with the other musicians and the audience are otherworldly. That's what keeps me playing."

Please click here to see the GitHub repository of the research project.