Artificial intelligence can help you understand music better

Algorithms and technology have so far helped listeners to more of the same music. Now, UiO researchers are working on new technology that can get people interested in a greater musical variety.

Chords, beat, timbre, rhythm and harmony. All these elements of music contribute to make it sound the way it does. But have you thought about why you like particular kinds of music?

“Music is a magical thing, when you think about it. When you listen, you feel many emotions. You understand that it is a kind of language, but you cannot see what is happening. For most people, it is a mystery.”

So says Olivier Lartillot, researcher at RITMO Center for Interdisciplinary Studies in Rhythm, Time and Motion. He is developing new digital tools that he hopes will make the magic of music available to everyone.

First on his list is Norwegian folk music.

“Folk music is so rich, and a treasure for Norwegian culture. Still, not many people listen to it. If we create a tool that can help people understand music, folk music may have a renaissance in Norway,” says Lartillot.

Better understanding will open a world of music

Olivier Lartillot believes technology can expand people’s taste in music through a better understanding of musical elements.

“Often, people are interested in the music that they listen to all the time, and that they understand easily. Some types of music seem more complicated, and are therefore less accessible,” he says.

“If we can give people tools to understand music better, we will also provide access to a lot of new music. It will benefit not only the individual, but the music itself—and the diversity of the entire ecosystem of music.”

Musicologists also lack complete knowledge about the music they study. Lartillot, who has combined advanced computational music analysis with insights from musicology for many years now, can bring them closer to the answers they are looking for.

“We are now approaching a stage in the research where we can make tools that can understand the logic of the music,” says Lartillot.

Training artificial intelligence with a fiddle

Currently, the best tool for music analysis with computers is artificial intelligence and so-called machine learning.

“You train the machine by teaching it that this is a certain type of music and these are the notes it should recognize. By ‘hearing’ examples, the machine tries to understand what is happening. After working through enough examples, it can detect notes automatically.”

The music the machine is intended to transcribe is the National Library of Norway’s catalog of folk music. According to Lartillot, Norwegian folk music and especially the Hardanger fiddle is difficult material for the machine.

“The large amount of examples needed was initially not available. Therefore, we asked musicians, the professional fiddler Olav Luksengård Mjelva, and students from the Norwegian Academy of Music, to play for us, and designed a software where the sounds were visualized and they could place the notes for us.”

Lartillot’s colleague, postdoctoral fellow Anders Elowsson, is currently using the manual annotations to teach the machine how to automatically detect the notes played by the Hardanger fiddle.

“This is work that can take hundreds of hours, on large powerful computers,” says Lartillot.

The next step, which is where they are now, is detecting the beats. This is complex in fiddle music. But when the machine masters that too, an interactive tool in the form of, for example, an app is within reach.

Visualization helps music understanding

With the app Lartillot wants to create, you will be able to browse the folk music catalog and go on a journey of discovery. When you find a tune you like, the app can point you in the direction of something in a similar style.

One of the ways you can interact with the music is by watching it.

“When you watch a video clip it plays on more of your senses which in turn makes you feel more. I think this can help many people to understand folk music better,” says Lartillot.

The work in not done yet. Lartillot is trying to find the fine balance between adding enough detail to make you understand more than you do by just listening, and too much detail—which may turn your attention away from the music.

“It can even be a kind of gamification: an interactive app where you start with a simple visualization. When you master and understand what is going on in the music, you gradually get a more complex version.”

The same technology that will become an app on your smartphone can also be used on stage.

Lartillot is currently preparing a visualization for a concert where The Danish String Quartet will play Bach’s fugues. As the four musicians play through the themes, the audience will be able to see how the different voices repeat each theme on a large screen.

“My hypothesis is that especially untrained listeners will understand more if they can also see what is happening musically. The themes will appear one after the other on the screen, so when the musicians start playing a new repetition of it, you will see the patterns.”

Remix of old folk tunes

Another way the app he develops will be able to provide more insight into folk music, is that you can play with it and make your own remixes.

“For me it was an epiphany. When I could put beats in top of the music, it opened up to me and I saw the logic. Suddenly Norwegian folk music became much more interesting to me,” says Lartillot.

Such an app will be available to everyone free of charge. A professional DJ or composer, but also a regular listener, can find a recording from, say the 1940s, and add electronic beats.

“Because the app detects the notes, you can transform and change the sound, for example with electronic beats or electric guitars. It will be possible to make music that sounds completely new, even if it is the same structure.”

In the short term, the new app can only be used with specific folk music. In the longer run, Lartillot wants to expand the knowledge of the artificial intelligence, in order for it to be used for all kinds of music.

The researcher also sees a number of other uses for the app.

“Because it is about how we perceive details in music, it can be used in music therapy and in research on musical cognition. I also hope that the music industry will discover this technology and integrate it into its streaming services.”

MIRAGE—A Comprehensive AI-Based System for Advanced Music Analysis is a research project at RITMO Center for Interdisciplinary Studies in Rhythm, Time and Motion. The goal is to improve computers’ capability to listen to and understand music, and to use that to conceive technologies to facilitate music understanding and appreciation.