Baidu, the Chinese internet giant, has created a new AI program to explore the connection between art and music. The Baidu AI Composer creates original music inspired by different pieces of art, evoking the mood of each picture in a musical representation.
According to the promotional video released by Baidu, the Baidu AI Composer uses image recognition, connected to the ‘world’s largest neural network’, to identify the subject, mood, and even cultural signifiers of a piece of art. These are filtered through a matrix of hundreds of billions of samples and AI training features, using trillions of parameters, to create a complete and original piece of music inspired by the specific piece of art observed.
The AI program first identifies elements of the picture – are people represented, or is the focus on nature, or objects, or is it an abstract piece? The AI is trained to extract attributes from labeled image data, assigning a mood to elements of the picture, for example, deciding if the overall tone is warm, upbeat, or melancholy. The programmers at Baidu gave the AI access to an enormous library of music.
The program breaks down a composition into separate musical units, and marks the ‘crucial label’ between lyrics and music. After identifying the key elements and tone of the art piece under inspection, Baidu’s AI Composer creates a correlation matrix between the crucial labels and musical units. It feeds the labeled image data through the correlation matrix and uses it to create original music based on the picture.
One of the first original compositions created by the AI Composer was based on Van Gogh’s Starry Night, included in the video above. Created when the AI Composer was still in an experimental phase, the tone of the music is soft and slow, with the use of minor keys to represent melancholy, or thoughtfulness. Another, based on a Chinese watercolor of galloping horses, is faster and more urgent, with a steadier beat that emulates the motion of the art.
The Baidu AI Composer can be viewed in a new exhibit at the Ullens Center for Contemporary Art in Beijing through August 2016.