AI delivers creative rendition of classic HK song

Featuring the lyrics "the meandering stream flows southward, running to Hong Kong to have a look," the song Pearl of the East has been sung across generations for over three decades, bringing a sense of belonging to many Hong Kong people.

At a recent concert in the city, the song was once again played to celebrate the 25th anniversary of Hong Kong's return to the motherland. The performance was exceptional this time: it was sung by a choir of artificial intelligence musicians.

Accompanied by an orchestra from the Hong Kong Baptist University, the AI choir, consisting of 320 virtual singers, followed the gestures of the conductor to present the classic tune against the backdrop of an AI-generated 3D animation.

"This could be the world's very first human-machine collaborative performance," says Guo Yike, vice-president of HKBU and an AI expert.

The AI-powered performance was part of a research project led by Guo. The project aims to explore new opportunities that AI-based art technology could bring to Hong Kong and has landed government funding of over 52 million Hong Kong dollars (about $6.6 million) for a period of five years.

For the AI singers, it was quite a learning process. At the very beginning, the virtual musicians had trouble singing high notes, even occasionally going out of tune.

"Sound is about frequency. It is not difficult to teach them to sing as long as the frequency is adjusted constantly," Guo says.

The bigger challenge was ensuring that the AI singers could understand the gestures of the conductor. The communication was made possible by sensors tied to the waist, shoulders and wrists of the conductor, who repeatedly practiced with the AI singers to deliver a better performance.

After almost six months of training, the AI singers were able to understand the language of the conductor well enough to present the music at a perfect tempo.

But what really amazed Guo and his team was the ability of AI to interpret the lyrics and create the accompanying 3D visual narrative. A machine needed to first understand the meaning of the song before generating nearly 3,000 images and connecting them with natural transitions that matched the song.

For instance, when the lyrics say "flowing to fragrant river to have a look," the machine needs to understand that "fragrant river" stands for Hong Kong, and then create relevant images.

What the technology achieved was beyond Guo and his team's imagination. When presenting the lyrics, AI first drew a pearl, which then blossomed in layers to become the petals of the bauhinia, an emblem of Hong Kong.

"We were all stunned at the moment. The concept of 'fragrant river' was presented in such an unexpected way. This was the creativity of the machine itself," Guo says.

For Guo, what AI can create goes far beyond mere imitation of human behavior. With the help of input from human beings, machines can create art that is distinct from human creation, he says.

"Perception is very important to scientific breakthroughs. We changed our perception, and here comes the breakthrough," he says.

Guo says his team will continue the study to empower art with technology. In the future, they hope to present new AI-powered art forms, such as opera.

"There is no end to exploration, and nothing is impossible," he says.