In fact, even before this, our music industry has utilized systems and computing power for composition and music producing across all fields of the music…
The New Industry Where Music AI and the Metaverse Meet: Web 3.0
Rio Jang │ CEO of Emotionwave
The match between AlphaGo, an artificial intelligence, and Lee Sedol on March 9, 2016, was a turning point that announced the arrival of the age of artificial intelligence to the world. Since then, it seems like all industries are living through an era of great change and upheaval due to artificial intelligence in 2022. Artificial intelligence is playing a role as a new industrial paradigm across almost all fields, including medicine, chemistry, agriculture and livestock, and content. So, what kind of impact is it having on the music field?
It also seems that the music field is at a turning point of a new paradigm through combination with artificial intelligence. Music AI is deeply rooted in our lives across almost all fields, including composition, singing, arrangement, performance, education, and appreciation.
**The Beginning of AI Composition and Neural Networks**
In fact, even before this, our music industry has utilized systems and computing power for composition and music producing across all fields of the music industry, such as electronic music, computer music, and algorithmic composition, to an extent that one could consider the algorithm composition methods presented by algorithmic composition researcher Eduardo Reck Miranda as a starting point. This can be seen as having contributed to lowering the hurdle of music composition even for people who have not received professional training or education.
Step Sequencer or Arpeggiator can also be considered good tools and systems that allow for the creation of structured music even with simple input. Furthermore, Sound Synthesis Technology and DSP (Digital Signal Processing) technology belong to complex technologies that digitally reproduce, store, and transform sounds found in natural phenomena through complex mathematical formulas and algorithms. Since system-based automatic composition or sound synthesis technology is also generated data originating from systems centered on specific algorithms and logic, it can be seen that composition and music producing were achieved with the help of artificial intelligence and systems.
However, when referring to the latest artificial intelligence, it primarily refers to AI composition and AI music that have been newly generated data and content through neural network-based learning based on quantitative and structured datasets, which is applied in Generative AI.
Representative music AI services and technologies such as AIVA and AWS Deep Composer can also be seen as examples based on this artificial intelligence technology, and active research is being conducted using AI models such as CNN, RNN, and GAN.
**Challenges for the Popularization of AI Music**
However, services like AIVA or AWS DeepComposer provide an interface that requires prior knowledge at a level suitable for music experts or computer science experts, making it difficult for the general public to use. The final music data provided by AIVA is also in MIDI format, so in order to make the finally provided music up to the standard of commonly distributed music, a series of subsequent processes such as arrangement, mixing, and mastering are necessary. A new AI composition platform that complements this has emerged, with Japan's Soundraw being a representative example. Soundraw provides trendy music based on genre and mood selection. It is notable that it provides loop-based music based on various genres and customers' emotional use cases. In my analysis, it seems to be pre-producing loops mapped to various virtual instruments based on chord patterns generated by AI, and then recommending pattern music most suitable for the customer's main query through an AI recommendation engine. Soundraw appears to have accessed a more practical and popular market by providing stock-style music in various genres, which is the core of current media consumption trends.
**AI Music and Performance**
The beginning of AI performance, starting from the keyword 'automatic performance,' can be traced back to the early automatic performance instrument called the Orchestrion, which was based on punch cards using actuators and robotics technology. In the past, an instrument was invented that allowed a machine to play various music by utilizing mechanical devices and punch cards instead of sheet music, which became the origin of today's AI instruments. However, this could be seen as the repetitive movement of a programmed machine rather than performance utilizing intelligence. There were many limitations in terms of scalability, interconnectivity, controllability, and creativity, and since manufacturing was limited by production techniques that relied heavily on manual labor, there were also limitations in mass production. However, with the development of electronic music, the landscape changed revolutionarily when MIDI, a digital instrument standard interface, was released in 1983. Most existing electronic instruments and automatic performance instruments connect and communicate based on MIDI. The neural network-based artificial intelligence mentioned earlier also proceeds with new MIDI-based creation by learning from strategically prepared music information data in the form of MIDI files. The intelligent connection relationship between existing AI music and instruments is established in this way.
**AI Music and AIoT**
This creates a need for connecting and owning instruments applied with artificial intelligence music. We have given sound systems, called instruments, the ability to play by themselves and endowed them with limited intelligence so they can communicate and perform various pieces on their own according to communication standards. Now it is about connection and conversation with infinite intelligence connected online. Conversing with a virtual AI music intelligence that can converse presents a new attraction to us who own rigid instruments. We are now able to sing, create, arrange, and sing through conversation and simple mobile phone operation via connections with new personalized music intelligences connected to the internet. This is made possible by various interactivity designs based on online platforms. Through artificial intelligence instruments and AI music, we can enjoy music and performances realized with the sound of actual instruments, not speakers. This allows acoustic music to be utilized in effective education, broadcasting, studios, entertainment spaces, etc., making our lives and industries more convenient and abundant.
Metaverse and Web 3.0 Virtual AI Music
To date, real-life music activities and information have been digitized and recreated and created through artificial intelligence. Now, it is the digitalization of real activities through virtualization and digital twins. The virtualization and visualization of music activities are already being digitized and visualized through games, computer music sequencers, computer music score creation, and music videos, providing various experiences to music lovers. Furthermore, recently, music IPs have already been virtualized and assetized in various virtual spaces representing metaverse platforms such as Roblox, Zepeto, and if Land. This is also owned by various customers, distributed in the form of tickets or digital goods linked with NFTs. This is also a flagship event with a large transitional nature as the industry shifts to Web3.0. It is expected that there will be various music experiences in metaverse spaces based on IPs. However, for it to become a larger platform, everyone must be able to perform, create, educate, and interact easily in the metaverse space. Artificial intelligence music technology is expected to be the key core technology that helps overcome these limitations of such platforms. Even if you only play the piano at home, it becomes a digital twin in the metaverse space where you can perform and easily give a concert. The artificial intelligence technology that makes this easy to reproduce is not far from a future where my singing voice can be corrected through AI technology, even if perfect singing is not possible, or where I can release an album or perform in virtual space via NFTs.
Rio Jang
Emotionwave's CEO Rio Jang launched the AI concert system Rima Public in 2020 and oversaw the Intelligent IoT Application Diffusion Project from the Ministry of Science and ICT. Recently, they launched Aimple Global, a digital human NFT platform, and MEW, a metaverse global KPOP education platform.
