When So-Hee Woo constructs sets for artists like The Kid Laroi or Trippie Redd, her tools are not hammer and saw, but 3D computer graphics programs and game engines for building virtual worlds.
Ms Woo, head of XR design at Encore, makes computer-generated settings for augmented reality (AR) performances, which fans watch within the Encore app, paying 10 US cents for each "clap".
If that sounds like a lot to digest, relax — even the people working at the interface of music and online virtual spaces, or what's loosely called the metaverse, find the pace of change dizzying.
"All the artists at first are definitely unsure of what it is," Ms Woo says.
In-app AR performances are just one of many approaches to the idea of bringing music to the metaverse, combining the thrill of crowded live events with the massive potential audiences of online platforms.
The forced isolation of the pandemic saw a flurry of attempts to bring gigs to people's living rooms, from YouTube sets to entire virtual festivals complete with jiving avatars and sound-and-light shows.
Often, these attempts didn't quite hit the mark. It turned out the multi-sensory buzz of a packed venue was hard to replicate.
But there was enough success to prove the potential of virtual concerts.
Now, as the necessary technology continues to improve, millions are being invested in wildly divergent formats for live music in the metaverse, in a race to find the one that's most popular.
These experiments could change how you experience music — but probably not in the way that you expect.
Wait … what's the metaverse again?
Essentially, the metaverse is a collection of linked spaces — some entirely virtual, some physical but with virtual elements – in which you or your avatar are able to have any number of immersive, 3D experiences in real time.
The idea is that one day, there would be some level of "interoperability". That is, those spaces could talk to one another.
If your avatar bought a t-shirt, or styled their hair in a certain way, those characteristics could be carried from one virtual world (for example, one owned by Meta) to another (one owned by a game developer like Epic). So too would any data you generate, like preferences.
As you may have guessed, these immersive virtual worlds don't yet exist — despite all the talk, and Facebook's decision last year to rebrand as Meta, a metaverse company.
But the metaverse isn't only big-haired avatars in tight-fitting t-shirts. It can also be a more subtle blending of virtual and physical worlds.
That's where augmented reality (AR) comes in. For example, have you ever used a homewares app to test how a table or couch might look like in your home? Or pointed your phone at the sky to discover different constellations? That's AR in action.
"You probably already know a lot of different apps and different experiences you can have that are pointing towards the metaverse," Sofie Hvitved, head of media at the Copenhagen Institute for Future Studies, says.
"This overlaying, this interaction between our virtual and physical worlds, is the most interesting [thing] regarding the metaverse.”
Where music and the metaverse meet
You may be wondering, when do I get to attend a concert in New York's Madison Square Garden (or anywhere) from my living room in Australia?
And if I can't do that, what's the point, anyway?
Re-creating anything like the feeling of attending a live event would be difficult. At the very least, it would require many technological breakthroughs, including photo-realistic 3D imaging and accurate hand-and-eye tracking.
On top of that, "all senses play an important role," says Patricia Haueiss, a Sydney-based consultant in the metaverse and blockchain space. That includes touch, smell, and even taste.
"The multi-sensory integration of spatial audio, scent and connection of haptic suits with experiences would bring it to the next level."
Instead of these fully immersive experiences, we currently have a spectrum of different approaches.
At one end are those that essentially try to recreate a physical performance but within an entirely virtual space, using the existing (limited) technology. Think crowds of milling, bobbing avatars.
And at the other are those that provide an overlay of online interaction over physical performances.
Last year, for instance, Justin Bieber donned a motion-capture suit that translated his movement onto a digital avatar and danced to a soundtrack of a few of his hits, at a concert attended in the metaverse.
And just last week, English hyperpop queen Charli XCX performed her songs in a commercially branded "superstar galaxy", running on the gaming platform Roblox.
This and other metaverse concerts have drawn millions of viewers, but they come with a catch: expensive production, and an audience that skews towards young gamers who are on the platforms already.
As a result, metaverse applications have recently launched that are more accessible and require less costly production from artists.
'Democratising' metaverse show production
The performance app company Encore, which launched earlier this year, is trying to "democratise" metaverse shows, co-founder and chief executive officer Jonathan Gray says.
"Only the top 0.00001 per cent of artists today are getting what you would call a metaverse show."
The Kid Laroi's show in May, for instance, had no fancy avatars and relatively simple 3D graphics that Ms Woo knocked together in a few hours.
The Australian rapper sang and danced against a green screen so that it looked like he was inside a '90s computer game.
Encore splits profits with the artists, who are "usually making" at least $US2,000 per show from those 10-cent claps, says Mr Gray. Crowd sizes have varied from less than a hundred to 14,000.
This week, the company released a new app for artists to make their own AR environments, including 3D objects and visual effects, so they don't have to use designers like Ms Woo.
"And that's getting you closer to what can be done for six and seven figures by these huge global recording artists," Mr Gray says.
Entertainment company Live Nation and social media platform Snapchat have taken a similar route to Encore, with AR content that runs on mobile phones. In April, they co-launched AR filters that festival patrons could use during performances.
“When you look at the Live Nation experience, they are trying to see how they can engage people in new ways … on top of their experiences in the physical world," says Ms Hvitved from the Copenhagen Institute for Future Studies.
Ms Haueiss, the Sydney metaverse consultant, says AR at festivals could provide extra information, like "animations, instructions like the closest camp sides, location of your friends, and updates about line-up in real-time".
"Companies are testing and playing around, organising festivals, galleries, parties, concerts … the sky is the limit."
Gamifying music-making
Another hint at where entertainment is heading came in March, when Epic Games, the maker of blockbuster video games like Fortnite, bought the independent online music platform Bandcamp.
Epic hasn't revealed its plans for Bandcamp, but it's clear it sees an opportunity in combining music and the metaverse.
In 2020, a Travis Scott concert in Fortnite saw 12 million players attend and brought a record number of players online.
Australian start-up Splash, based in Brisbane, is also experimenting in this space.
It has created a game in Roblox (a gaming platform) where players have access to a suite of loops — guitars, synths, vocals, etc — that they can use to write music, which they can then perform as an avatar.
Fans can rate their performance, which affects their level of fame within the game. There is also a currency within Splash that players can earn and use to purchase things like clothing for their avatar.
Most Roblox users are teens or children and too young to perform in clubs, says Thomas Stenning, who works in machine learning at Splash.
"The person that's going to DJ in a club is most likely over the age of 18 or has a whole bunch of benefits [like] having expensive DJ decks and has probably spent multiple years honing their skills," he says.
He sees the potential of the game as unlocking the creativity of people who would otherwise not think of themselves as musicians.
The open, immersive spaces available on Roblox may be a sign of what's to come, similar to the way MySpace blazed a path in social media that Facebook later followed, Mr Stenning says.
"Some people call what Roblox is the beginning of the metaverse."
What's holding us back?
Aside from the technical challenges of creating interactive virtual worlds attended by millions of users at once, there's the simple problem that most people don't own VR headsets.
"Hardware, such as mixed-reality headsets, need to become less bulky, more affordable, with long battery life, a wide field of view, and stellar quality even in difficult situations such as direct sunlight," Ms Haueiss says.
And even if we had the headsets and the suits, could Australia's currently lacklustre internet support the vast amounts of data needed to render 3D spaces in real time?
Perhaps for these reasons, some of the most ambitious and well-funded recent metaverse concerts have been marked by technical glitches and frustrations.
A Meta-hosted concert, held in February and featuring the Foo Fighters, saw fans locked out after the digital lobby crashed, while the immersive 3D experience was marred by visible human camera operators.
On top of the technical questions, there are social ones. Who will own the metaverse? Will it be open like the world wide web, or proprietary? Will it be accessible to all or niche and gated? These questions remain unanswered.
Despite the potential problems, Ms Hvitved says we should focus on the direction of technology and the behaviours of young audiences, rather than any specific company vision or technology.
"The famous way of looking at this is: if [Henry] Ford had asked people what they wanted, they wanted faster horses," she says.
"Sometimes it's difficult to see these solutions."
What's your ABC story?
Share your story by filling in the form below, or you can email yourABCstory@your.abc.net.au.
