I normally am ahead of the curve on technology but today I saw a YouTuber named Codemiko. A quick search will show you that she/he is a virtual reality character played by someone. The person conducts interviews as the virtual reality player. Mouth and body movement involved. Can someone explain this like I’m 5 with details so I can learn how to do it?
In: Technology
Well, there are lots of different ways of doing it, but essentially they all boil down to two things: Some means of capturing what the user’s face is doing, and some means of animating the character based on that information.
The first part is achieved through facial recognition software that can locate your eyes, eyebrows, mouth and other prominent facial features and track what they’re doing. A camera points at your face and logs what you do. Move your head side to side and it’ll see you do that and say to the bit of the program that animates the model “yo, move the head by this amount in this direction”.
The second part is similar to how video games animate characters, only where video games have their animations pre-built as files, these virtual youtuber softwares create the animations in real-time by interpreting the data that the camera part of the software sends to it. These animations are then applied to a 3D model. There are also 2D versions which are usually all grouped together and called “Live2D”. Rather than animate a 3D model, these split up a 2D image into multiple layers and move the layers independently of one another, moving some more than others to simulate the turning of a head.
Mouth animation sometimes tracks the physical movement of your mouth – this is easy to do but can result in something that doesn’t really match what you’re saying because you don’t usually move your mouth much when you speak. You can also achieve it by listening to what’s being said, and having the program guess what mouth shape to make based on the vowel sounds it hears. This can be more accurate, but it’s harder to do and it can look bad if not done well.
[this video](https://www.youtube.com/watch?v=jE2tnHhPuS8) gives a good insight into how the animating function works. It looks like she’s using a mouth shape thing rather than a listening thing to do mouth shape. You can see that strange body suit she’s wearing – that helps the camera track her movements, because it can recognise red stripe on black background and go “ah this must be this limb!” The 11:10 timestamp demonstrates what happens to the model when the bits the camera is using to track the body are removed. It’s still tracking the suit, but as it’s being removed it’s resulting in weird distortions because arms are suddenly in the wrong place. This is all commercially available (really expensive) technology btw, it’s basically a home version of the suits they use for motion tracking in hollywood movies.
Also if you want to do it yourself, there are cheaper options (there are free Live2D softwares for example, and if push comes to shove you can even use certain VR programs to do it – a Vtuber called Uehasu is a good example of how you can do it in VR), but you’ll still need to splurge on a model. Those things don’t come cheap.
Latest Answers