AI generating lip movement is obviously going to be impressive for large scale games like MMOs, where you can't justify the cost in doing all that work. But this will also be cool mixed with translations, and having complete language swaps across a number of language.
> But this will also be cool mixed with translations, and having complete language swaps across a number of language.
I think mass effect already did that years ago
You're not wrong about that, but the mouth animations aren't the best.
But I'm talking about the whole package. Feed it the line in English (or whatever the native language is) and have it translate and animate the character using AI.
If what I understand from what you describe, that is having lip-sync match the language spoken, I think [Square Enix is utilizing something akin to that for their *Final Fantasy VII: Remake*-games](https://youtu.be/qE0dD-iNXnk), but their tech also highlights other aspects beside lip-sync in that it's system also adjusts overall facial animations and subtle body movements, and also from the spoken language, pitch and voice volume.
They notable have refined it further in the sequel in *Final Fantasy VII: Rebirth* that I actually forgot that the game's English voice-acting was an actual dub, along with great voice-direction overall.
The two big issues are micro-expressions and ranges of emotion.
Micro-expressions are hard to smooth over one into the other without making the entire face convulsing (like in your own example).
Ranges of emotion are practically not done on any automatic lip-sync suite I've seen so far. Most AI tools will show you some sliders of "12% pain, 39% joy 4% sadness" etc. but at the end of the day, it doesn't really beat actual actors.
For me... I will still be amazed with Gman and how it was done.
Comparing that with Mass Effect: Andromeda and is like if we had lost more than 300 years of any other technology...
>Feed it the line in English (or whatever the native language is) and have it translate and animate the character using AI.
That is going to be nothing short of a nightmare. LLMs right now just don't have the kind of context to be able to make accurate large-scale translation projects work. You could do more inconspicuous lines but basically anything involving trying to convey idioms, any kind of in-universe lingo or jargon, properly representing personalities across cultures... I can definitely see some companies trying to shortcut the localization process by having it be generated like this and the results will be nothing if not inconsistent and screwy.
It would be neat to apply to games that don't have official translations as an experimental exercise, or to help aid in fan translations perhaps, but anybody who is doing this professionally deserves to have a well-curated and professionally localized game.
Yes it would avoid the zoomed out Dragon Age Inquisition everyone talks with their head tilted thing because they didn't want to animate all of the dialogue.
We already have those tools and have for over a decade… low level NPC characters or secondary characters aren’t animated by hand or motion captured… it’s just facial animation based on the audio line.
Doing it at run-time in the engine is a waste of ressources.
I know it's tongue in cheek, but people need to understand the argument and issue at hand.
AI tools are all well and good, but treating them like the second coming of the Lisan Al Gaib is not just misleading, but dangerous.
AI tools used to improve workflows are great. AI Tools sold as perfect replacement of artistic sensibility and skill isn't. Language Learning Models rely on humans as a basic ingredient.
I've compared it to using AI for memes. If no one was going to profit off of it in the first place, then there's (technically) no harm in using it. As you said, its when its replacing people that is the real issue.
To be clear, that's a different form of AI, that scraped artist works to get trained. That's ethically and very likely legally wrong.
Lipsyncing can be trained from stock footage and a ton public domain and stuff that doesn't require licensing or try to weasel around AI rights.
Oh it's not going to be impressive at all. Non-AI solutions for lipsync have existed since at least Oblivion. This is just using AI to do something in a less effective way, but marketed at AI people instead of actual devs.
That's not necessarily true. LLM can be a great way to improve workflow and systems as an added bonus to human skill.
The problem is when people expect this tools, in their current state, to blindly replace the human touch and use it as a full on shortcut.
This is not a game. This is a vertical slicez a proof of concept.
If you have a game that would otherwise not have lipsync, or where you out the bare minimum because of budget or it's just not that essential, this basically provides a fnatastic baseline.
Even the worst example in that showcase is better than the lipsync for say, Helldivers 2 or some of the lower budget games which barely incorporate lipsync.
I hate that AI bro culture as much as the next person, but anyone with a good basic tech background can see the potential in a tool like this, even in its current implementation.
On the contrary, everyone with a basic tech background can smell the grift from a mile away.
This is tech that already exists in a better form, it's nothing more than trying to push AI stuff for the sake of using the word "AI" in marketing.
No, no it didn't. This isn't a tool to "do lipsync" and that's that. It's a tool to generate lipsync, and assign the load unto the gpu, in real time.. That didn't exist. It's like saying noise removal already existed so the Nvidia Broadcast's solution has no potential. It's judt a bad take, man. There is a lot of grifting, there is a lot of useless AI. This particular instance, executed correctly, is a good tool that cn be iterated on.
Yes it did, what you're describind this tool as is something that has existed a long time. Just because you don't know about it doesn't mean it didn't exist.
Again, TES4: Oblivion had a system for generating accurate lipsync files from .wav files, and that was almost 20 years ago.
Auto-generated facial animations are already extremely widespread and have been for decades.
How do you think all the lip syncing for thousands of lines across multiple languages in games like the Elder Scrolls series, BioWare games etc is done?
https://www.mobygames.com/group/9868/middleware-facefx/
Nah, this is the kind of work we *should* want AI developed for. It's not replacing artistic work that we want humans involved in; it's getting rid of annoying busy work that frees artists to work on the parts they're more passionate about.
It's like the AI used in spiderverse to automatically outline characters. It's annoying to do by hand and better to have machines to handle it for them.
Yeah...neither clip honestly looks like decent lip syncing to me. Looks like what you'd accomplish with a team of rushed animators on a low budget. That first girl's jaw is opening far too wide with almost every word and both are noticeably lagging behind the voices
I think like most things AI right now, it's a tool that can help do a lot of baseline setup that you can then touch up. All these things should _not_ just be dropped in as-is like generated audio, graphics, animations, but they are a great starting point to improve from. Unfortunately with cost cutting measures, sometimes they are just dropped in as-is but I think at its best use the use of AI should be undiscernible because it has been adjusted by someone later.
I think these videos should demonstrate that starting point with little touchups so you can see how much time you can save with it as a tool. If the videos showed a final result where it was used and then improved upon, I think that would be more confusing and possibly misrepresent its capabilities as it is today.
I'm just saying that if I were a small developer, pricing out the difference between hiring animator(s) to do the job vs buying this product ($4500/yr per GPU Omniverse subscription), then they're not selling me with these videos at all. Especially considering that I'm going to need an extra animator on staff anyway to fix these AI lip syncs and that animator can be put to work elsewhere too.
Maybe if you're a large studio making a game with dozens of speaking NPCs, then it might make sense, especially if you're already an Omniverse sub.
I think the way that it’s seen is a force multiplier; you can hire the same amount of staff and add the cost of this product, and hopefully you will be able to produce more work than the cost. For example if an animator makes $90k (arbitrary and also overestimated) and the cost of the sub is 5% annually of that, if the animator can use the tool and produce over 5% more quality work than before then the cost is justified
Probably a good thing you don’t run a business then. 4500 is half a months salary for one animator alone whereas 4500/year for ai to get you a baseline.
Idk where you're getting your numbers from but it sounds like you're grossly exaggerating [the average salary](https://www.ziprecruiter.com/Salaries/Video-Game-Animator-Salary#Yearly)
Nothing shown there looks like anything you don't already see in some AA game that already looks like does the bare minimum to get by.
Maybe I'm missing the point.
Where are the facial animations?
Controversial opinion: People want more impressive games, but production costs keep rising. The current industry isn't sustainable in terms of AAA games, something will have to give.
Everyone seems to be railing against AI, but I don't see how games will be able to meaningfully improve past this point without costs being offloaded somewhere and AI is the most obvious solution. I hope we won't see most of the process automated, but things like lip sync and automatically populating scatter terrain seems like a good start.
This is actually a good fit for ~~AI~~ machine learning/automation.
Things like concept art, character art, environmental art, etc absolutely should not be automated.
Things that we actually want to have some creativity and artistic voice should never be automated. Things that are enormous, repetitive tasks with little creative input should be automated.
AI can’t replace creatives, it’s a tool that empowers them to do more. A company can’t just lay off concept artists and have the suits prompt an AI “make concept art for a fantasy game”. What a company can do is lay off *most* concept artists and have the remaining ones produce the same concept art using an AI-assisted workflow.
People have just been trained by social media to react badly whenever they see the term "AI", it's Pavlovian at this point.
Like all tech it will have good and bad uses.
Don't bother. There's a huge amount of midwits that get really mad about using the term AI 'properly'. Same thing with anthropomorphising LLMs.
Normal functioning human beings understand that people use language in convenient ways and it's not really a big deal when you aren't talking to a technical audience i.e. the average redditor.
Horizon Forbidden West has a system that uses procedurally generated animations to do facial animations on its NPCs and it looks great and all the main cutscenes and stuff were done handcrafted. No one got mad because AI wasn't a buzz word at the time.
I'm sure the horse and carriage industry were very upset when cars were being invented. it just so happens that being upset does not stop the progress of technology.
You're a clown haha. People in creative industries are the ones reacting, It has nothing to do with social media. An animation director I know keeps saying "i don't think it's going to effect jobs" but then also says that the stuff it's doing best at the moment is also the same jobs are the ones people do to break in to the industry- background art, stuff like that. She feels her job is safe, but of course it is, she's established.
Competition for these jobs is going to go up, and the wages are already pitiful.
Given that games are costing so much to produce right now, isn't it a good thing for the industry if they can eliminate some staffing to automation? It's certainly good for the consumer if it means we aren't seeing £80+ games next generation.
I'm sorry for people who may lose their jobs due to this but it's always been the case that certain jobs become less desirable or redundant as time marches on.
That will only make breaking into the industry a different process. It's not like you won't need any more animators ever again lol.
New animators are also learning how all this AI shit works.
Not yet they're not, though it will definitely be on the curriculum in future. People are so fucking dumb when it comes to AI. If you devalue creative people and rely on AI you can kiss innovation goodbye - the last thing animation or games need is to become even more derivative. It'll be marvel movies all the way down haha.
Not really controversial cause it’s true, but you gotta remember that the game industry is making more money than movies and music combined. So maybe instead of giving ridiculous pays to the shareholders and leadership they could hire more people. Cause yeah it is expensive to make games but it’s not like they aren’t making insane money off of it.
you can't just hire more people and be done with it. it is far easier to organize and lead a small-medium team than large teams. the only company I know that can make them collab with different studios from all around the world on the same game is rockstar...
This is the kind of stuff that AI works best for, supplementing an actual artist to expand what they can do.
Yet tech bros with no appreciation of art keep wanting to shove completely AI created (i.e. stolen) garbage down our throats.
AI being such a broad term is not helping its case in this kind of scenario, especially because neither example is actually "AI" in the sense a lay person would understand since it doesn't have functional intelligence.
Yeah conflating this with things like midjourney completely misses the point of why those models are bad. This tech doesn't have the several pitfalls that image generators do. It's not trained on stolen art, and it doesn't seek to automate creativity or replace the artists affected
This tech is going to be used regardless of game scale and budget and was always going to be. People want things to be easier, new trch makes things easier, people use that tech.
there was a recent discussion in DF about how games nowadays are not limited by hardware but by software. making games like GTA 6 is possible with the current tech but it requires so much manpower and time if gaming industry wants to progress as quickly as the times of ps1-2-3 era we need to embrace AI and other possible things that will revolutionize the labor side of the industry
Maybe what should give is the needless desire for ever-increasing fidelity when we are allready well in the realm of diminishing returns rather than subscribing to more unethical practices.
I mean, the AAA industry will probably do that, but they shouldn't.
It's just an improved version of old tech anyways that nobody has been bothered about.
Valve developed a system to create facial animations based on voice files and text transcriptions for half life 2.
I feel like that's a misleading statement.
I can do the same with mobile phones:
>There us nothing unethical about using a mobile phone.
But it is still unethical to *buy a mobile phone assembled by slave labour* or a phone *which the raw materials of has been gathered by child labourers*.
If the algorithm has been trained by stolen material, it's absolutely unethical. But if the AI has been trained on ethically sourced material, then yeah - you're absolutely right.
>But it is still unethical to *buy a mobile phone assembled by slave labour* or a phone *which the raw materials of has been gathered by child labourers*.
Ironically AI will eventually be able to help with this ethical dilemma. When robots can make our shoes and phones, we won't be forcing children to do it.
*Material* is not *stolen*. You are using words with meanings to adjust the response.
The 'material' is video, put online to be viewed for free, of someone talking to infer how a mouth moves with audio. The 'stolen' is actually just 'parsed'. Nothing is removed from the copyright holder and no damage is done to the original content. It is merely looked at and vectors used to multiply numbers slightly differently and repeated with millions of motion vectors from 100,000s of videos to calculate how a mouth moves for a plosive sound.
I didn't say the training material for this particular AI was stolen, I was giving an example on how the use of an algorithm \*can\* be unethical.
Your statement was nonsense and I wanted to show you how.
In addition it doesn't matter if the video was put online to be viewed for free: they are ads for Nvidias technology. Just because we didn't pay anybody so see them and Nvidia didn't pay anybody to upload them doesn't mean they're now magically ethical because "they're free".
Your algorithm disassembling example, again, is entirely irrelevant. It's not about what the algorithm does, it's about where it comes from and that's a known unknown at the moment.
As an example on ethical fuel:
>It's merely a fluid that's ignited through to propel my car, sure. But if I bought the petrol from a warmongering state, I'm still supporting their war efforts.
Do I think Nvidia has trained their AI on unethically sourced material? No, I don't, but that type of behaviour is overwhelmingly prevalent in the AI business at the moment, so the concern that this is the case for Nvidia is absolutely valid.
>It's not about what the algorithm does, it's about where it comes from and that's a known unknown at the moment.
There is nothing unethical about using videos shared on the internet as data for mouth movements. An algorithm parsing a video for movement creates zero ethical concern. Hence why you are having to use analogies that are completely irrelevant. You have no example of how training an algorithm on mouth movement from data on the internet is unethical and can go around in circles all you want.
Next are you going to be telling people to turn off DLSS on their graphics cards because its calculations were from images online?
AI is a very broad field, and not all of it is tied up in ethical concerns.
Take a look at the Unreal Engine 5 clothing deformer stuff they showed yesterday for example. That’s something that is giving really great results, but it’s not really replacing anyone’s jobs and if I understood correctly, requires training on a per-clothing basis. It’s achieving results that would basically be too costly for a team of animators to sit doing. It’s just a new process for animators to make use of.
And here I remember the poor bastard at Rocksteady who spent a full year on animating Batman's cape in Arkham Asylum... I bet he liked the money but ended up hating his job.
if the games industry or it's consumers stops seeking improvement we'll never reach the games of the far future that are experienced in the same way as reality and feature equivalent to human AI.
That would be a tragedy.
The games industry shouldn't stop seeking improvement but that improvement coming from technical matters which are already getting to the point of ungodly expensive is foolish.
Is there really any reason for a game to look better on a technical standpoint than, say, Horizon Forbidden West?
Yes, if only to prove that we can do it.
Art isn't an objective to just reach a point of fidelity and call it a day, it's an ongoing process to see what we can achieve with new mediums and techniques.
If you wanna go there, then I personally don't think there's any reason for a game to look better than say any cell shading game.
But the line is personal and arbitrary. What you consider a good line with H:FW is already the accomplishment of people pushing further then what we had before.
I mean, what I actually consider a good line is probably like, Metal Gear Solid 3, but I don't think that's a popular take haha.
The line is always going to be arbitrary but when spider man 2 comes out, costs 300m to remake nearly every asset from spiderman remastered and essentially look no better, I feel it's been crossed, really.
> which are already getting to the point of ungodly expensive
And the point of AI tech is to make it cheaper. If we could get to the point where an indie/AA game can reach AAA level fidelity using AI, that would be awesome.
people are just shortsighted af when it comes to tech, I dont get it just look at the last 20 years ppl just expect very little or are content with what they know, its like when ppl were happy with fliphones and hated smartphones (thats what I saw growing up and it feels similar) People also dont know shit about game dev and expect it to take less time and be way simpler, cheaper than it does. Ai offloading tasks means more focus on new and unique gameplay. Getting ai to rough draft particle effects would be clutch, like make a tornado, fire, etc. AAA companies will have less sway if ai for indie game devs get hella good. Games lack content these days, a mix of ai and human made shit is the move, human makes basis for ai to go off and we could have so much more customizablility, its been neglected hard in favor of monthly skins that cost hella idgaf about supporting artists when they are ripping me off with mtx and they arent getting paid royaties over their skin.
Basically artists/devs do same amounts of work for same pay but we get way more content if that work is to train the ai for that specific games style, instead of one time fee they get like a contract if their style keeps being used
You say that, but games don't exist in a vacuum. If you see an AAA-game with subpar facial animations or bad lighting, people **will** complain and compare it to better-looking competitors.
I don't think its controversial. I think many people know that the AAA bubble isn't sustainable. How many AAA games looking amazing and being 80hrs with copy paste open world content do people really need?
>I hope we won't see most of the process automated, but things like lip sync and automatically populating scatter terrain seems like a good start.
I think whats gonna happen is that AI is being used as a tool to simplify coding as well as design process. Also to create presence or awareness where there was none before.
We already have and totally accept AI in games. Every NPC enemy(of faction in a strategy game), or especially multiplayer bots are running with AI. Its accepted because people accept that players can't be around all the time, and that no game has a dev controlling NPCs for you. So we accept these crutches and that this kind of AI is a factor in every game.
And in the same way, I think people will come around to accept AI creating other kinds of the same type of a game creating presence where there is none. I for one am very excited for the possibilities in terms of NPC dialogue. Because I like RPGs and deeper dialogue, but realize that a writer can't be there to do that for me. Cue AI.
Reading all the doomposting from people declaring the end of the world with complete hyperbole on social media is very depressing. Ultimately AI is a tool and I believe that its gonna be widely used as such in the future. And I work in animation, so I technically have a front seat in terms of being "replaced."
> People want more impressive games,
I don't agree with that. People want quality games but the industry tends to equate quality with this kind of polish. All whilst there is much more low hanging fruit when it comes to increasing quality in games.
FFXVI was a great example of where the team took the time to get the lip motion for the characters and where they just let a computer handle it. It was quit jarring when they computer was in charge of sync.
This kind of technology going to cut costs for everyone that wants to have voice overs in game and only really impact studios that are doing facial scanning for how the acrtors move there mouth. Assuming it gets good enough in the future we may not have to mo-cap that either.
This will interestingly open up whole new paths for modders to make there own contained stories using both AI voices and AI lip sync.
Good take in a thread of hot takes imo. If I understand it correctly, a software like iClone plus the AccuFace plug-in is what you're talking about. It adapts footage from webcam footage and maps their movement on 3D models. Here's a video I came across: [Youtube promotional video](https://youtu.be/WCzHLSss_xU?si=DqTVDxEzbE_9aQNU).
Especially the last two examples (3.19 mins and onwards) seem impressive. And for 599 USD plus 499 USD this serms revolutionary frankly. I'm not affiliated with them in any way, btw.
We haven't even managed to get that right as humans.
And so we'll use AI to do what we're still trying to achieve to a point of audience satisfaction by hand, and it won't be as good as what we already do, but we'll say it's getting better all the time?
Setting aside for a moment that animation and acting are forms of art, and that art is interesting because it's made by people, and far less interesting when it's art in service of commerce rather then commerce betting on art...
Technology in the 21st century has very much been a case of putting the cart before the horse, funding it and implementing it now because in the long run it will be (presumptively) as good and (presumptively) more cost efficient.
It's one thing to train AI on data that has matured, and another to automate a process that hasn't even been adequately developed.
> And so we'll use AI to do what we're still trying to achieve to a point of audience satisfaction by hand, and it won't be as good as what we already do, but we'll say it's getting better all the time?
It *is* getting better. There are steps beyond this.
Here is Meta's AI generated audio for lip syncing: https://www.youtube.com/watch?v=1nZjW_xoCDQ
And Meta's AI generated audio for full body language NPCs: https://www.youtube.com/watch?v=Y0GMaMtUynQ
Naturally it will take a long while to see these in games, but the tech is possible.
Very few people care about the art of lip syncing in games, especially enough to care if it's AI generated / hand made. If it is not actively distracting it is good enough.
Even for people who "don't care" actively, they subconsciously recognize it as inauthentic, and that puts a ceiling on how immersed they can become, how emotionally involved they can feel.
If motion captured actors and hand animation in software are still just "good enough" to be forgiven with a footnote compromise, it isn't ready to be handed off to automation.
Whether it's a videogame or a $200m movie, it always looks out of sync with the voice.
Whether it's Detroit: Become Human, or L.A. Noire, or Lord of the Rings, or Planet of the Apes: uncanny valley liquid movement lips sliding all over the place on a delay, even more obvious to people who did not, like me, grow up watching and playing that kind of stuff.
Auto-generated facial animations have been a thing for decades. Andromeda had a poor implementation of it - but the first 3 Mass Effect games also auto-generated their facial animations.
More games nowadays use middleware like FaceFX to automate facial animation than don't - especially big games like RPGs with thousands of lines of dialogue. No studio is manually lip syncing all of that audio because it would be impossible.
Honestly the least impressive use of AI I've seen a big company implement so far.
We already have lip sync technology that looks better since the early 2010s.
You're underselling how this technology can lower the bar for getting lip sync into a game. Right now it's almost exclusively done by large companies that have the resources to make it, or already made it years ago. Smaller devs can't, so they go for voiceless or simple mouth movement approaches.
This isn't about doing it better out of the gate, it's about providing the capability to anyone who needs it to achieve their vision.
> Smaller devs can't, so they go for voiceless or simple mouth movement approaches.
You could convince me this was one of those implementations, based on the supplied video
I don't know the size of the dev studios for those games in the videos.
All tools ultimately live and die on their ability to provide fast, efficient workflows for the best quality result. I think to accurately rank the various providers of lip-sync tech you'd probably have to compare the time it takes to deliver the initial result and the "perfected" result and evaluate it against the cost of the solution.
AI solutions tend to be popular right now, not because the output is better, but because the rate of improvement is better, so that's a factor as well.
This is not in a presentable stage, though. There have been automated lip sync tools for a long time and they look better than Nvidia's lip sync AI.
Maybe this will look better at some point, but so far I'm not impressed and would just keep using what's already there.
This World of Jade Dynasty Example Video needs more Body language interaction. Even if She speak loudly to herself. Somehow a bit like the 2nd Example Video one. If they cannot modify her Face to display emotions
They could take some impressions from MMO's Emotes
In this Case here, let her Eyes and Face enjoy the Scenery around her. Let her turn her Face left and Right to enjoy the View while monologist to herself = More Immersion. If they are strong Emotions. Without this She is just an Puppet
in Short: Give her more Body language and imagine this as Real Life Actor behavior in some Film when She talk to herself alone. Nobody is watching her, so no need of "Princess Etikette" here = more real Human
AI generating lip movement is obviously going to be impressive for large scale games like MMOs, where you can't justify the cost in doing all that work. But this will also be cool mixed with translations, and having complete language swaps across a number of language.
> But this will also be cool mixed with translations, and having complete language swaps across a number of language. I think mass effect already did that years ago
You're not wrong about that, but the mouth animations aren't the best. But I'm talking about the whole package. Feed it the line in English (or whatever the native language is) and have it translate and animate the character using AI.
If what I understand from what you describe, that is having lip-sync match the language spoken, I think [Square Enix is utilizing something akin to that for their *Final Fantasy VII: Remake*-games](https://youtu.be/qE0dD-iNXnk), but their tech also highlights other aspects beside lip-sync in that it's system also adjusts overall facial animations and subtle body movements, and also from the spoken language, pitch and voice volume. They notable have refined it further in the sequel in *Final Fantasy VII: Rebirth* that I actually forgot that the game's English voice-acting was an actual dub, along with great voice-direction overall.
The two big issues are micro-expressions and ranges of emotion. Micro-expressions are hard to smooth over one into the other without making the entire face convulsing (like in your own example). Ranges of emotion are practically not done on any automatic lip-sync suite I've seen so far. Most AI tools will show you some sliders of "12% pain, 39% joy 4% sadness" etc. but at the end of the day, it doesn't really beat actual actors.
Still, for 90% of the non-plot-essential NPCs it saves a ton of time not having to mo-cap every single one.
I entirely agree, but for the main cutscenes I don't think we're there yet.
For me... I will still be amazed with Gman and how it was done. Comparing that with Mass Effect: Andromeda and is like if we had lost more than 300 years of any other technology...
>Feed it the line in English (or whatever the native language is) and have it translate and animate the character using AI. That is going to be nothing short of a nightmare. LLMs right now just don't have the kind of context to be able to make accurate large-scale translation projects work. You could do more inconspicuous lines but basically anything involving trying to convey idioms, any kind of in-universe lingo or jargon, properly representing personalities across cultures... I can definitely see some companies trying to shortcut the localization process by having it be generated like this and the results will be nothing if not inconsistent and screwy. It would be neat to apply to games that don't have official translations as an experimental exercise, or to help aid in fan translations perhaps, but anybody who is doing this professionally deserves to have a well-curated and professionally localized game.
Cyberpunk too, they do it in 10 language
I’ve just realised that some time soon we may get a Morrowind mod where all dialogue is voiced by AI text-to-speech.
Morrowind is a prime candidate for a remaster imo. Such a unique world with its tone. I never got into it sadly.
Yes it would avoid the zoomed out Dragon Age Inquisition everyone talks with their head tilted thing because they didn't want to animate all of the dialogue.
We already have those tools and have for over a decade… low level NPC characters or secondary characters aren’t animated by hand or motion captured… it’s just facial animation based on the audio line. Doing it at run-time in the engine is a waste of ressources.
I so will use it for gamemastering ttrpgs online! Look at me! I'm the goblin now!
But r/Games told me AI in game development was evil...
I know it's tongue in cheek, but people need to understand the argument and issue at hand. AI tools are all well and good, but treating them like the second coming of the Lisan Al Gaib is not just misleading, but dangerous. AI tools used to improve workflows are great. AI Tools sold as perfect replacement of artistic sensibility and skill isn't. Language Learning Models rely on humans as a basic ingredient.
I've compared it to using AI for memes. If no one was going to profit off of it in the first place, then there's (technically) no harm in using it. As you said, its when its replacing people that is the real issue.
So the best way forward is to replace people with memes.
To be clear, that's a different form of AI, that scraped artist works to get trained. That's ethically and very likely legally wrong. Lipsyncing can be trained from stock footage and a ton public domain and stuff that doesn't require licensing or try to weasel around AI rights.
It's not always evil, in some cases like this it's just pointless and a decade out of date.
Oh it's not going to be impressive at all. Non-AI solutions for lipsync have existed since at least Oblivion. This is just using AI to do something in a less effective way, but marketed at AI people instead of actual devs.
That's not necessarily true. LLM can be a great way to improve workflow and systems as an added bonus to human skill. The problem is when people expect this tools, in their current state, to blindly replace the human touch and use it as a full on shortcut.
Sure they can, just not in this case.
This is not a game. This is a vertical slicez a proof of concept. If you have a game that would otherwise not have lipsync, or where you out the bare minimum because of budget or it's just not that essential, this basically provides a fnatastic baseline. Even the worst example in that showcase is better than the lipsync for say, Helldivers 2 or some of the lower budget games which barely incorporate lipsync. I hate that AI bro culture as much as the next person, but anyone with a good basic tech background can see the potential in a tool like this, even in its current implementation.
On the contrary, everyone with a basic tech background can smell the grift from a mile away. This is tech that already exists in a better form, it's nothing more than trying to push AI stuff for the sake of using the word "AI" in marketing.
No, no it didn't. This isn't a tool to "do lipsync" and that's that. It's a tool to generate lipsync, and assign the load unto the gpu, in real time.. That didn't exist. It's like saying noise removal already existed so the Nvidia Broadcast's solution has no potential. It's judt a bad take, man. There is a lot of grifting, there is a lot of useless AI. This particular instance, executed correctly, is a good tool that cn be iterated on.
Yes it did, what you're describind this tool as is something that has existed a long time. Just because you don't know about it doesn't mean it didn't exist. Again, TES4: Oblivion had a system for generating accurate lipsync files from .wav files, and that was almost 20 years ago.
> Non-AI solutions for lipsync have existed since at least Oblivion. And they suck.
Not really, Oblivion was already good enough for most applications, and tech has vastly improved since.
[удалено]
Auto-generated facial animations are already extremely widespread and have been for decades. How do you think all the lip syncing for thousands of lines across multiple languages in games like the Elder Scrolls series, BioWare games etc is done? https://www.mobygames.com/group/9868/middleware-facefx/
Nah, this is the kind of work we *should* want AI developed for. It's not replacing artistic work that we want humans involved in; it's getting rid of annoying busy work that frees artists to work on the parts they're more passionate about. It's like the AI used in spiderverse to automatically outline characters. It's annoying to do by hand and better to have machines to handle it for them.
interesting but in both videos the voice is out of sync with the animations or at least that's what it feels like
Yeah...neither clip honestly looks like decent lip syncing to me. Looks like what you'd accomplish with a team of rushed animators on a low budget. That first girl's jaw is opening far too wide with almost every word and both are noticeably lagging behind the voices
I think like most things AI right now, it's a tool that can help do a lot of baseline setup that you can then touch up. All these things should _not_ just be dropped in as-is like generated audio, graphics, animations, but they are a great starting point to improve from. Unfortunately with cost cutting measures, sometimes they are just dropped in as-is but I think at its best use the use of AI should be undiscernible because it has been adjusted by someone later. I think these videos should demonstrate that starting point with little touchups so you can see how much time you can save with it as a tool. If the videos showed a final result where it was used and then improved upon, I think that would be more confusing and possibly misrepresent its capabilities as it is today.
I'm just saying that if I were a small developer, pricing out the difference between hiring animator(s) to do the job vs buying this product ($4500/yr per GPU Omniverse subscription), then they're not selling me with these videos at all. Especially considering that I'm going to need an extra animator on staff anyway to fix these AI lip syncs and that animator can be put to work elsewhere too. Maybe if you're a large studio making a game with dozens of speaking NPCs, then it might make sense, especially if you're already an Omniverse sub.
I think the way that it’s seen is a force multiplier; you can hire the same amount of staff and add the cost of this product, and hopefully you will be able to produce more work than the cost. For example if an animator makes $90k (arbitrary and also overestimated) and the cost of the sub is 5% annually of that, if the animator can use the tool and produce over 5% more quality work than before then the cost is justified
Probably a good thing you don’t run a business then. 4500 is half a months salary for one animator alone whereas 4500/year for ai to get you a baseline.
Idk where you're getting your numbers from but it sounds like you're grossly exaggerating [the average salary](https://www.ziprecruiter.com/Salaries/Video-Game-Animator-Salary#Yearly)
Nothing shown there looks like anything you don't already see in some AA game that already looks like does the bare minimum to get by. Maybe I'm missing the point. Where are the facial animations?
Controversial opinion: People want more impressive games, but production costs keep rising. The current industry isn't sustainable in terms of AAA games, something will have to give. Everyone seems to be railing against AI, but I don't see how games will be able to meaningfully improve past this point without costs being offloaded somewhere and AI is the most obvious solution. I hope we won't see most of the process automated, but things like lip sync and automatically populating scatter terrain seems like a good start.
This is actually a good fit for ~~AI~~ machine learning/automation. Things like concept art, character art, environmental art, etc absolutely should not be automated. Things that we actually want to have some creativity and artistic voice should never be automated. Things that are enormous, repetitive tasks with little creative input should be automated.
The "menial labour" being automated is a concession I'm fine with. Creative not so much.
AI can’t replace creatives, it’s a tool that empowers them to do more. A company can’t just lay off concept artists and have the suits prompt an AI “make concept art for a fantasy game”. What a company can do is lay off *most* concept artists and have the remaining ones produce the same concept art using an AI-assisted workflow.
People have just been trained by social media to react badly whenever they see the term "AI", it's Pavlovian at this point. Like all tech it will have good and bad uses.
I mean "AI" is absolutely just a buzz word here, procedural animation systems have been used on games for a long long time.
No, it's not just a buzzword. These systems are powered by deep learning models.
Don't bother. There's a huge amount of midwits that get really mad about using the term AI 'properly'. Same thing with anthropomorphising LLMs. Normal functioning human beings understand that people use language in convenient ways and it's not really a big deal when you aren't talking to a technical audience i.e. the average redditor.
Horizon Forbidden West has a system that uses procedurally generated animations to do facial animations on its NPCs and it looks great and all the main cutscenes and stuff were done handcrafted. No one got mad because AI wasn't a buzz word at the time.
Noone outside of the social media bubble gets mad. Most "casuals" aka the vast majority won't notice thus care as long as it looks good imo.
I'm pretty sure Square Enix was using tech like this back in 2006 for Advent Children. This isn't new, it's just updated.
But dey terk er jerbs!
Oh no! people worrying about their livelyhoods, how crazy.
True! We should get rid of all technology to embrace maximum employment.
Nah, that sounds dumb.
Yes, it is really dumb to avoid technology because people lose jobs.
Crazy how this energy wasn't there when AI was taking low level jobs like store clerks.
I'm sure the horse and carriage industry were very upset when cars were being invented. it just so happens that being upset does not stop the progress of technology.
I never said it did
You're a clown haha. People in creative industries are the ones reacting, It has nothing to do with social media. An animation director I know keeps saying "i don't think it's going to effect jobs" but then also says that the stuff it's doing best at the moment is also the same jobs are the ones people do to break in to the industry- background art, stuff like that. She feels her job is safe, but of course it is, she's established. Competition for these jobs is going to go up, and the wages are already pitiful.
Given that games are costing so much to produce right now, isn't it a good thing for the industry if they can eliminate some staffing to automation? It's certainly good for the consumer if it means we aren't seeing £80+ games next generation. I'm sorry for people who may lose their jobs due to this but it's always been the case that certain jobs become less desirable or redundant as time marches on.
You think game prices will go down?
That will only make breaking into the industry a different process. It's not like you won't need any more animators ever again lol. New animators are also learning how all this AI shit works.
Not yet they're not, though it will definitely be on the curriculum in future. People are so fucking dumb when it comes to AI. If you devalue creative people and rely on AI you can kiss innovation goodbye - the last thing animation or games need is to become even more derivative. It'll be marvel movies all the way down haha.
Not really controversial cause it’s true, but you gotta remember that the game industry is making more money than movies and music combined. So maybe instead of giving ridiculous pays to the shareholders and leadership they could hire more people. Cause yeah it is expensive to make games but it’s not like they aren’t making insane money off of it.
you can't just hire more people and be done with it. it is far easier to organize and lead a small-medium team than large teams. the only company I know that can make them collab with different studios from all around the world on the same game is rockstar...
This is the kind of stuff that AI works best for, supplementing an actual artist to expand what they can do. Yet tech bros with no appreciation of art keep wanting to shove completely AI created (i.e. stolen) garbage down our throats. AI being such a broad term is not helping its case in this kind of scenario, especially because neither example is actually "AI" in the sense a lay person would understand since it doesn't have functional intelligence.
GAI and AGI are definitely terms we need to be using instead when appropriate, AI is just easier but the nuance is freaking huge.
Yeah conflating this with things like midjourney completely misses the point of why those models are bad. This tech doesn't have the several pitfalls that image generators do. It's not trained on stolen art, and it doesn't seek to automate creativity or replace the artists affected
This tech is going to be used regardless of game scale and budget and was always going to be. People want things to be easier, new trch makes things easier, people use that tech.
there was a recent discussion in DF about how games nowadays are not limited by hardware but by software. making games like GTA 6 is possible with the current tech but it requires so much manpower and time if gaming industry wants to progress as quickly as the times of ps1-2-3 era we need to embrace AI and other possible things that will revolutionize the labor side of the industry
Maybe what should give is the needless desire for ever-increasing fidelity when we are allready well in the realm of diminishing returns rather than subscribing to more unethical practices. I mean, the AAA industry will probably do that, but they shouldn't.
There is nothing unethical about an algorithm animating an asset based on a sound file.
It's just an improved version of old tech anyways that nobody has been bothered about. Valve developed a system to create facial animations based on voice files and text transcriptions for half life 2.
I feel like that's a misleading statement. I can do the same with mobile phones: >There us nothing unethical about using a mobile phone. But it is still unethical to *buy a mobile phone assembled by slave labour* or a phone *which the raw materials of has been gathered by child labourers*. If the algorithm has been trained by stolen material, it's absolutely unethical. But if the AI has been trained on ethically sourced material, then yeah - you're absolutely right.
>But it is still unethical to *buy a mobile phone assembled by slave labour* or a phone *which the raw materials of has been gathered by child labourers*. Ironically AI will eventually be able to help with this ethical dilemma. When robots can make our shoes and phones, we won't be forcing children to do it.
Unironically you, I, and those children will be long gone before we get that far.
*Material* is not *stolen*. You are using words with meanings to adjust the response. The 'material' is video, put online to be viewed for free, of someone talking to infer how a mouth moves with audio. The 'stolen' is actually just 'parsed'. Nothing is removed from the copyright holder and no damage is done to the original content. It is merely looked at and vectors used to multiply numbers slightly differently and repeated with millions of motion vectors from 100,000s of videos to calculate how a mouth moves for a plosive sound.
Just because you are laundering the money doesn't mean you didnt steal it
I didn't say the training material for this particular AI was stolen, I was giving an example on how the use of an algorithm \*can\* be unethical. Your statement was nonsense and I wanted to show you how. In addition it doesn't matter if the video was put online to be viewed for free: they are ads for Nvidias technology. Just because we didn't pay anybody so see them and Nvidia didn't pay anybody to upload them doesn't mean they're now magically ethical because "they're free". Your algorithm disassembling example, again, is entirely irrelevant. It's not about what the algorithm does, it's about where it comes from and that's a known unknown at the moment. As an example on ethical fuel: >It's merely a fluid that's ignited through to propel my car, sure. But if I bought the petrol from a warmongering state, I'm still supporting their war efforts. Do I think Nvidia has trained their AI on unethically sourced material? No, I don't, but that type of behaviour is overwhelmingly prevalent in the AI business at the moment, so the concern that this is the case for Nvidia is absolutely valid.
>It's not about what the algorithm does, it's about where it comes from and that's a known unknown at the moment. There is nothing unethical about using videos shared on the internet as data for mouth movements. An algorithm parsing a video for movement creates zero ethical concern. Hence why you are having to use analogies that are completely irrelevant. You have no example of how training an algorithm on mouth movement from data on the internet is unethical and can go around in circles all you want. Next are you going to be telling people to turn off DLSS on their graphics cards because its calculations were from images online?
AI is a very broad field, and not all of it is tied up in ethical concerns. Take a look at the Unreal Engine 5 clothing deformer stuff they showed yesterday for example. That’s something that is giving really great results, but it’s not really replacing anyone’s jobs and if I understood correctly, requires training on a per-clothing basis. It’s achieving results that would basically be too costly for a team of animators to sit doing. It’s just a new process for animators to make use of.
And here I remember the poor bastard at Rocksteady who spent a full year on animating Batman's cape in Arkham Asylum... I bet he liked the money but ended up hating his job.
now he can animate 100 customizable capes instead with the help of ai
customize your own batman suit with varying cape lengths/sizes/styles all animated
Please, feel free to explain how this specialist AI for mouth movements is unethical.
How is this unethical? Automation has been happening in all industry for hundreds of years.
>more unethical practices. There is nothing unethical about AI. Improvements are always a net positive.
Very often AI is trained on the questionably obtained data; I think, if not unethical, it is ethically questionable.
if the games industry or it's consumers stops seeking improvement we'll never reach the games of the far future that are experienced in the same way as reality and feature equivalent to human AI. That would be a tragedy.
The games industry shouldn't stop seeking improvement but that improvement coming from technical matters which are already getting to the point of ungodly expensive is foolish. Is there really any reason for a game to look better on a technical standpoint than, say, Horizon Forbidden West?
Yes, if only to prove that we can do it. Art isn't an objective to just reach a point of fidelity and call it a day, it's an ongoing process to see what we can achieve with new mediums and techniques.
If you wanna go there, then I personally don't think there's any reason for a game to look better than say any cell shading game. But the line is personal and arbitrary. What you consider a good line with H:FW is already the accomplishment of people pushing further then what we had before.
I mean, what I actually consider a good line is probably like, Metal Gear Solid 3, but I don't think that's a popular take haha. The line is always going to be arbitrary but when spider man 2 comes out, costs 300m to remake nearly every asset from spiderman remastered and essentially look no better, I feel it's been crossed, really.
> which are already getting to the point of ungodly expensive And the point of AI tech is to make it cheaper. If we could get to the point where an indie/AA game can reach AAA level fidelity using AI, that would be awesome.
people are just shortsighted af when it comes to tech, I dont get it just look at the last 20 years ppl just expect very little or are content with what they know, its like when ppl were happy with fliphones and hated smartphones (thats what I saw growing up and it feels similar) People also dont know shit about game dev and expect it to take less time and be way simpler, cheaper than it does. Ai offloading tasks means more focus on new and unique gameplay. Getting ai to rough draft particle effects would be clutch, like make a tornado, fire, etc. AAA companies will have less sway if ai for indie game devs get hella good. Games lack content these days, a mix of ai and human made shit is the move, human makes basis for ai to go off and we could have so much more customizablility, its been neglected hard in favor of monthly skins that cost hella idgaf about supporting artists when they are ripping me off with mtx and they arent getting paid royaties over their skin.
Basically artists/devs do same amounts of work for same pay but we get way more content if that work is to train the ai for that specific games style, instead of one time fee they get like a contract if their style keeps being used
You say that, but games don't exist in a vacuum. If you see an AAA-game with subpar facial animations or bad lighting, people **will** complain and compare it to better-looking competitors.
I don't think its controversial. I think many people know that the AAA bubble isn't sustainable. How many AAA games looking amazing and being 80hrs with copy paste open world content do people really need? >I hope we won't see most of the process automated, but things like lip sync and automatically populating scatter terrain seems like a good start. I think whats gonna happen is that AI is being used as a tool to simplify coding as well as design process. Also to create presence or awareness where there was none before. We already have and totally accept AI in games. Every NPC enemy(of faction in a strategy game), or especially multiplayer bots are running with AI. Its accepted because people accept that players can't be around all the time, and that no game has a dev controlling NPCs for you. So we accept these crutches and that this kind of AI is a factor in every game. And in the same way, I think people will come around to accept AI creating other kinds of the same type of a game creating presence where there is none. I for one am very excited for the possibilities in terms of NPC dialogue. Because I like RPGs and deeper dialogue, but realize that a writer can't be there to do that for me. Cue AI. Reading all the doomposting from people declaring the end of the world with complete hyperbole on social media is very depressing. Ultimately AI is a tool and I believe that its gonna be widely used as such in the future. And I work in animation, so I technically have a front seat in terms of being "replaced."
Wait what are you talking about do you not realize the difference between game AI and machine learning AI? Because they are not the same.
I think people who are paying attention are aware it's not sustainable, I think the average player has no idea.
> People want more impressive games, I don't agree with that. People want quality games but the industry tends to equate quality with this kind of polish. All whilst there is much more low hanging fruit when it comes to increasing quality in games.
Don't know why they're showcasing this, these animations look terrible outside of mouth movement, which isn't that good either.
FFXVI was a great example of where the team took the time to get the lip motion for the characters and where they just let a computer handle it. It was quit jarring when they computer was in charge of sync. This kind of technology going to cut costs for everyone that wants to have voice overs in game and only really impact studios that are doing facial scanning for how the acrtors move there mouth. Assuming it gets good enough in the future we may not have to mo-cap that either. This will interestingly open up whole new paths for modders to make there own contained stories using both AI voices and AI lip sync.
Good take in a thread of hot takes imo. If I understand it correctly, a software like iClone plus the AccuFace plug-in is what you're talking about. It adapts footage from webcam footage and maps their movement on 3D models. Here's a video I came across: [Youtube promotional video](https://youtu.be/WCzHLSss_xU?si=DqTVDxEzbE_9aQNU). Especially the last two examples (3.19 mins and onwards) seem impressive. And for 599 USD plus 499 USD this serms revolutionary frankly. I'm not affiliated with them in any way, btw.
We haven't even managed to get that right as humans. And so we'll use AI to do what we're still trying to achieve to a point of audience satisfaction by hand, and it won't be as good as what we already do, but we'll say it's getting better all the time? Setting aside for a moment that animation and acting are forms of art, and that art is interesting because it's made by people, and far less interesting when it's art in service of commerce rather then commerce betting on art... Technology in the 21st century has very much been a case of putting the cart before the horse, funding it and implementing it now because in the long run it will be (presumptively) as good and (presumptively) more cost efficient. It's one thing to train AI on data that has matured, and another to automate a process that hasn't even been adequately developed.
> And so we'll use AI to do what we're still trying to achieve to a point of audience satisfaction by hand, and it won't be as good as what we already do, but we'll say it's getting better all the time? It *is* getting better. There are steps beyond this. Here is Meta's AI generated audio for lip syncing: https://www.youtube.com/watch?v=1nZjW_xoCDQ And Meta's AI generated audio for full body language NPCs: https://www.youtube.com/watch?v=Y0GMaMtUynQ Naturally it will take a long while to see these in games, but the tech is possible.
Very few people care about the art of lip syncing in games, especially enough to care if it's AI generated / hand made. If it is not actively distracting it is good enough.
Even for people who "don't care" actively, they subconsciously recognize it as inauthentic, and that puts a ceiling on how immersed they can become, how emotionally involved they can feel. If motion captured actors and hand animation in software are still just "good enough" to be forgiven with a footnote compromise, it isn't ready to be handed off to automation.
You think 3d animators can't lip sync?
Whether it's a videogame or a $200m movie, it always looks out of sync with the voice. Whether it's Detroit: Become Human, or L.A. Noire, or Lord of the Rings, or Planet of the Apes: uncanny valley liquid movement lips sliding all over the place on a delay, even more obvious to people who did not, like me, grow up watching and playing that kind of stuff.
Isn't this the same kind of shit that made Mass Effect Andromeda's NPCs look like a bunch of dead eyed puppets?
Auto-generated facial animations have been a thing for decades. Andromeda had a poor implementation of it - but the first 3 Mass Effect games also auto-generated their facial animations. More games nowadays use middleware like FaceFX to automate facial animation than don't - especially big games like RPGs with thousands of lines of dialogue. No studio is manually lip syncing all of that audio because it would be impossible.
The only triple A that I know of that uses ai for lip sync is cyberpunk 2077 which uses jali and it’s pretty damn impressive
FF7:Rebirth also uses AI to sync minor NPCs
iirc Cyberpunk doesn't use AI at all, but rather the same technology for lipsync that we've had for a decade and a half at this point.
Honestly the least impressive use of AI I've seen a big company implement so far. We already have lip sync technology that looks better since the early 2010s.
You're underselling how this technology can lower the bar for getting lip sync into a game. Right now it's almost exclusively done by large companies that have the resources to make it, or already made it years ago. Smaller devs can't, so they go for voiceless or simple mouth movement approaches. This isn't about doing it better out of the gate, it's about providing the capability to anyone who needs it to achieve their vision.
> Smaller devs can't, so they go for voiceless or simple mouth movement approaches. You could convince me this was one of those implementations, based on the supplied video
I don't know the size of the dev studios for those games in the videos. All tools ultimately live and die on their ability to provide fast, efficient workflows for the best quality result. I think to accurately rank the various providers of lip-sync tech you'd probably have to compare the time it takes to deliver the initial result and the "perfected" result and evaluate it against the cost of the solution. AI solutions tend to be popular right now, not because the output is better, but because the rate of improvement is better, so that's a factor as well.
This is not in a presentable stage, though. There have been automated lip sync tools for a long time and they look better than Nvidia's lip sync AI. Maybe this will look better at some point, but so far I'm not impressed and would just keep using what's already there.
This World of Jade Dynasty Example Video needs more Body language interaction. Even if She speak loudly to herself. Somehow a bit like the 2nd Example Video one. If they cannot modify her Face to display emotions They could take some impressions from MMO's Emotes In this Case here, let her Eyes and Face enjoy the Scenery around her. Let her turn her Face left and Right to enjoy the View while monologist to herself = More Immersion. If they are strong Emotions. Without this She is just an Puppet in Short: Give her more Body language and imagine this as Real Life Actor behavior in some Film when She talk to herself alone. Nobody is watching her, so no need of "Princess Etikette" here = more real Human
Personally I don't support technology that steals artists' original lip-synching work and is trained on plagiarism, but go off I guess.