Rendered at 08:40:49 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
mike_hearn 12 minutes ago [-]
This is the sort of thing that the GPT-Live model would excel at when it launches. I'm not sure you'd need the ALE model at all if they launch Live in the API with acceptable pricing as it solves the latency issues with voice entirely. It's an impressive piece of engineering that hasn't been discussed much yet, I guess because it's only a ChatGPT feature and not accessible to developers. The model can dispatch tools asynchronously while speaking which would be ideal for a game.
What you'd want is maybe some kind of Live model with voice warping so it can be given different Skyrim themed 'Nordic' voices, and then custom tools to interact with the game engine.
usernomdeguerre 4 hours ago [-]
>The game runs on Windows, the audio processing and brain runs on my M4 MacBook. It could all run on Windows (provided there is dedicated ~12gb or more gpu ram for it)
I do wonder if this is an avenue for console gaming that might be practical in a few years; AI-centric hardware that might be too beefy or expensive for regular users, but can extend new or existing games. Kinda like the expansion paks of old.
unfortunate that the "ALE" design wasn't opensourced (couldn't find a link in their post) but I would be interested in learning more about the design, in particular what sort of data pipeline was necessary from skyrim to give this sort of action flexibility?
pantelisk 4 hours ago [-]
I will be open sourcing it soon :) my code is a bit dirty (the whole system is 3 pieces. The game adapter / a websocket bridge between game and brain / the brain itself) and it runs on 2 different (local) machines currently.
Ale is what makes this work locally, I felt a little conscious about it as I am not sure if it is a novel approach or somebody comes out and claims I rediscovered BERT or something (though ale runs at 1/10 the cost of BERT).
toilet 9 minutes ago [-]
Great concept and presentation. How much knowledge about the world does the agent have? Does Skyrim accurately model what their in game character can actually see and hear?
seabombs 4 hours ago [-]
I must say absolutely hilarious video. The "persona" of the dog is great. As someone who is generally pretty "keep your AI out of my art" this looks very fun to play. I can imagine this being cleverly integrated as a primary feature of a game (this must already be in the works). Ideally as a small model able to run locally alongside the game.
The moment where the dog is going on about "something foul in the air" as the player is attacked by a wolf ("F--- dude you could have warned me!") was great comedy.
MattDamonSpace 4 hours ago [-]
New mediums always absolutely knock the humanity organism for a bit
rao-v 5 hours ago [-]
I like the approach here. It’s cleverer than just “dump context to LLM”. I couldn’t quite figure out if it genuinely runs purely locally though.
If nothing else - this is how NPCs should work in games moving forward!
pantelisk 4 hours ago [-]
Hello, author here. I was intentionally a little vague about this because this is kind of the thing that makes the whole approach work.
An LLM predicts the next token. If you're trying to predict the next token in a mathematics competition, or while playing a deep strategy game, being a much larger and more capable model helps enormously. To predict that next token correctly, the model effectively needs to model a bunch of possible future states - even if that is a second order (unintended) effect, it is what is seems to be happening.
This is basically the Ilya (and Dario) argument that prediction, understanding, and compression are the same thing (deep rabbit hole) from a few years ago.
----
In my opinion; this is a beautiful idea, but videogames do not need most of that. Videogames (and games in general) shine when character behavior is predictable, and when NPCs are a little dumb (just a little).
We already have very good small roleplaying models — Qwen 3.5 4B/9B/30B-A3B. Nowhere near frontier models at general reasoning. But they can act and write in a very engaging way. Good at roleplaying but very weak at reasoning. They just need a little nudge at reasoning...
And that's the key. The player has already expressed their intent: attack that guy, go look over there, cover me, find the key that shines and is golden, etc. A constrained world, with a constrained set of actions. Instead of asking the model to reason over an enormous space of possible futures, we're mostly asking it to map: player intent + current world state → a small sequence of plausible actions.
As for the "dump context to an LLM". It's basically. "You are roleplaying as X - you experienced Y - you like/dislike (dispositions) Z, you remember Alpha, your journal says Delta. Player orders you to do Gamma. - "What do you respond and do?"
It kind of works (as you can see in the videos I posted). I am not going against the grain, big models are better, but do we need those models for everything?
alwaysanoobie 24 minutes ago [-]
What I don't understand is how are you passing the world state to the model?
Say for instance when you ask the dog to do an action when you launch an arrow upwards how is the LLM continuously tracking the state of the game to be able to respond?
oscarmcdougall 4 hours ago [-]
Love the concept of void mode. A companion that follows you across games, that's cool.
mihau 15 minutes ago [-]
I expected slop, but this is really good! (both demo and write-up)
I guess this means you can fus ro dah the LLM off a mountain, so there is that.
aussieguy1234 2 hours ago [-]
Next up: A smart glasses companion that follows you around in the game known as life.
Ive seen something like this in sci fi films.
dosenbrot 13 minutes ago [-]
Dennou Coil had this exact same concept.
Hilliard_Ohiooo 5 hours ago [-]
Video game seem like the perfect fit for llm use and I don't understand why it hasn't happened already.
goda90 4 hours ago [-]
It's expensive, early attempts didn't impress, and many gamers are hostile to the tech because it threatens the livelihood of game creators and impacts gaming hardware prices.
nottorp 1 hours ago [-]
It depends on if you want samey games or anything creative.
You could have complex realistic dialogue for innkeeper #28917 in the little hamlet of ass backwards, but:
One: do you really want that? There is such a thing as too much realism in a video game.
Two: do you really want all NPCs in all games to speak nigerian business english?
at the current token prices, far too expensive, but in medium/long term I think we will see a revolution in this field, the concepts are already there (and the expensive proofs of concept already work)
snowram 2 hours ago [-]
It did, look at the game Suck Up or a variety of Skyrim mods. Problem is gamers are hostile to any obvious trace of IA in their games. Also, consumer hardware is overall too weak to run a convincing llm model.
3 hours ago [-]
numpad0 2 hours ago [-]
Because games are entertainment, but outputs generated by current AI are rage inducing and felt hostile, not entertaining. It just isn't past the "make it work" stage.
pphysch 2 hours ago [-]
LLMs have only been really good for a couple years. You can bet that the remaining AAA studios are working hard on figuring this out in current unreleased products.
What you'd want is maybe some kind of Live model with voice warping so it can be given different Skyrim themed 'Nordic' voices, and then custom tools to interact with the game engine.
I do wonder if this is an avenue for console gaming that might be practical in a few years; AI-centric hardware that might be too beefy or expensive for regular users, but can extend new or existing games. Kinda like the expansion paks of old.
unfortunate that the "ALE" design wasn't opensourced (couldn't find a link in their post) but I would be interested in learning more about the design, in particular what sort of data pipeline was necessary from skyrim to give this sort of action flexibility?
Ale is what makes this work locally, I felt a little conscious about it as I am not sure if it is a novel approach or somebody comes out and claims I rediscovered BERT or something (though ale runs at 1/10 the cost of BERT).
The moment where the dog is going on about "something foul in the air" as the player is attacked by a wolf ("F--- dude you could have warned me!") was great comedy.
If nothing else - this is how NPCs should work in games moving forward!
An LLM predicts the next token. If you're trying to predict the next token in a mathematics competition, or while playing a deep strategy game, being a much larger and more capable model helps enormously. To predict that next token correctly, the model effectively needs to model a bunch of possible future states - even if that is a second order (unintended) effect, it is what is seems to be happening.
This is basically the Ilya (and Dario) argument that prediction, understanding, and compression are the same thing (deep rabbit hole) from a few years ago.
----
In my opinion; this is a beautiful idea, but videogames do not need most of that. Videogames (and games in general) shine when character behavior is predictable, and when NPCs are a little dumb (just a little).
We already have very good small roleplaying models — Qwen 3.5 4B/9B/30B-A3B. Nowhere near frontier models at general reasoning. But they can act and write in a very engaging way. Good at roleplaying but very weak at reasoning. They just need a little nudge at reasoning...
And that's the key. The player has already expressed their intent: attack that guy, go look over there, cover me, find the key that shines and is golden, etc. A constrained world, with a constrained set of actions. Instead of asking the model to reason over an enormous space of possible futures, we're mostly asking it to map: player intent + current world state → a small sequence of plausible actions.
As for the "dump context to an LLM". It's basically. "You are roleplaying as X - you experienced Y - you like/dislike (dispositions) Z, you remember Alpha, your journal says Delta. Player orders you to do Gamma. - "What do you respond and do?"
It kind of works (as you can see in the videos I posted). I am not going against the grain, big models are better, but do we need those models for everything?
Say for instance when you ask the dog to do an action when you launch an arrow upwards how is the LLM continuously tracking the state of the game to be able to respond?
tl;dr: let's you talk to all NPCs via LLM. Last time I tried it, latency was too much for it to be enjoyable, your approach seems to be much faster.
https://www.youtube.com/watch?v=yAEMVVtnUqM
Ive seen something like this in sci fi films.
You could have complex realistic dialogue for innkeeper #28917 in the little hamlet of ass backwards, but:
One: do you really want that? There is such a thing as too much realism in a video game.
Two: do you really want all NPCs in all games to speak nigerian business english?
Reminder for 2: https://www.theguardian.com/technology/2024/apr/16/techscape...