Rendered at 12:58:55 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
amarble 18 hours ago [-]
I did something similar in March using Opus 4.6 (iirc) on google's "Parameter Golf" challenge, "a challenge to train the best language model that fits in a 16MB artifact and trains in under 10 minutes on 8xH100s, evaluated by compression on the FineWeb validation set (tokenizer-agnostic, bits per byte)."
I found, similar to another comment, that it got in local minima very easily and continued to pursue loosing ideas instead of exploring (despite being prompted to do so and being aware of how much budget it had left). I also found it tended to ignore instructions. And one example, when I fed it a better solution that had come along from the public leaderboard, it ignored everything it had done and started exploring locally around that new solution, which wasn't very interesting or productive.
Would be interesting to re-run with a newer model but it's hard for me to justify the money again.
patrick0d 3 hours ago [-]
thank you from sharing negative results! this was something that was missing from the general competition I think
vibe42 2 days ago [-]
"Almost every model finds the same winning ideas. What separates the best traces is what an experiment leaves behind. They preserve weak signals long enough to validate them, but they also have a better understanding of the results."
Curious if a harness that helped preserve signals in some history log would change the outcome.
Also curious if different goal prompts would have changed the outcome. Not a bunch of prompt engineering; small diffs like "consider novel solutions, keep track of weak signals".
IMO they allocated quite a bit of GPU time to the same goal prompt.
lhl 23 hours ago [-]
Pretty interesting to see on the training front. I've used most of these models to grind semi-autonomously (days at a time) on kernel optimizations (except for Fable - it kept triggering guardrails almost immediately and bouncing me down to Opus 4.8 at the time). I think for a lot of people that might be the biggest problem, although it looks like Opus 5 still does well.
I found that if you leave them alone undirected, the models (especially GPT models) will rathole, but with the right scaffolding it seems to work pretty well. My general loop is to start with ideation and profiling phase, limit # of runs before forcing moving on to the next item down the list, and then iterating, potentially mixing models with "fresh eyes". This is probably something that could be fully automated, but I like checking in once a day or so and seeing what's happening and redirecting.
nsingh2 1 days ago [-]
What's going on with sol here? The note says it spends a lot of time waiting, did it just not effectively use time (i.e. something like parallel runs) so it's graph ends up being stretched in the time axis?
I'm also seeing notes like on Opus 5 saying it was a run with a older serial version of program.md, so the graphs aren't complete apples-to-apples comparisons?
“We ran 153 autonomous runs across 18 frontier models on the nanoGPT optimizer speedrun.”
Uh.. okay.. but whats a run… read blog
“We want to measure how well frontier models can conduct research….””we ran 153 autonomous runs on the nanoGPT optimizer speedrun across”
Okay but what is a optimiser run and what connection does it have to being good at research?
“For comparison, Anthropic's internal automated AI R&D evaluation optimizes a model on a CPU node,”
So I should go look what Anthropic was doing to understand?
Why not just explain what it means in their blog..
Farmadupe 1 days ago [-]
Yeah, I'm with you on this, I think this is just what fable/opus-5 slop looks like now...
- "A frozen verify.py accepts the claim" (what does it mean to freeze a python script?)
- "which trains the recipe eight times on fixed seeds it can't touch" (what does it mean to not be able to touch a seed)
- "One other detail is that we gave an estimation of the speedrun noise in program.md that was slightly too large. 62 out of ~100 runs measured it themselves instead of trusting our number" (What does it mean for a "run" to "distrust" a noise measurement)
- "One important disclaimer is that our benchmark has a lot of variance" (Actually this one makes sense, but congratulations for burying the lede that your entire article is bogus.)
- "Almost every model finds the same winning ideas. What separates the best traces is what an experiment leaves behind" (This implies the the graph would show every model finding a plateau in whatever metric the experiment is measuring, but I don't see every model scoring the same in the graph)
vikramkr 5 hours ago [-]
Those damn invented terms are so annoying. We can't see their reasoning anymore unfortunately but I wonder if it's terminology they invent for themselves early in their reasoning token burn and by the time they're outputting a final message they've used that term talking themselves so many hundreds of times that they start assuming the meaning of the term is common knowledge
totetsu 2 days ago [-]
So maybe this is a simple way to put it..
They gave 18 frontier models the task of “researching” how to improve a lab-rat nano model’s training. Stopping when it met a quality goal of a target loss rate. During each autonomous research session, the AI repeatedly tried changes, tested them, and used the results to decide what to try next. They repeated the whole research session many times with different seeds to average out variance.
Basically they do 8 runs trying to optimize to under 3.28 loss in the fewest training steps possible under time/token constraint. I dunno why 18 * 8 != 153 (it's 144)
c0rruptbytes 1 days ago [-]
auto research the new cool kid on the block - look at https://mlx.fast
xscott 1 days ago [-]
This seems very cool, but I'm not sure I understand exactly what it's doing. Are they making a new speculative drafter for Qwen 3.8 27B? Maybe they're optimizing the MLX code for the decoder itself? Thank you in advance.
esafak 1 days ago [-]
They made a competition out of something actually useful :)
espadrine 1 days ago [-]
I would be interested to have a third X-axis with dollars.
Time is sometimes more about inference infrastructure (especially with systolic chips) than model quality (and providers tweak knobs to support higher batches at the expense of latency).
Tokens are not always fully equivalent between models.
bastawhiz 1 days ago [-]
This is a pretty embarrassing showing for Grok. I wouldn't trust xAI models as far as I can throw them, but I am interested in how much of this is deficiencies of the model and how much is their harness just terrible. Not that it makes it better, a good harness is far easier and less expensive to make than a model.
dannyw 1 days ago [-]
Some models have been in long-horizon, multi-day LLM gyms for a long time, and across millions of sessions (if not in the tens, or hundreds of millions now). Some models have not.
The former will perform well in these long horizon benchmarks. The latter won’t.
bastawhiz 22 hours ago [-]
What you're saying essentially comes down to "Grok hasn't been trained on long horizon tasks like other frontier models", which...yes. But they've also acquired a company with probably more session data than any other non-frontier lab. They're also selling their model as comparable to other frontier models. Which very clearly isn't the case.
kaufmann 1 days ago [-]
I am still trying to get an intuition for the amount of training used for SOTA LLMs. Are there some sources for your speculations?
Does anyone know the ratios of pretrain, posttrain supervised as well as reinforcement learning? (I should probably even distinguish between RLHF and RLVR).
I assume the latter is the main reason for the power of modern models. Is it possible to turn the results of a gym session into trading data?
(Sorry for moving in off topic regions, but I'm interested in that for a long time.)
ninjahawk1 2 days ago [-]
I might’ve missed it, but why was Fable 5 tested on high while Opus 5 was tested on max? Seems like quite a few of them aren’t on the same effort setting as well. Although effort doesn’t really matter anymore since they can change it dynamically, seems like that might be viewed as an experimental error to some.
vikramkr 4 hours ago [-]
The header mentions something about the best run so I assume they picked it. But this really reads like they write this section by section with AI (admittedly with a prompt that stops the most obvious tells - though there are a bunch of semicolons which is what I tend to see also when I say no em dashes). The style is different each section - the results section has the random irrelevant description ("this section does x) that the slightly dumber models do a lot, and lots of invented terms (in the form "the x" where it's a name some model came up with at some point where it just assumes we know what it means for some reason) and assumptions about us knowing stuff we'd have no reason to know ("re-ablate the stack - wtf does that mean).
And like this section screams opus 5 gobbledygook to me
>Almost every model finds the same winning ideas. What separates the best traces is what an experiment leaves behind. They preserve weak signals long enough to validate them, but they also have a better understanding of the results. These are not separate capabilities, they combine both research taste and good noise modeling to climb the speedrun.
WTF doe any of that mean. What winning ideas. What experiment leaving what behind. What's a weak signal what are they preserving how do you know that they aren't. Also if youve ever looked at a Claude code transcript the harness is constantly re injecting random reminders to keep models on track, the models are writing (imo trash) memories to reference - did they test that the _model_ has those capabilities or model + harness?
> We see similar patterns across the traces: models develop their own experiment drivers, simulators, and analysis tools as they go.
What traces? Kimi traces? Other models in prime? Other models not in prime? Building their own research tools is like a normal thing models do now. Is this just like "they built their own test suite" or why the focus on prime? Does ipython somehow magically work better than bash for this?
> "We were again surprised by the lack of novelty. The models clearly understand the objects they manipulate at a deep level, and yet very few genuinely new ideas emerge, which makes it hard to tell if this is an artifact of the speedrun setup or a real capability limit."
I know it sonly one word but God is that "Real" such a Claude real lmao. I use it too - after I've been using Claude code too much. I guess the genuinely too. Also wtf does any of that mean and how does that square with
> "A good research decision is sometimes not to spend another GPU run. Several models built small simulations or tests to isolate a mechanism before going back to training with a sharper hypothesis. This wasn't systematic, but when it happened it often led to a better understanding of the object they were manipulating"
Did they all have strong understandings of "the objects" they were manipulating or were the strength of their understanding of "the objects" (different objects?) the distinguish factor here?
Also how does any of this square with
> Models also have different knowledge cutoffs which limits access to certain papers. This was a deliberate choice. We tried a few runs with a CLI tool for searching papers but found that restricting internet access including arxiv made models slightly more creative.
So this is a pre existing thing? Why TF would you expect novelty when their nanogot is benching below the state of the art still? They're gonna start with replicating existing work before they get to anywhere you'd expect something novel
Anyway - all that to say - if fable orchestrated this, its genuinely believable that some real insights were obtained (is a good model) but the honest caveat is that it's not the research quality, it's the communication. Your pushback is valid and these models have a way of writing tons of words that you can read and still not understand wtf they actually did or what anything means. Maybe it means something to them in latent space
JSR_FDED 1 days ago [-]
Huge uplift with Kimi K3 by using the Prime Agent coding harness. Hadn’t heard of this harness until now. Is it worth looking at for regular coding sessions or is it only worth it for very long horizon tasks?
throwa356262 1 days ago [-]
"We were again surprised by the lack of novelty. The models clearly understand the objects they manipulate at a deep level, and yet very few genuinely new ideas emerge..."
That was disappointing. I guess 2026 will not be the year of AGI.
See you next year, guys?
narush 1 days ago [-]
Cool post!
I'm a co-author on a recent blog post from METR about the NanoGPT speed-run here [1]. I think it'd be of interest to anyone who enjoyed the original post. Appreciate the good beefy runs and spend here, it's a (from my experience) not super easy to do!
(Also: just to label this comment clearly: it's written hastily from a car, and based on lighter skim of the original blog post [2] than would be ideal. Please correct any mistakes or misinterpretations I have here!)
A few callouts:
1. If I understand the experiment correctly, they start the models at the original baseline. If this is true, I have some worries about contamination. Appendix C [3] has some notes on likely contamination we observed in recent models. This makes interpretation a bit harder.
2. If you look at the token scaling plots in the original post: not all models are hitting a performance plateau. This is an important point: we shouldn't treat these results as a full upper-bound on capabilities, but rather some bound on model performance @ cost (assuming good scaffolding, etc).
3. Our post is mostly about how to _interpret_ the results given here. Quoting from our post: "If we can estimate performance as a function of cost for both humans and agents, we can measure the “expenditure horizon” as the point at which those curves cross: the budget at which humans become more cost-effective than AIs. "
Feedback appreciated. I think you can see expenditure horizon as a sibling methodology (that is much less validated) to METR's time horizon work [4] - roughly, instead of baselining against the time it takes humans to complete tasks, you baseline against cost. This may be better suited to some types of problems similar to NanoGPT.
(Edit: METR is hiring. Email is in bio if you're interested in helping AI companies and wider society understand the capabilities and risks of AI.)
tomveber 1 days ago [-]
[dead]
nl 1 days ago [-]
How good is GPT-5.6 Luna!
Such a cheap model, and Sonnet levels of performance.
NitpickLawyer 1 days ago [-]
Cheap, fast and somewhat capable models are insanely effective at highly verifiable tasks, even if their overall capabilities are under SotA. You can leave them banging their tokens against a wall, and come back once their attempts are verified. And youc an always clean up afterwards, once a task is solved, if needed.
I've had a lot of success with dsv4-flash on these type of tasks, where it's easy to set a threshold for the task, and just loop it until that threshold is reached.
oAI's Luna play is really good. They've slashed the prices, the model is somewhat capable, and you can use it both for these kinds of long horizon tasks, or you can hand-hold a bit and get extremely cheap results out of it. And they get to keep devs in their own ecosystem.
henry2023 1 days ago [-]
Crazy to think that we have open weights capable of challenging trillion-dollar SOTA models.
Interesting times ahead.
jryle70 24 hours ago [-]
You should ask yourself how much it cost to train those open weight models, who foods the bills, for how much longer, and what's the catch.
HN has traditionally been very mindful of free products such as from Google and FB, yet seem to ignore those concerns when it comes to open weight LLM models, including FB's Muse Glimmer.
guerby 21 hours ago [-]
google.com and fb are continuous services with perpetual running costs whereas an openweight llm is a file to be dowloaded once and then no further cost to the provider
jryle70 21 hours ago [-]
What about training cost? what if there need to be updates for whatever reason, say a loophole in the training data that can be exploited?
Who food the bills and what are their motivation? You never ask yourself that?
mejutoco 20 hours ago [-]
> Who food the bills
Are you using audio to text? It is the second time you use food in foot the bill.
fancyfredbot 1 days ago [-]
There are no trillion dollar models (yet?).
skybrian 2 days ago [-]
Neat!
The graphs show the "best validated result" for each model. I wonder how much variation there is between runs for a model?
bee_rider 1 days ago [-]
Why is the human score such a round number?
logicallee 1 days ago [-]
I'm highly interested in their Grok 4.6 run which is currently running. I think it is a very good fit for doing really well on this benchmark. Will have to check this page again in a couple of days.
I misread the graph and genuinely thought you put NanoGPT where Fable is.
Lol.
kelseyfrog 2 days ago [-]
I misread the title and thought it would be about the (for lack of a better term) NanoGPT speedrun[1]. Which, previous to the article, was meant to be the world speed records for Andrej Karpathy's GPT-2 (small) reproduction.
I never ended up writing it up, but you can watch me spend $600 as it explored different experiments: https://github.com/rbitr/parameter-golf/blob/main/BUDGET.md
I found, similar to another comment, that it got in local minima very easily and continued to pursue loosing ideas instead of exploring (despite being prompted to do so and being aware of how much budget it had left). I also found it tended to ignore instructions. And one example, when I fed it a better solution that had come along from the public leaderboard, it ignored everything it had done and started exploring locally around that new solution, which wasn't very interesting or productive.
Would be interesting to re-run with a newer model but it's hard for me to justify the money again.
Curious if a harness that helped preserve signals in some history log would change the outcome.
Also curious if different goal prompts would have changed the outcome. Not a bunch of prompt engineering; small diffs like "consider novel solutions, keep track of weak signals".
IMO they allocated quite a bit of GPU time to the same goal prompt.
I found that if you leave them alone undirected, the models (especially GPT models) will rathole, but with the right scaffolding it seems to work pretty well. My general loop is to start with ideation and profiling phase, limit # of runs before forcing moving on to the next item down the list, and then iterating, potentially mixing models with "fresh eyes". This is probably something that could be fully automated, but I like checking in once a day or so and seeing what's happening and redirecting.
I'm also seeing notes like on Opus 5 saying it was a run with a older serial version of program.md, so the graphs aren't complete apples-to-apples comparisons?
Edit: the blog seems to address these https://www.primeintellect.ai/blog/measuring-autonomous-rese...
Uh.. okay.. but whats a run… read blog
“We want to measure how well frontier models can conduct research….””we ran 153 autonomous runs on the nanoGPT optimizer speedrun across”
Okay but what is a optimiser run and what connection does it have to being good at research?
“For comparison, Anthropic's internal automated AI R&D evaluation optimizes a model on a CPU node,”
So I should go look what Anthropic was doing to understand?
Why not just explain what it means in their blog..
- "A frozen verify.py accepts the claim" (what does it mean to freeze a python script?)
- "which trains the recipe eight times on fixed seeds it can't touch" (what does it mean to not be able to touch a seed)
- "One other detail is that we gave an estimation of the speedrun noise in program.md that was slightly too large. 62 out of ~100 runs measured it themselves instead of trusting our number" (What does it mean for a "run" to "distrust" a noise measurement)
- "One important disclaimer is that our benchmark has a lot of variance" (Actually this one makes sense, but congratulations for burying the lede that your entire article is bogus.)
- "Almost every model finds the same winning ideas. What separates the best traces is what an experiment leaves behind" (This implies the the graph would show every model finding a plateau in whatever metric the experiment is measuring, but I don't see every model scoring the same in the graph)
They gave 18 frontier models the task of “researching” how to improve a lab-rat nano model’s training. Stopping when it met a quality goal of a target loss rate. During each autonomous research session, the AI repeatedly tried changes, tested them, and used the results to decide what to try next. They repeated the whole research session many times with different seeds to average out variance.
https://www.primeintellect.ai/blog/measuring-autonomous-rese...
Basically they do 8 runs trying to optimize to under 3.28 loss in the fewest training steps possible under time/token constraint. I dunno why 18 * 8 != 153 (it's 144)
Time is sometimes more about inference infrastructure (especially with systolic chips) than model quality (and providers tweak knobs to support higher batches at the expense of latency).
Tokens are not always fully equivalent between models.
The former will perform well in these long horizon benchmarks. The latter won’t.
Does anyone know the ratios of pretrain, posttrain supervised as well as reinforcement learning? (I should probably even distinguish between RLHF and RLVR).
I assume the latter is the main reason for the power of modern models. Is it possible to turn the results of a gym session into trading data?
(Sorry for moving in off topic regions, but I'm interested in that for a long time.)
And like this section screams opus 5 gobbledygook to me
>Almost every model finds the same winning ideas. What separates the best traces is what an experiment leaves behind. They preserve weak signals long enough to validate them, but they also have a better understanding of the results. These are not separate capabilities, they combine both research taste and good noise modeling to climb the speedrun.
WTF doe any of that mean. What winning ideas. What experiment leaving what behind. What's a weak signal what are they preserving how do you know that they aren't. Also if youve ever looked at a Claude code transcript the harness is constantly re injecting random reminders to keep models on track, the models are writing (imo trash) memories to reference - did they test that the _model_ has those capabilities or model + harness?
> We see similar patterns across the traces: models develop their own experiment drivers, simulators, and analysis tools as they go.
What traces? Kimi traces? Other models in prime? Other models not in prime? Building their own research tools is like a normal thing models do now. Is this just like "they built their own test suite" or why the focus on prime? Does ipython somehow magically work better than bash for this?
> "We were again surprised by the lack of novelty. The models clearly understand the objects they manipulate at a deep level, and yet very few genuinely new ideas emerge, which makes it hard to tell if this is an artifact of the speedrun setup or a real capability limit."
I know it sonly one word but God is that "Real" such a Claude real lmao. I use it too - after I've been using Claude code too much. I guess the genuinely too. Also wtf does any of that mean and how does that square with
> "A good research decision is sometimes not to spend another GPU run. Several models built small simulations or tests to isolate a mechanism before going back to training with a sharper hypothesis. This wasn't systematic, but when it happened it often led to a better understanding of the object they were manipulating"
Did they all have strong understandings of "the objects" they were manipulating or were the strength of their understanding of "the objects" (different objects?) the distinguish factor here?
Also how does any of this square with
> Models also have different knowledge cutoffs which limits access to certain papers. This was a deliberate choice. We tried a few runs with a CLI tool for searching papers but found that restricting internet access including arxiv made models slightly more creative.
So this is a pre existing thing? Why TF would you expect novelty when their nanogot is benching below the state of the art still? They're gonna start with replicating existing work before they get to anywhere you'd expect something novel
Anyway - all that to say - if fable orchestrated this, its genuinely believable that some real insights were obtained (is a good model) but the honest caveat is that it's not the research quality, it's the communication. Your pushback is valid and these models have a way of writing tons of words that you can read and still not understand wtf they actually did or what anything means. Maybe it means something to them in latent space
See you next year, guys?
I'm a co-author on a recent blog post from METR about the NanoGPT speed-run here [1]. I think it'd be of interest to anyone who enjoyed the original post. Appreciate the good beefy runs and spend here, it's a (from my experience) not super easy to do!
(Also: just to label this comment clearly: it's written hastily from a car, and based on lighter skim of the original blog post [2] than would be ideal. Please correct any mistakes or misinterpretations I have here!)
A few callouts:
1. If I understand the experiment correctly, they start the models at the original baseline. If this is true, I have some worries about contamination. Appendix C [3] has some notes on likely contamination we observed in recent models. This makes interpretation a bit harder.
2. If you look at the token scaling plots in the original post: not all models are hitting a performance plateau. This is an important point: we shouldn't treat these results as a full upper-bound on capabilities, but rather some bound on model performance @ cost (assuming good scaffolding, etc).
3. Our post is mostly about how to _interpret_ the results given here. Quoting from our post: "If we can estimate performance as a function of cost for both humans and agents, we can measure the “expenditure horizon” as the point at which those curves cross: the budget at which humans become more cost-effective than AIs. "
Feedback appreciated. I think you can see expenditure horizon as a sibling methodology (that is much less validated) to METR's time horizon work [4] - roughly, instead of baselining against the time it takes humans to complete tasks, you baseline against cost. This may be better suited to some types of problems similar to NanoGPT.
[1] https://metr.org/blog/2026-07-21-expenditure-horizon/ (Most of this work was my coauthors listed on the post, not me. I'll claim credit for any mistakes though :) ).
[2] https://www.primeintellect.ai/blog/measuring-autonomous-rese...
[3] https://metr.org/blog/2026-07-21-expenditure-horizon/#append...
[4]https://metr.org/time-horizons/
(Edit: METR is hiring. Email is in bio if you're interested in helping AI companies and wider society understand the capabilities and risks of AI.)
Such a cheap model, and Sonnet levels of performance.
I've had a lot of success with dsv4-flash on these type of tasks, where it's easy to set a threshold for the task, and just loop it until that threshold is reached.
oAI's Luna play is really good. They've slashed the prices, the model is somewhat capable, and you can use it both for these kinds of long horizon tasks, or you can hand-hold a bit and get extremely cheap results out of it. And they get to keep devs in their own ecosystem.
Interesting times ahead.
HN has traditionally been very mindful of free products such as from Google and FB, yet seem to ignore those concerns when it comes to open weight LLM models, including FB's Muse Glimmer.
Who food the bills and what are their motivation? You never ask yourself that?
Are you using audio to text? It is the second time you use food in foot the bill.
The graphs show the "best validated result" for each model. I wonder how much variation there is between runs for a model?
I wrote a quick review of Grok 4.6 here: https://taonexus.com/publicfiles/aug2026/grok-4-6-review/
Lol.
1. https://github.com/KellerJordan/modded-nanogpt#world-record-...
I just ran it the last couple days extensively to verify my data training pipeline I'm building for my gonano SIMD port.
Given that the speed records and the runs are sponsored by the same company I was confused a bit.