It’s hardly surprising that a model optimized for replacing StackOverflow couldn’t survive in the untamed wilderness. As for writing a paper… you must’ve missed the fact that academia is currently in a crisis precisely because LLMs are better at writing papers than most students.
By the way, the paper the blog post you link to as a source links to as a source benchmarked LLMs on graph diagrams, textile patterns and 3D objects. It is not news that the language model would do poorly on visual-heavy tasks.
Sorry, I assumed you would have actually read the DELEGATE-52 study linked instead of just the abstract. For “a model optimized for replacing StackOverflow” that is “better at writing papers than most students” LLMs sure did pretty bad at those tasks over multiple rounds.
As the chart on page 7 of the paper shows, LLMs are good at exactly the kind of tasks you’d expect (producing and manipulating language), and bad at exactly the kind of tasks you’d expect (doing almost anything else). All this paper shows is that (1) they aren’t AGI, and (2) as a consequence of not being AGI they aren’t good unsupervised.
What the fuck? The only task that didn’t degrade across most models was Python. Very basic things like JSON, Makefiles, and schemas got screwed. Fiction, emails, and food menus got screwed. Did you even bother to read the legend? If you consider a single pass to be “producing and manipulating language” you didn’t bother to read the idiotic article you started this thread in support of. Good luck.
Edit: why do you lie?
Catastrophic corruption (80 and below) occurs in more than 80% of model, domain combinations.
The only task that didn’t degrade across most models was Python.
Yeah, after 20 cycles of unsupervised iteration on the task. Gemini 3.1 Pro doing as well as it did under that experiment setup is quite remarkable actually.
It’s hardly surprising that a model optimized for replacing StackOverflow couldn’t survive in the untamed wilderness. As for writing a paper… you must’ve missed the fact that academia is currently in a crisis precisely because LLMs are better at writing papers than most students.
By the way, the paper the blog post you link to as a source links to as a source benchmarked LLMs on graph diagrams, textile patterns and 3D objects. It is not news that the language model would do poorly on visual-heavy tasks.
Sorry, I assumed you would have actually read the DELEGATE-52 study linked instead of just the abstract. For “a model optimized for replacing StackOverflow” that is “better at writing papers than most students” LLMs sure did pretty bad at those tasks over multiple rounds.
As the chart on page 7 of the paper shows, LLMs are good at exactly the kind of tasks you’d expect (producing and manipulating language), and bad at exactly the kind of tasks you’d expect (doing almost anything else). All this paper shows is that (1) they aren’t AGI, and (2) as a consequence of not being AGI they aren’t good unsupervised.
Why do you lie like this?
What the fuck? The only task that didn’t degrade across most models was Python. Very basic things like JSON, Makefiles, and schemas got screwed. Fiction, emails, and food menus got screwed. Did you even bother to read the legend? If you consider a single pass to be “producing and manipulating language” you didn’t bother to read the idiotic article you started this thread in support of. Good luck.
Edit: why do you lie?
Yeah, after 20 cycles of unsupervised iteration on the task. Gemini 3.1 Pro doing as well as it did under that experiment setup is quite remarkable actually.
The paper does not show what you are arguing.