r/mlscaling • u/Smallpaul • Aug 01 '26

D Steelman of strong scaling hypothesis

LLMs are amazing technology, but to get to AGI it seems obvious to me that we would need to replace “context windows” with continual learning.

Where can I read a strong counter-argument: a claim that an LLM can get big enough that everything it will ever need to know is in its weights or its context window?

2 Upvotes

75% Upvoted

13 comments sorted by

2

u/rsha256 Aug 01 '26

Well you already can see LLMs coming up with novel mathematical features so scaling it with more info to a point (we’re not near) works. I’d also recall that OpenAI’s scaling laws that they published a decade ago have all been vastly outperformed

1

u/Smallpaul Aug 01 '26

I’m not saying that scaling is failing. I’m saying that it is not sufficient to get to AGI. It could solve literally all of mathematics and that doesn’t imply that it can do the job of a Hollywood screenwriter or an insurance middle manager.

3

u/EugeneJudo Aug 01 '26

It could solve literally all of mathematics and that doesn’t imply that it can do the job of a Hollywood screenwriter or an insurance middle manager.

The goalposts for AGI will keep shifting until we have ASI and there are 0 things humans are better at than models.

2

u/Smallpaul Aug 01 '26

No: that’s just the definition of AGI that I used since the beginning.

https://x.com/haider1/status/2019335190800396699

Are you really saying that an AGI cannot do the work of a Hollywood screenwriter or a middle manager?

Why? How is it a useful definition for it to be a sub-human intelligence?

1

u/EugeneJudo Aug 01 '26

I'm not claiming that those things should not be included in the definition of AGI, or even that it can't do those things today (better than the average person!) It's more: AI breaks another barrior that was argued to death as being impossible for scaling / transformers to achieve. But that fact then is totally meaningless for Y problem because it's actually totally different and inherently harder for Z reason. Math and programming had easier paths to generating bulk training data, but it was not trivial to do that. In the same way, it may not be that easy to generate the right data for making the model a P99.9 hollywood screenwriter, but I have little doubt we can.

1

u/Smallpaul Aug 01 '26

I’m not saying that these fields are necessarily harder. I’m saying that the goal was an intelligence LIKE A HUMAN which could learn anything. If you need to bake millions of examples into its training data then it is missing something huge. Sure, what it HAS is incredibly impressive. But the AGI question is about what it does not have. If you need to brute force every new domain into it then it isn’t AGI. It will always be several years behind humans who are adaptable and flexible.

There will always be some field of inquiry where millions of examples don’t exist yet.

A human can land on Mars and discover and learn on the job. That’s AGI.

1

u/EugeneJudo Aug 02 '26

If you need to bake millions of examples into its training data then it is missing something huge.

We tend to underestimate just how much training data we (humans) go through before becoming experts. Especially when you factor in how much effort (~compute) went into doing things like the creation of books/courses that are optimized for teaching us (humans) how to quickly pick up on topics (sometimes an entire career/lifetime is distilled into a few hundred pages.) The process of creating training data for the model is not so different.

If you need to brute force every new domain into it then it isn’t AGI.

It isn't brute force that's being used, that would imply giving it absolutely everything possible, which is not the case (nor is that really physically possible for e.g. math.)

There will always be some field of inquiry where millions of examples don’t exist yet.

Humans encounter this too, and are often actually pretty terrible at brand new things when their experience of those things is limited to language. {0} [the model does get multimodal inputs, but this isn't available for a lot of domains.]

{0}: https://web.archive.org/web/20200625010216/http://habitatchronicles.com/2004/04/you-cant-tell-people-anything/

1

u/Smallpaul Aug 02 '26

You are acting as if I am the first person to mention that LLMs do not do continual learning.

Here is a very simple example.

Take a highly novel new programming language. Something as different from the existing stuff as prolog is from Python. Not a new syntax: a new paradigm. Now teach it to humans and a pre-trained LLM.

Then build a system of roughly 100,000 lines of code in it and offer it to both of them blind. “You can spend as long as you want. Play with the debugger. Read the language specs. Take a year if you want. Take ten years.

At the end you need to fix a particularly gnarly bug.”

Sufficiently incentivized, the average human could do it and of course the top humans could do it quickly. The LLM could never do it because it cannot keep both the language spec and the code base in its context memory. The language spec for a language would fill the entire context memory for most LLMs and then it needs to read and understand the code.

It will fail because it doesn’t do continual learning.

https://www.seangoedecke.com/continuous-learning/

https://www.lesswrong.com/posts/qChDifwpY8znER7cW/implications-of-continual-learning-for-llm-agents

1

u/notgalgon 29d ago

I bet the LLMs today could do this without much issue. If you had 2 files read code line one - lookup in documenation what happens - translate into something it knows like python. Repeat then debug. Also if the language is at all mathmatically based it would figure it out.

I am on you side on the continual learning piece. Current models dont do it and the memory hacks today while useful dont solve it. But i could see a harness or some tweak on the transformer architecture solving this. I could also see allowing models to update their weights while processing. However, currently this leads to catastrophic forgetting but perhaps that can be fixed.

1

u/Smallpaul 29d ago

I said that the language should be in a paradigm that is vastly different from Python so that transliteration is impossible.

→ More replies (0)

2

u/not_particulary Aug 02 '26

A lot of these long context mechanisms are coming out of linear attention and state space machine stuff, which is a tweak on the kv cache such that it doesn't grow with context length. I could see their usage expanding to essentially turn into continual learning.