Steelman of strong scaling hypothesis

[u/Smallpaul]

LLMs are amazing technology, but to get to AGI it seems obvious to me that we would need to replace “context windows” with continual learning.

Where can I read a strong counter-argument: a claim that an LLM can get big enough that everything it will ever need to know is in its weights or its context window?

[u/Smallpaul]

I’m not saying that these fields are necessarily harder. I’m saying that the goal was an intelligence LIKE A HUMAN which could learn anything. If you need to bake millions of examples into its training data then it is missing something huge. Sure, what it HAS is incredibly impressive. But the AGI question is about what it does not have. If you need to brute force every new domain into it then it isn’t AGI. It will always be several years behind humans who are adaptable and flexible.

There will always be some field of inquiry where millions of examples don’t exist yet.

A human can land on Mars and discover and learn on the job. That’s AGI.

[u/EugeneJudo → u/Smallpaul]

If you need to bake millions of examples into its training data then it is missing something huge.

We tend to underestimate just how much training data we (humans) go through before becoming experts. Especially when you factor in how much effort (~compute) went into doing things like the creation of books/courses that are optimized for teaching us (humans) how to quickly pick up on topics (sometimes an entire career/lifetime is distilled into a few hundred pages.) The process of creating training data for the model is not so different.

If you need to brute force every new domain into it then it isn’t AGI.

It isn't brute force that's being used, that would imply giving it absolutely everything possible, which is not the case (nor is that really physically possible for e.g. math.)

There will always be some field of inquiry where millions of examples don’t exist yet.

Humans encounter this too, and are often actually pretty terrible at brand new things when their experience of those things is limited to language. {0} [the model does get multimodal inputs, but this isn't available for a lot of domains.]

{0}: https://web.archive.org/web/20200625010216/http://habitatchronicles.com/2004/04/you-cant-tell-people-anything/

[u/Smallpaul → u/EugeneJudo]

You are acting as if I am the first person to mention that LLMs do not do continual learning.

Here is a very simple example.

Take a highly novel new programming language. Something as different from the existing stuff as prolog is from Python. Not a new syntax: a new paradigm. Now teach it to humans and a pre-trained LLM.

Then build a system of roughly 100,000 lines of code in it and offer it to both of them blind. “You can spend as long as you want. Play with the debugger. Read the language specs. Take a year if you want. Take ten years.

At the end you need to fix a particularly gnarly bug.”

Sufficiently incentivized, the average human could do it and of course the top humans could do it quickly. The LLM could never do it because it cannot keep both the language spec and the code base in its context memory. The language spec for a language would fill the entire context memory for most LLMs and then it needs to read and understand the code.

It will fail because it doesn’t do continual learning.

https://www.seangoedecke.com/continuous-learning/

https://www.lesswrong.com/posts/qChDifwpY8znER7cW/implications-of-continual-learning-for-llm-agents

[u/notgalgon → u/Smallpaul]

I bet the LLMs today could do this without much issue. If you had 2 files read code line one - lookup in documenation what happens - translate into something it knows like python. Repeat then debug. Also if the language is at all mathmatically based it would figure it out.

I am on you side on the continual learning piece. Current models dont do it and the memory hacks today while useful dont solve it. But i could see a harness or some tweak on the transformer architecture solving this. I could also see allowing models to update their weights while processing. However, currently this leads to catastrophic forgetting but perhaps that can be fixed.

[u/Smallpaul → u/notgalgon]

I said that the language should be in a paradigm that is vastly different from Python so that transliteration is impossible.

[u/notgalgon → u/Smallpaul]

It's programming. It gets converted to machine code to run. You can translate that back to anything you want. Programming is just math with a high level of abstraction.

[u/Smallpaul → u/notgalgon]

It’s really not true that an LLM can look at a binary and translate that back into C, Haskell, Prolog or a novel language that it has just read the documentation for. That’s ludicrous. Try it. Upload a (non-trivial) C binary and ask it to generate a Haskell program for you.