understanding technical roadblocks to agi

Lately, I've been quite interested in the underlying architecture of how most LLM-based systems work.

As I explore this, the topic of Artificial General Intelligence is becoming more interesting, less from the "impact on the economy" angle and more from a "what are the technical roadblocks that prevent us from getting to AGI" angle. I'm finding it a very helpful thought experiment to deepen my understanding of the different layers of the AI stack.

By this definition, if we had AGI, it should be able to autonomously execute on the following scenarios without human intervention.

"Develop a drug to solve osteosarcoma"

"Run this startup and make $1M in MRR"

"Perform this neurosurgery"

They might sound a bit silly, but I think it serves as an interesting thought experiment on what would need to happen for AI to be able to perform these tasks without humans involved.

I took a first pass at trying to answer this question with the following take (which I later understood was not quite accurate):

My initial take was that the core technical capabilities of AGI already exist, the issue is more so an engineering and harness problem. Kind of like quantum computing, where the underlying physics is largely understood but engineering it reliably at scale remains the hard part.

Specifically, I thought if we threw unlimited tokens at a goal and gave the AI all the "context required" it should be able to exhibit AGI-like attributes.

However, this only works for a narrower definition of AGI in more deterministic domains. Then it really isn't AGI.

Back to the question, what needs to happen?

Continual Learning

When human beings learn something hard, they don't just collect "context" or memories. They build understanding. The brain almost reorganizes itself in the process of developing competence. In more technical terms, synaptic connections strengthen and weaken as a person develops an internal model of a field.

LLMs learn during training, and the weights basically freeze. This means every new conversation or new fact doesn't impact the model's weights after training.

Well, what about RAG? (Retrieval-augmented generation: where it can source documents from that domain and give an answer) Can't the core intelligence model use the information received from RAG to think through important problems?

Turns out, not really.

The reason RAG does not solve this is that access to knowledge is not the same thing as acquired competence.

A helpful analogy is neurosurgery. If you gave a normal person every neurosurgery textbook ever written, they might learn the terminology, understand the broad concepts, and even answer some exam-style questions. But they would not be able to perform surgery tomorrow. The real bottleneck is how information is translated to internal skill.

A neurosurgeon is not just someone with more facts in memory. They have developed a compressed, experience-shaped model of anatomy, pathology, tissue behavior, motor control, risk, and real-time decision-making. They have seen enough cases to know which details matter, when a plan is failing, when anatomy is distorted, when to continue, and when to stop. That expertise is built through years of supervised feedback loops, not just reading.

This is the gap between RAG and continual learning.

RAG gives the model an external library. It can retrieve relevant passages and use them inside the current context window. That makes the model more informed in the moment, but it does not necessarily make the model more capable after the moment. Once the context disappears, the model's underlying weights have not changed. The information was available, but not internalized.

Fine-tuning gets closer, but even fine-tuning on textbooks mostly teaches the model to predict domain-specific text. It may become better at explaining procedures, using terminology, summarizing papers, and imitating clinical reasoning. But that is still different from learning the underlying skill. Real expertise requires interaction with the world: attempts, errors, corrections, consequences, and feedback. In surgery, the feedback signal is not "did you predict the next word correctly?" It is "did the patient outcome improve, did you preserve function, did you avoid unnecessary damage, did you respond correctly when reality deviated from the plan?"

That is why continual learning is hard. The challenge is not merely letting the model update its weights. The challenge is knowing what should cause an update, which parts of the model should change, how much they should change, and how to prevent the new learning from corrupting older capabilities.

Humans solve this through a messy but powerful combination of memory, attention, embodiment, reward, emotion, repetition, sleep, supervision, and plasticity. Current LLMs mostly do not. They can use new information in context, but they do not reliably convert lived experience into durable, generalizable competence.

Long-Horizon Tasks

What current systems are good at are individual steps but fall apart as the time horizon of the task gets longer. Every year, this time horizon continues to increase, but the question is whether it would be able to operate over months or years.

Fundamentally, LLMs work on the principle of "Given everything I've seen so far, what should the next token be?"

It doesn't really hold a persistent state of memory or plans. For example, agentic coding tools that keep working until a goal is met still follow the same principle. The longer they work, the longer they hold memory / context.

Perhaps there are clever ways to engineer the system so you compact context or stage gate access to certain context. However, I doubt we're able to push the engineering enough for it to last beyond days, maybe max a month.

Even if it's able to keep memory, the other problem is knowing when to autonomously course correct and rethink strategy as it's executing on the goal. Humans aren't perfect at this either, but current AI systems are considerably more brittle when the environment shifts mid-task.

Reliability

When you convert text into an embedding, you're mapping it to a point in a very high dimensional space. A typical embedding model produces vectors anywhere from 768 to 3000+ dimensions, depending on the model. Each dimension loosely corresponds to some learned feature of meaning.

The standard way to measure closeness in this space isn't Euclidean distance (straight line between two points). It's cosine similarity, the angle between two vectors.

Why angle and not distance? Because you care about direction, not magnitude. A short document and a long document about the same topic should be "close" even if their vectors have very different lengths.

Now here's the crowding problem as we increase context or horizon time.

In low dimensions, similarity is meaningful. In 2D or 3D, if two points are close, they're genuinely similar. There's enough space for things to spread out. In high dimensions, something strange happens. This is called the curse of dimensionality.

As you add dimensions, the volume of the space grows exponentially. But your data points don't grow at the same rate. So everything ends up roughly the same distance from everything else under Euclidean distance. Cosine similarity handles this better by normalizing for magnitude, but it has its own limits: the embedding model itself has finite resolution, meaning it can only represent so much semantic nuance before different concepts start bleeding together in the same region of the space.

This creates problems for RAG/context systems because when asked "which stored vectors are closest to this query vector?", the signal gets noisy:

Many semantically unrelated chunks end up with nearly the same cosine similarity to your query. The ranking between "relevant" and "somewhat relevant" becomes unreliable.

Can These Be Solved?

I like how Elon thinks about most problems, and it's quite simple: if it doesn't break the laws of physics, it's possible.

The harder question is: which of these are physics problems, which are science problems, and which are engineering problems?

Continual learning sits closest to the science end. The problem is not that models technically cannot update their weights. They can. The problem is that there is no widely accepted mechanism for how a model should continuously learn from experience without corrupting what it already knows.

This is why skill formation matters so much. A model can become better at explaining a domain, summarizing procedures, using the right terminology, and answering exam-style questions. But that is still not the same thing as developing competence.

That kind of expertise comes from feedback loops. You try, you fail, you get corrected, you see consequences, and over time your internal model changes. Current LLMs do not really do that. They can use new information in context, and RAG can give them access to more information, but access is not the same as internalized capability. Once the context disappears, the model has not necessarily changed.

So continual learning is not "just let the weights update." The model would need to know what is worth learning, what should be ignored, what is generally true, what is user-specific, how much to update, and how to avoid forgetting older capabilities. The human brain does solve some version of this, so it is hard to call it impossible. But it remains a science problem.

Long-horizon autonomy feels more like a mix of science and engineering, but I'd place it closer to engineering. We are already seeing meaningful progress here with agentic systems: chaining steps, managing context, using tools, writing intermediate plans, checking outputs, and rethinking strategy midway through execution. The better systems are starting to look less like a single model call and more like an operating loop: plan, act, observe, critique, re-plan, and continue.

The open question is whether these scaffolds can remain stable over very long time horizons. It is one thing for an agent to complete a 20-step coding task. It is another thing for it to pursue a goal over weeks, recover from unexpected failures, notice when its plan is wrong, and not slowly drift away from the original objective. Still, this feels like the kind of problem where better architecture, better evals, better memory systems, and better tool use can compound quickly.

Reliability in high-dimensional retrieval feels more like a fundamental constraint, or at least the least obviously solvable one. There are clever ways to chunk, embed, retrieve, re-rank, summarize, and process information. But retrieving the right information from a giant knowledge space is not trivial, especially when the thing you need is not obviously keyword-matched, when the relevant detail is buried in a long document, or when the answer requires connecting weak signals across time.

This becomes even harder as the time horizon stretches from minutes to days, months, or years. A model might retrieve the right document today, but can it maintain a coherent understanding of a project over a year? Can it remember which facts are stale, which decisions were reversed, which assumptions were invalidated, and which details matter now? Some of this will be solved through better architecture. Some of it may run into deeper limits around compression, context, memory, and search.

So will all of this be solved in the next 10 years?

My instinct is yes, but with an important caveat. The fact that the biological brain solves these problems means they are not impossible in principle. Humans do continual learning, long-horizon autonomy, and high-dimensional retrieval every day. We learn from experience, pursue goals over years, and pull the right memories into consciousness when they matter.

But the brain did not solve these problems with a bigger context window and a vector database. It solved them through a very different architecture: embodiment, reward, emotion, attention, sleep, memory consolidation, forgetting, motor feedback, and constant interaction with the world.

So I do think these problems are solvable. I just don't think the answer is simply "scale the current thing." My guess is that AGI requires more than bigger models. It likely requires new systems around models: memory, agency, feedback, self-correction, continual learning, and some way of turning experience into durable capability.