I use these models every day. Not to experiment with, not as a side interest - they are in the actual loop of how I build software now, and have been for long enough that the novelty has completely worn off.
That is what caused the problem. Once the novelty goes you start noticing shape.
Somewhere around the point where I had shipped enough with them to stop being impressed, I started getting a specific feeling I could not shake: that I was seeing the same things come back. Not the same code, exactly. The same moves. Ask for a service and you get the structure that most services on GitHub have. Ask for a fix and you get the fix that worked for the most people. Everything it handed me was competent and everything it handed me was something that already existed somewhere, slightly rearranged.
Which is, of course, exactly what it does. It is a next-token predictor. Given what has been written, produce what most plausibly comes next. I knew that the whole time. But knowing it as a fact about the architecture and feeling it in the output are two different things, and the second one is what made me stop and actually think.
Because the question that follows is not obvious at all.
Can pattern matching find something genuinely new?
I do not mean new-to-me. I mean new. A thing nobody had put together before. My first instinct was no, obviously not - you cannot interpolate your way to something that is not in the data. But I sat with that for a while and it stopped being obvious, because the follow-up question is harder:
Is that not what we do?
Every invention I can think of looks, in hindsight, like a recombination. Someone knew about two fields and noticed they fit. Someone applied a technique from one place to a problem in another. Read enough history of science and it is patterns all the way down. So if human invention is also recombination over prior work, then the difference between us and the model is quantity, not kind, and the whole feeling I was having is just snobbery.
I could not settle it. I genuinely did not know, and it is not the sort of thing you settle by having opinions at it.
Then I went looking for people who had thought about this properly, and found that two of the most serious mathematical minds alive have been circling exactly this question for years, from opposite ends. Neither of them is arguing about whether the code compiles. They are arguing about what kind of thing the process is.
The two positions, before the maths
Roger Penrose won the Nobel Prize in Physics in 2020, shared with Reinhard Genzel and Andrea Ghez, for showing that black hole formation is a robust prediction of general relativity. That is not what he is arguing about here. His claim about minds is older, from The Emperor's New Mind in 1989 and Shadows of the Mind in 1994, and it is roughly this: human mathematical understanding is not an algorithm. Not a slow algorithm, not a poorly implemented one. Not one at all. He argues this from Gödel's incompleteness theorems, and he has never really moved off it.
Terence Tao won the Fields Medal in 2006 and is, by most accounts, the strongest working mathematician of his generation. He has been unusually public about actually using these models on real research, and his read is more practical. He has described the experience of working with them as being like working with a mediocre but not completely incompetent graduate student: useful for the parts of a proof that are mechanical, not to be trusted with the part that requires knowing which direction to go.
These two are not having an argument with each other. They are answering different questions. But putting them side by side is what made me realize we have built something we do not have a good word for, so I want to do the maths properly first and then come back to the word.
Most things are not computable, and this is provable in a page
Start somewhere solid. Before any question about minds, there is a hard result about machines that almost nobody outside computer science has been walked through, and it takes about ten minutes.
A program is a finite string of characters. That is all it is. So let us count them.
Line up every possible program by length. There are finitely many one-character strings, finitely many two-character strings, finitely many of length three, and so on forever. You can therefore put every program that could ever be written into a single numbered list: all the length-1 ones first, then length-2, and onward. Nothing is left out, because every program has some finite length and gets reached eventually.
P_0, P_1, P_2, P_3, ...
That is the first half. Now count the things we might want programs to do.
Take the simplest interesting task: a function that takes a whole number and
answers yes or no. Write yes as 1 and no as 0, so one such function starts
f(0) = 1, f(1) = 0, f(2) = 0 and carries on forever. Nothing exotic.
That is one function. There are others, one for every possible way of answering at every input, and that whole collection is what we are about to count.
Claim: the set of all of them cannot be put in a numbered list.
Suppose they could. Suppose someone hands you a list claiming to contain every one of them:
f_0 : f_0(0) f_0(1) f_0(2) f_0(3) ... f_1 : f_1(0) f_1(1) f_1(2) f_1(3) ... f_2 : f_2(0) f_2(1) f_2(2) f_2(3) ... f_3 : f_3(0) f_3(1) f_3(2) f_3(3) ...
Now build a new function g by walking down the diagonal and flipping every
value you find:
g(n) = 1 - f_n(n)
Is g on the list? It cannot be f_0, because g(0) was defined to differ from
f_0(0). It cannot be f_1, because they differ at 1. For every n, g differs
from f_n at position n. So g differs from every function on the list, and
the list was supposed to be complete.
The list cannot exist. There are strictly more yes/no functions than there are positions in any numbered list.
Put the two halves together:
programs : countable (can be listed) yes/no functions : uncountable (cannot be listed) therefore almost every function has no program at all
"Almost every" is doing real work there. It is not that a few awkward cases slip through. The computable functions are a vanishingly thin sliver inside the space of all functions. If you could pick a function at random, the probability of landing on one any machine could compute is zero.
Every computer that has ever run, and every computer that ever will, works inside that sliver.
A concrete one: the halting problem
The counting argument proves uncomputable things exist without showing you one. Turing gave a specific one in 1936, and the proof is short enough to hold in your head.
We want a program H that reads another program and its input, and reports
whether that program eventually finishes or runs forever.
Suppose H exists. Then I can write this:
D(P):
if H(P, P) says "halts":
loop forever
else:
haltD takes a program, asks whether that program halts when fed itself, and then
does the opposite.
Now run D on D.
If H says D halts on input D, then by D's own definition it loops forever.
So H was wrong.
If H says D loops forever on input D, then by the definition it halts
immediately. So H was wrong.
H is wrong either way, on an input we constructed from H itself. The only
assumption was that H exists, so H does not exist. There is no general
procedure for deciding whether an arbitrary program terminates.
This is not a gap waiting for better technology. No amount of compute closes it, because the contradiction is built out of whatever machine you propose.
How fast the wall arrives
Here is the part that made this concrete for me rather than philosophical.
Take Turing machines with n states. Some run forever. Of the ones that stop,
one runs longest. Call that number of steps BB(n) - the busy beaver function.
BB(1) = 1 BB(2) = 6 BB(3) = 21 BB(4) = 107 BB(5) = 47,176,870 BB(6) = larger than anything you can write in ordinary notation
Five states. Forty seven million steps. And then the sequence leaves the universe
entirely: BB(6) is not merely large, it exceeds what you can express with
exponentials stacked on exponentials.
BB is not computable, and the reason is lovely: if you could compute it, you
could solve the halting problem. Run any n-state machine for BB(n) steps, and
if it has not stopped by then it never will, because BB(n) is by definition the
longest any halting machine of that size takes. So a BB calculator is a halting
oracle, and we just proved those do not exist.
A perfectly well defined sequence of whole numbers. Each value is a definite, finite number. And no machine can produce them.
Gödel, and the step Penrose takes
Now the part Penrose builds on.
In 1931 Gödel showed that any formal system F which is consistent, has
mechanically checkable rules, and is strong enough to describe arithmetic, has a
statement it cannot settle. The construction encodes, inside F's own language,
a sentence that says of itself:
G_F = "this statement has no proof in F"
Follow it through. If F proves G_F, then F has proved a statement asserting
its own unprovability, which makes F inconsistent. So a consistent F cannot
prove G_F.
But look at what that means. G_F says it has no proof in F. We just showed it
has no proof in F. So G_F is true, and F cannot prove it.
You cannot patch this by adding G_F as an axiom. The new system has its own
Gödel sentence. It is not a bug in one system, it is a property of formal systems.
Penrose's move is the next step. We sat here and worked out that G_F is
true. The machine running F could not. If our mathematical understanding were
some formal system F, we could not have seen what we just saw. Since F was
arbitrary, no formal system captures human mathematical insight. Therefore
understanding is not computation, and whatever the brain is doing must involve
physics that is not algorithmic.
That is the argument, and it has been contested hard for thirty years. I think the objection is strong and I want to state it properly, because a version of this essay that only gave you Penrose's side would be advocacy rather than thinking.
Where I land: Penrose has not proved that minds are non-computable. But he has
put his finger on something real, and the objection does not make it go away.
There is a difference between deriving if F is consistent then G_F, and
stepping outside F to ask whether F was the right system in the first place.
That step is not a move within the game. It is a judgement about which game to
play. Nothing in the formal machinery tells you when to make it.
What Tao is actually saying
Tao is not doing metaphysics. He has been using the tools, and reporting.
His observations, as I read them, come to three things. These models are genuinely useful for the mechanical portion of mathematics - rearranging, checking cases, recalling a standard technique, writing out a routine argument that would have taken an afternoon. They fail in a specific and dangerous way, producing arguments that are locally plausible at every step and globally wrong, which is a much harder failure to catch than obvious nonsense. And the part they do not touch is the part that decides whether research succeeds: knowing which problem is worth attacking and which direction is likely to pay.
The graduate student comparison is precise, and worth sitting with. A mediocre graduate student is not stupid. They know the material. They can execute a technique you name. What they cannot yet do is look at an unfamiliar problem and sense which of thirty available approaches is the one. That sense is what a PhD is actually for, and it is the thing that does not come from having read everything.
That third point is the one that answered my question, or at least turned it into a sharper one. The complaint is not that the tool produces bad work. It is that the part it cannot do is the part where the direction gets chosen - and choosing the direction is where new things come from.
The word
So here is where I have ended up, and this is mine rather than either of theirs.
Cleverness is finding the answer inside a space you have been handed. Enormous recall, fast pattern completion, fluent recombination of things that have been seen. Measured by benchmarks, and models are now extraordinary at it - better than me at almost every task that has this shape.
Intelligence includes knowing the space is wrong. Stepping outside the system, noticing the question is malformed, sensing that the standard approach will not work here before spending three weeks failing with it.
What we have built is a superb cleverness engine. That is not a criticism. Most of what I do in a day is cleverness, and having it available at that speed has changed how I work. But the two capabilities are different, and calling the whole thing intelligence has made it hard for people to see the seam - which is exactly where the failures live.
The mathematics above does not prove that a machine can never make the outside step. Nothing here rules it out. What it does establish is that the space of things no algorithm can reach is not small and not exotic - it is almost everything, and it starts at problems as ordinary as "does this program stop". Any claim that scale alone gets us across that gap has to explain how, and I have not seen the explanation.
Why an engineer should care
This is not just philosophy. It changes how I use these tools.
The confident-and-wrong failure is structural. A model producing a plausible wrong proof is not malfunctioning. It is doing precisely what it does - producing the continuation that best fits the pattern. Plausibility is the objective. It was never truth. So verification is not a nice practice, it is the whole job.
Trust it in proportion to how well-trodden the ground is. Standard API, common idiom, well-documented library: excellent, use it. Novel constraint, an interaction nobody has written up, a system whose failure mode is specific to your architecture: that is the outside step, and it is where the fluent answer is most dangerous, because it will still sound the same.
Deciding what to build stays yours. Every model I use will happily help me build the wrong thing beautifully. Nothing in the training objective rewards saying "this problem is not worth solving." That judgement is not in the space.
Read the reasoning, not the answer. The time I spend reading generated code carefully is not overhead on top of the work, it is the work. If you cannot follow it well enough to find the bad line, you cannot use it for anything that matters.
So can pattern matching find something new
Back to the thing I could not settle.
My honest answer, after all of the above, is: partly, and not in the way that matters most.
Recombination is real and it is not nothing. A great deal of human invention genuinely is one field's technique carried into another field's problem, and a system that has read everything has a real advantage at spotting those. I was wrong to be snobbish about it. Some of what looks like insight is coverage, and we have built something with more coverage than any person will ever have.
But the recombination story leaves out the step that makes the recombination possible. Somebody has to decide the two things are worth putting together, and that decision is not itself a recombination - it is a judgement about which question is the right one, made before any evidence exists that it will work. Every account of invention I know quietly skips over that part and then declares the whole thing pattern matching.
That is the same gap Penrose points at with G_F and Tao points at with
direction. Not a lack of knowledge. A step that is not a move inside the game.
So the feeling I had was not snobbery, but it was not quite right either. The models are not failing to be creative. They are doing the half of creativity that we could formalize, extremely well, and the half we could not formalize is still sitting there untouched - and it was always the half that decided what got invented.
Where I actually stand
I do not think Penrose has proved his conclusion, and I have said why. I also do not think the objection rescues the claim on the other side. Nothing above shows a machine can never make the outside step. What it shows is that the territory no algorithm reaches is not a strange corner - it is almost everything, starting at questions as ordinary as whether a program stops. Anyone claiming scale alone carries us across has to say how, and I have not seen it said.
Which leaves me roughly where I started, except that the feeling now has a shape.
I still use these models every day, for the same work, and I am not giving them up. What has changed is that I stopped reading the repetition as a defect to be fixed in the next version. It is the shape of the thing. Cleverness is the half we managed to automate, it is genuinely remarkable, and it is most of what a working day consists of.
The other half is still mine, and the useful consequence is that I know which half to pay attention to. When the answer comes back fast and fluent and fits the shape of every answer I have seen before, that is the tool working. When I cannot tell whether the problem is even the right problem, no amount of asking is going to help, and that is the part of the job I should stop trying to delegate.