The first surprise of this afternoon was finally finding an informed and sensible discussion of the implications of AI agents for hep-th research, in the form of a twitter thread by stringking42069.
The second was learning from the twitter thread about Xi Yin’s ongoing project to have GPT 5.5 write a QFT textbook under his supervision. The past week there had been rumors that he was hired by OpenAI. If he’s now on their payroll, what he’s getting paid to do presumably is this textbook, which is a very active ongoing project.
The current state of the textbook is at this github repository. I don’t see a pdf anywhere there, but you can get the tex files by cloning the repository with
git clone https://github.com/xiyin137/QFT
and then tex’ing
monograph/tex/main.tex
He’s working on this right now, with latest changes 6 minutes ago. What I downloaded produced a 3527 page pdf document.
I’ve just skimmed through the thing, and it’s quite fascinating, raising a host of questions. This is clearly a work in progress, on its way to a document with tens of thousands of pages (or more…). Also, it’s undoubtedly the first of many such projects to come. I know from experience that writing a textbook is a huge effort, and AI agents very plausibly could take over a lot of the work. So, one set of questions is about what the future of textbooks, specifically QFT textbooks, will be.
The first obvious comment is that this document is useless as a “textbook”, in the sense of something one could use to learn the subject from. No one is going to learn QFT in any useful sense by trying to read these thousands of pages. Sections of it might be useful to experts in the same way that a badly-written research monograph is.
When I wrote a QM (and some elementary QFT) textbook, a big part of the experience was the following process. Starting from a certain conception in my mind of what the right way to think about a topic was, I’d start writing, and then after a while realize that there was a better, clearer way to think about the topic, so lots of material had to be thrown out or completely rewritten. Sometimes I came to this realization because things were getting too complicated and it became clear there was a simpler way. Sometimes the new insight came from getting stuck on a calculation: at one point, days spent chasing signs that wouldn’t match led to understanding that I was thinking about the dual of the vector space I should have been thinking about.
For an AI agent to be able to write a good textbook, I think it will need to somehow embody that kind of process: realizing when a line of exposition needs to be abandoned because there is a better way to describe what is really going on.
For a QFT textbook, a big set of issues is the unsolved problem of what the best way to think about QFT really is. Just going out to the current literature, grabbing what is there and then trying to rework it as a textbook/monograph, results in a huge, undigested mass of various inconsistent ways of thinking. This may be useful for making clearer to us what’s wrong with the current state of the field, but not useful for anyone who wants to better understand what is really going on.
In any case, I’m curious to see how this project evolves, as well as others like it that surely will come. Initially their role will likely be just to provide examples of what doesn’t work: AI agents fed millions of pages of crud will just produce more crud. Can they develop real insight about fundamental issues in theoretical physics, or can human beings with real insight turn them into useful tools for progress? I’ve no idea how that will play out in the long term. In the short term, I think what we’ll see is just more crud, often dressed up and sold to the public as innovation by the PR departments of our new tech oligarchies.
Another obvious concern is that this isn’t just a massive plagiarism project. The training data for the bot presumably includes the top QFT textbooks. So you drop the top QFT textbooks into a computer, and it regurgitates the same material but with new grammatical choices and some slight variations of all the exercises it ingested. Send the royalty checks to OpenAI and Yin now?
Bob,
To be fair, this isn’t like most QFT textbooks. You could learn QFT from those, this is something different.
With unrestricted access to artificial intelligence, a textbook-writing project may be conceptualized as an iterated function system. In this framework, the process consists of successive rewritings of an initially extensive text, each iteration aiming to improve internal consistency and conceptual clarity. At certain stages, the procedure may exhibit simple recurrence; at others, it may display behavior analogous to convergence toward a fractal attractor. In principle, this iterative refinement suggests the possibility of approaching a limiting object—a text that is both logically coherent and maximally transparent.
The effective use of AI systems may be compared to the use of horses: some are better suited to carrying substantial loads, while others excel in speed. In all cases, however, the outcome depends critically on the skill and judgment of the rider.
Hi Peter,
“He’s working on this right now, with latest changes 6 minutes ago.”
No, he is not. If you look at the GitHub page, the number and size of commits are way too big for a human — the 300+ line patches have been arriving a several times per hour, continuously for at least 72 hours, and probably much longer (see https://github.com/xiyin137/QFT/graphs/commit-activity for a graph).
What Xi Yin has done was most probably to make a script that will ask GPT to read the manuscript, suggest an improvement, formulate it as a patch, upload it to GitHub, and repeat. On average, GPT seems to be able to submit cca 1000 patches per week, which amounts to a single 300-line patch once every ten minutes, continuously for 3-4 weeks now.
A human would need to sleep, at least some part of this time.
I am curious to see the final outcome of this kind of experiment. GPT seems to be continuously modifying the text, and the main hypothesis of the experiment is that this process will converge. A human expert could take, say, monthly snapshots of some particular chapter, and study the evolution of the text, in various ways, to see if the quality of the material presented in that chapter is improving, or stagnating, or declining.
My feeling is that the GPT writing process will fail to converge, at least if left without some major guardrails. There are many ways to phrase and re-phrase any particular topic in QFT, and a lot of these ways have been explored already in existing (human-made) literature. I am guessing that GPT will just be oscillating between them, rather than converging onto any one of them.
Best, 🙂
Marko
I submitted a pull request and got an informative answer
https://github.com/xiyin137/QFT/pull/710
Thanks Marko,
That clarifies what is going on. This morning I just got the latest version, which is 57 pages longer than the one from yesterday afternoon.
There are vast amounts of junk in this document, for it to improve, the agent writing it needs to not just add more junk, but reorganize and delete a lot of what is already there.
My experience is very limited, but from what I’m seeing the best way to understand the current capabilities of AI agents in math and theoretical physics is as those of a not very talented graduate student who somehow has immediate access to all the literature. Asked to produce a textbook/monograph, such an agent regurgitates vast amounts of undigested stuff.
Will new versions of these AI agents start behaving like very talented graduate students, or start developing the kind of deeper understanding that humans develop from many years of thinking about and working on an topic? Or will they always just be versions of the current state of this project?
Marko,
If it’s based on continuously running the best frontier AI agents, another question this project raises is that of what it is costing and who is paying for it.
Think of textbooks you love. I’d be surprised if they weren’t in some way idiosyncratic in contents or exposition. Although an LLM could produce such a book, the generic balanced AI style ain’t that. Old masters poured their heart and soul into their books, and it showed; it sounds as though this author is just pouring tokens and GPU hours.
Moreover, LLMs are in a sense a lossy compression of all reference books. In an old fashioned reference book, you look in the index and if it has your topic you turn to that page. In an LLM, you query the index and get a probabilistic answer for what might have been in such an entry. It’s strange to go full circle and create new old-fashioned reference books with an LLM.
Hi Peter,
I believe writing an AI QFT book is a terrible idea. The most the “author” can do is try to filter out the junk from what the AI got wrong from the work of thousands of authors who actually know the subject.
Having said that, I think AI can be a great learning support tool if you use it the right way. Sometimes, a passage in a book (written by a human who actually knows what he’s doing) is a bit obscure, and I confess I use AI to ask some questions. AI can get things wrong, but if you know how to interact with it and if you are actually learning, you will only be satisfied once it actually makes sense, and you cross-check it with other sources.
But an AI book is the worst of all worlds. It probably gives you an unknown amount of junk without the benefit of the interaction that AI provides.
Arkadiusz Jadczyk — regarding “At certain stages, the procedure may exhibit simple recurrence; at others, it may display behavior analogous to convergence toward a fractal attractor. In principle, this iterative refinement suggests the possibility of approaching a limiting object—a text that is both logically coherent and maximally transparent.”, I suspect a better analogy is with a ball rolling around a potted surface that ultimately winds up in one of a very large number of possible holes (i.e. locations where the potential energy has a local minimum). Which hole the ball winds up in will be very sensitive to “instructions” that cause the A.I. to make certain compromises between clarity and completeness in exposition.
I looked at tiny portions of the manuscript that directly relate to work that I did and which I think I ought to know something about. I see a few issues. It makes me wonder how reliable the material would be on topics I know much less well. There are a few examples. I confine myself to two such:
1. “Moore-Seiberg duality data” is discussed in section 70.5. There are problems with the last paragraph of the section. There are a couple of minor things: For example, no discussion is given of the relation of \theta_i to the conformal weights. (A relation is provided in the next section but without the subtraction by c/24 which makes the statements about the SL(2,Z) representation wrong.) Most important, it omits the crucial main issue: Finding the OPE coefficients tying together left and right-movers. The last sentence of this section reads:
“The monograph therefore uses the equations as structural
constraints and as coordinates on known classes of rational CFT data; the definition of the CFT itself includes the analytic and Hilbert-space structures just listed.”
I had to puzzle over this a bit. I think I know what it is trying to say, but I do not think it is said very clearly. The title of the next section 70.6 promises that we will understand something about a modular tensor category. But no definition is given, and, crucially, there is no discussion of the invertibility of the S- matrix as defined by the braiding matrix. etc. etc.
2. Let’s now turn to the relation of Donaldson and Seiberg-Witten invariants in Conjecture 112.13. If I’m reading it correctly (who knows…), equation 112.36 is seriously wrong. There should be two sums over spin-c structures of SW invariants weighted by trigonometric functions. Physically they arise from the two singularities, the “monopole” and “dyon” singularity in the Seiberg-Witten special Kahler geometry. The “Moore-Witten” normalization is curious terminology because it appears in Witten’s paper 3 years before Moore and Witten wrote their paper giving a principled derivation of the relation of Donaldson to Seiberg-Witten invariants. I’ve never seen that terminology before. A discussion of the “Moore-Witten” derivation would have been much more informative.
Section 112.13 is something I’ve thought a bit about. I don’t see how anyone who is not already familiar with the literature can possibly understand it. There is no explanation of many crucial points and some notation isn’t fully explained. (For example – the function \Psi). The derivation of equation 112.48 is not so trivial and no reference is provided. How is the reader supposed to understand it? I suppose by using AI to find the original papers… Most important, I think it misses the main point: The wall-crossing of the u-plane integral at the strong coupling singularities *cancels* the contribution from the effective path integral valid in the neighborhood of those singularities. This allows one to *derive* the universal functions in the effective action of the LEET (containing a U(1) vm and charge 1 hm in the appropriate duality frame) valid in the neighborhood of the strong coupling singularities. It is the derivation of those universal functions that is at the heart of the relation of Donaldson to Seiberg-Witten invariants.
Again, these are tiny, relatively unimportant, pieces of this gigantic project. But if close and informed inspection reveals such difficulties in these two (and a few other) slivers, how am I supposed to put any trust in the remainder?
Gregory Moore
Gregory Moore,
Thanks! What you’re looking at and describing to us I think is the result of what the best current AI agent produces when it starts with your own writing and does whatever it does.
A good analog is what happens if you ask a not very good student with no expertise to read something and then try and explain it.
Math is different than physics here, since conventional rigorous math exposition is much better suited as input for an AI model (and rigorous formalization may even be possible).
We’ll see what happens, but as far as hep-th material goes, these current agents are just going to take what’s in the literature and make it worse.
Interesting. This sounds like the opposite of the way many people use AI. Many people would ask AI to answer a question using its knowledge of all the literature. While this project seems to be to synthesize all the literature on QFT into a giant document.
Perhaps a better use of AI to write textbooks would be to write custom expositions of a narrower topic. For example, “Given what you know about my background, please write an explanation for me of the relation between Seiberg-Witten and Donaldson invariants in around 50 pages.”