Reading this Nautilus article about Julian Barbour led me recently to something I don’t think I’ve ever read before, Dirac’s 1963 Scientific American article The Evolution of the Physicist’s Picture of Nature. There is a very famous quote from this article that I’ve often seen:
It is more important to have beauty in one’s equations than to have them fit experiment
but I was unaware of the context of that quote, in which the famous part is prefaced by “I think there is a moral to this story, namely that…” The story that Dirac had in mind was that of the discovery of the Schrödinger equation. Famously, Schrödinger first wrote down a relativistic wave equation (now known as the Klein-Gordon equation). This equation is what one quickly gets if one follows de Broglie’s idea that matter is described by waves, and uses the relativistic energy-momentum relation. Here’s the full story, as told by Dirac, giving his famous quote in context:
I might tell you the story I heard from Schrödinger of how, when he first got the idea for this equation, he immediately applied it to the behavior of the electron in the hydrogen atom, and then he got results that did not agree with experiment. The disagreement arose because at that time it was not known that the electron has a spin. That, of course, was a great disappointment to Schrödinger, and it caused him to abandon the work for some months. Then he noticed that if he applied the theory in a more approximate way, not taking into ac count the refinements required by relativity, to this rough approximation his work was in agreement with observation. He published his first paper with only this rough approximation, and in that way Schrödinger’s wave equation was presented to the world. Afterward, of course, when people found out how to take into account correctly the spin of the electron, the discrepancy between the results of applying Schrodinger’s relativistic equation and the experiments was completely cleared up.
I think there is a moral to this story, namely that it is more important to have beauty in one’s equations than to have them fit experiment. If Schrodinger had been more confident of his work, he could have published it some months earlier, and he could have published a more accurate equation. That equation is now known as the Klein-Gordon equation, although it was really discovered by Schrödinger, and in fact was discovered by Schrödinger before he discovered his nonrelativistic treatment of the hydrogen atom. It seems that if one is working from the point of view of getting beauty in one’s equations, and if one has really a sound insight, one is on a sure line of progress. If there is not complete agreement between the results of one’s work and experiment, one should not allow oneself to be too discouraged, because the discrepancy may well be due to minor features that are not properly taken into account and that will get cleared up with further developments of the theory.
There’s another remarkable aspect of this Scientific American article, something about it that would be completely inconceivable today: they write down three equations, including both the famous non-relativistic Schrödinger equation for the Coulomb potential, as well as the relativistic Klein-Gordon version.
Dirac notes that Schrödinger found his formulation of quantum mechanics in a very different way than Heisenberg found his:
Heisenberg worked keeping close to the experimental evidence about spectra that was being amassed at that time, and he found out how the experimental information could be fitted into a scheme that is now known as matrix mechanics. All the experimental data of spectroscopy fitted beautifully into the scheme of matrix mechanics, and this led to quite a different picture of the atomic world.
whereas
Schrödinger worked from a more mathematical point of view, trying to find a beautiful theory for describing atomic events, and was helped by De Broglie’s ideas of waves associated with particles. He was able to extend De Broglie’s ideas and to get a very beautiful equation, known as Schrödinger’s wave equation, for describing atomic processes. Schrodinger got this equation by pure thought, looking for some beautiful generalization of De Broglie’s ideas, and not by keeping close to the experimental development of the subject in the way Heisenberg did.
At the end of the article, Dirac makes the case that progress in fundamental physics may not come from a theorist like Heisenberg finding a scheme to match experimental results, but from a theorist like Schrödinger pursuing mathematical beauty:
It seems to be one of the fundamental features of nature that fundamental physical laws are described in terms of a mathematical theory of great beauty and power, needing quite a high standard of mathematics for one to understand it. You may wonder: Why is nature constructed along these lines? One can only answer that our present knowledge seems to show that nature is so constructed. We simply have to accept it. One could perhaps describe the situation by saying that God is a mathematician of a very high order, and He used very advanced mathematics in constructing the universe. Our feeble attempts at mathematics enable us to understand a bit of the universe, and as we proceed to develop higher and higher mathematics we can hope to understand the universe better.
This view provides us with another way in which we can hope to make advances in our theories. Just by studying mathematics we can hope to make a guess at the kind of mathematics that will come into the physics of the future. A good many people are working on the mathematical basis of quantum theory, trying to understand the theory better and to make it more powerful and more beautiful. If someone can hit on the right lines along which to make this development, it may lead to a future advance in which people will first discover the equations and then, after examining them, gradually learn how to apply them. To some extent that corresponds with the line of development that occurred with Schrodinger’s discovery of his wave equation. Schrödinger discovered the equation simply by .looking for an equation with mathematical beauty. When the equation was first discovered, people saw that it fitted in certain ways, but the general principles according to which one should apply it were worked out only some two or three years later. It may well be that the next advance in physics will come about along these lines: people first discovering the equations and then needing a few years of development in order to find the physical ideas behind the equations. My own belief is that this is a more likely line of progress than trying to guess at physical pictures.
The context for his famous quote is thus an argument for pursuing fundamental physics by looking for mathematical beauty, not giving up on a beautiful equation just because it doesn’t seem to fit experiment. As in Schrödinger’s case, more effort may be needed to understand the actual relationship of the equation to reality.
Besides this argument, which I’ve always been well aware of and sympathetic to (despite not knowing the context in which Dirac was making it), there’s something else I found very striking about the 1963 article. Dirac begins by explaining that the four-dimensional Lorentz symmetry of relativity is in a sense broken by the choice of a way of describing the state of the world:
What appears to our consciousness is really a three-dimensional section of the four-dimensional picture. We must take a three-dimensional section to give us what appears to our consciousness at one time; at a later time we shall have a different three-dimensional section. The task of the physicist consists largely of relating events in one of these sections to events in another section referring to a later time. Thus the picture with four dimensional symmetry does not give us the whole situation. This becomes particularly important when one takes into account the developments that have been brought about by quantum theory. Quantum theory has taught us that we have to take the process of observation into account, and observations usually require us to bring in the three-dimensional sections of the four-dimensional picture of the universe.
The special theory of relativity, which Einstein introduced, requires us to put all the laws of physics into a form that displays four-dimensional symmetry. But when we use these laws to get results about observations, we have to bring in something additional to the four-dimensional symmetry, namely the three-dimensional sections that describe our consciousness of the universe at a certain time.
Dirac also refers to work on canonical formulations of general relativity, aimed at quantizing gravity:
… if one insists on preserving four-dimensional symmetry in the equations, one cannot adapt the theory of gravitation to a discussion of measurements in the way quantum theory requires without being forced to a more complicated description than is needed bv the physical situation. This result has led me to doubt how fundamental the four-dimensional requirement in physics is. A few decades ago it seemed quite certain that one had to express the whole of physics in four-dimensional form. But now it seems that four-dimensional symmetry is not of such overriding importance, since the description of nature sometimes gets simplified when one departs from it.
Thinking about twistor unification has led me to some similar thoughts: a Euclidean formulation of quantum theory requires picking a choice of imaginary time direction and breaking SO(4) symmetry in order to define states. Dirac thinks of our consciousness as giving us access to the state of the universe defined on a 3d slice, but the twistor point of view is even more directly related to our conscious experience. A point in space time is defined by the sphere of light rays through the point, and it is this sphere that our vision gives us direct access to, with 3d space something we make up out of these spheres.
A common argument against Dirac’s point of view is that it’s engaging in mysticism. For another recent article that touches in a different way on the mystical nature of a discovery about fundamental physics, see this interview with Frank Wilczek, where he tells this story:
The mystical moment came while I was visiting Brookhaven National Laboratory, on Long Island. Somehow—I don’t remember how, exactly—I wound up alone, standing on a jerry-rigged observation platform above a haphazard mess of magnets, cables, and panels. This was a staging area for assembling detectors and renovating pieces of the main accelerator there, the Alternating Gradient Synchotron (AGS). I must have gotten separated from my host for a few minutes. In any case, there I was, alone inside an aircraft-hangar-sized metallic box, staring down at the kind of equipment that people use to explore the fundamentals of Nature experimentally.
And then it happened. It came to me, viscerally, that the intricate calculations I’d done using pen and paper (and wastebasket) might somehow describe this entirely different realm of existence—namely, a physical world of particles, tracks, and electronic signals, created by the kind of machinery I was looking at. There was no need to choose, as philosophers often struggled to do, between mind or matter. It was mind and matter. How could that be? Why should it be? Yet I somehow, I suddenly knew that it could be so, and should be so.
That was my mystical experience. I warned you that it was ineffable.
”It may well be that the next advance in physics will come about along these lines: people first discovering the equations and then needing a few years of development in order to find the physical ideas behind the equations.”
Didn’t string theorists try to do just this – except that many years of development didn’t produce the hoped-for reward?
The problem with the argument by beauty is, who gets to decide something is beautiful? I can assure you, the crackpots who come to my office and, more frequently now, send me e-mails (maybe they post on Tik-Tok as well, but I don’t follow that), believe their ideas are beautiful…. and they get quite angry when I don’t agree.
Dirac says “What appears to our consciousness is really a three-dimensional section of the four-dimensional picture.” What section is this? What reaches us at any instance is some information from the past null cone and its interior, which is a 4-dimensional region of the spacetime. Even if the spacetime is flat Minkowskian, all the other points on the hyperplane t=0 of our instantaneous rest frame are separated from us by spacelike intervals, so this hyperpalne cannot be a part of our consciousness. In the presence of gravity, it is interesting to note that without simplifying — and quite unrealistic — cosmological assumptions, the past null cones of points in the spacetime are horribly tangled, and global foliations by spacelike smooth hypersurfaces do not exist in general. The latest blog entry of David Mumford titled “Ruminations on cosmology and time”, dated March 1, 2021, makes this point quite forcefully.
Arnold Neumaier/CWJ,
I’ve often argued that string theory unification is not a beautiful idea (the string theory landscape is the antithesis of beautiful).
Dirac’s argument I think is not so much that you should use beauty to evaluate the success of a theory, but you should use it as a criterion for choosing a promising idea to pursue. That different people will disagree about what is beautiful then just implies that different people will pursue different research programs, which is fine.
Nitin Nitsure.
I don’t know why Dirac was bringing our consciousness into it. It seemed he was just saying that the usual way to get a state space in a relativistic theory is to choose a spacelike hypersurface.
Dirac’s Sci Am article is cited in Ch. 1 of Weinberg’s QTF. To the unintiated: both the original article (Dirac) and the volume (Weinberg) are ‘must read.’
In a humorous example of your observation, I searched in vain for equations in the 2010 reprint (that you linked to) of the original 1963 Dirac SA article.
A bit paradoxical, isn’t it, that Dirac’s preference for selecting beautiful theories over those based on empiri is based on the empirical evidence of how well the former work.
Art,
Good point. SciAm I guess no longer has the technology to reproduce an equation. For the original, with equations, one source is
https://www.jstor.org/stable/24936146
I am and always will be highly skeptical of reliance on esthetics, though of course scientists are human beings, and need to follow their instincts and intuitions to a certain extent to be able to function at all. Naive falsificationism is also, of course, inadequate and theorists shouldn’t be held in thrall to every new bit of data that experimentalists generate. This is clearly especially true in a new field.
But Dirac himself seemed to have followed his own esthetic sensibilities and doubts about empirical data right off a cliff with his “large numbers hypothesis”, and for every example of beauty shining a guiding light, I bet there are at least a thousand for which it was the road straight to perdition. I simply can’t put much faith in “beauty”, except as an occasional source of inspiration or motivation to try harder, which we all need at times in one form or another. The full account of human history simply precludes ascribing it any greater significance. It always seems to be true only with a very selective form of hindsight.
“Beauty” in equations is (A) in the eye of the beholder and (B) very largely a matter of the level of abstraction of the notation. The purpose of abstract notation is to manage complexity; that is not a matter of beauty, but of legibility, and it is a double-edged sword.
Dirac’s point, if he had one, was that ugly equations (cf. A and B above) carry, or ought to carry, some degree of presumption against their correctness (he was more explicit about this elsewhere). But the reason why we appreciate beauty in reality is not because it is universal, or even typical; it is because it is exceptional.
Even worse, the text refers to at the equation(s) that is/are no longer there, so clearly the editorial capacity is somewhat lacking too.
Perhaps a better description than beauty of what is sought by physicists is that the description be terse
Peter Woit
Thank you for your explanation of Dirac’s quote. However, you did not address the point about a possible lack of spacelike foliations. I am curious about the following points, and request you to shed light:
(1) Arguably, some of the most `beautiful’ spaces are homogeneous spaces G/H. A lot of physics is indeed done on such, with flat Minkowski space being an example. Representation theoretic methods go furthest for these spaces. However, the actual cosmological spacetime is said to be quite inhomogeneous, riddled with blackholes which are far from symmetric, and has a complicated global topology. (This will have to be called `ugly’ if beauty is understood via symmetry.) This has two consequences: On the one hand there is no nontrivial global symmetry, and so the role of representation theory will remain confined to what comes via gauge theory (via representations of the structure groups of some principal bundles on the spacetime). On the other hand, there is no reason to believe that the cosmological spacetime admits a good enough global foliation by spacelike hypersurfaces to enable the posing of physics problems in the familiar evolution (on a state space) format that you alluded to in your answer to my comment.
(2) It is interesting to see from your extensive quotation that Dirac recommends an approach to physics via advanced mathematics. I remember reading somewhere that Dirac never recognized in his writings the difference between a hermitian operator and a self-adjoint operator, an important distinction when the operators are unbounded. Dirac’s braket notation has no room for the domain of the operators, which is of importance here. As is well known from 1930’s, the definition of the braket of two unbounded self-adjoint operators will need us to apply the spectral theorem for unbounded self-adjoint operators, as the intersection of their domains can be too small. How does this square with Dirac’s recommendation not to shy away from using whatever mathematics that is needed for the physics that we want to understand? Einstein managed to learn the differential geometry that he needed, so what kept Dirac away from functional analysis?
(Note: I am not a professional physicist, and I will understand if you regard the above as unsuitable for your blog for whatever reason. In that case, I will appreciate a private answer. Thank you.)
A small correction: I should have said that (as is well known) the difficulty in defining the braket [A,B] in a straight forward way as the difference AB -BA is that the inverse image by one operator of the domain of the other operator may be too small, that is, it can fail to be dense in the Hilbert space. This is where one needs some more functional analysis, namely, the spectral theorem for unbounded self-adjoint operators and Stone’s theorem.
It looks like some survivor bias with having amounts of subjectively pretty theories, but only mentioning the very existence of the correct one. I understand though that Dirac would fall for it after the distrust he had to withstand before positrons got confirmed. But hey, if a theory is already successful on some stuff, one can cautiously trust it elsewhere too.
Nitin Nitsure,
1. I don’t know what Dirac would have said about that sort of problem with the usual formalism. I think he was just referring to the way the formalism deals with the simplest situation.
2. Dirac was arguing for the significance of mathematical beauty, not for advanced mathematical technology, especially not for the importance of analysis involving precise definitions and rigorous proofs.
It seems to me to be something of a category mistake to attempt to refute Dirac’s argument for beauty with a cry for falsifiability or something. It seems manifest that experimental testing/falsification is how a theory must ultimately be evaluated, but beauty can be a very good guide as to what to work on in the first place; or at least, if one wants to take it to be so, who can complain?
(Of course, nothing about the need for experiment truly seems “manifest” after string theory, and admittedly, people other than Peter or Dirac have tried to put the latter’s argument into service for ditching falsifiability altogether, so the category mistake may be a natural one at this point in history….)
Shocking: the years when Scientific American was a serious magazine. The article is a must read, indeed.
String theory was never an example of mathematical beauty, unlike, say, the Dirac equation. The notion that string theory is beautiful is yet another part of the hype that’s surrounded the subject for almost 40 years.
Was beauty really the guiding principle when Dirac discovered his equation? I always thought his most important motivation was that he was looking for an equation with only a first-order derivative in time.
It was discovered (relatively rapidly, if I remember correctly) that his equation was covariant (as it should be) and that half of the solution, if I may say so, reproduced the Pauli matrices for electron spin. What Dirac suggested for the other half, his hole theory, can hardly be described as beautiful.
So if you look at the genesis of the Dirac equation, the prime motivation seems to have been not beauty but physical intuition (that first order derivative in time). Acceptance that the equation was interesting came because it was related to experimental results (electron spin). The hole theory was rather ugly, but of course, later, positrons were discovered.
I’m not denying the Dirac equation is beautiful and immensely powerful, but I’m not certain that he pursued it because of its aesthetical qualities.
Pingback: Non-empirical Physics | Not Even Wrong