In this week’s Nature, Abraham Loeb, the chair of the Harvard astronomy department, has a column proposing the creation of a web-site that would act as a sort of “ratings agency”, implementing some mathematical model that would measure the health of various subfields of physics. This would provide young scientists with more objective information about what subfields are doing well and worth getting involved with, as opposed to those which are lingering on despite a lack of progress. Guess what Loeb’s main example is of the “lingering on” category?
In physics, the value of a theory is measured by how well it agrees with experimental data. But how should the physics community gauge the value of an emerging theory that cannot yet be tested experimentally? With no reality check, a less than rigorous hypothesis such as string theory may linger on, even though physicists have been unable to work out its actual value in describing nature…
Theory bubbles
The study of the cosmic microwave background provides an example of how theory and data can generate opportunities for young scientists. As soon as NASA’s Cosmic Background Explorer satellite reported conclusive evidence for the cosmic microwave background temperature fluctuations across the sky in 1992, the subsequent experimental work generated many opportunities for young theorists and observers who joined this field. By contrast, a hypothesis such as string theory, which attempts to unify quantum mechanics with Albert Einstein’s general theory of relativity, has so far not been tested critically by experimental data, even over a time span equivalent to a physicist’s career.
The problem of course is that of deciding who gets to make evaluations of what’s a healthy field and what isn’t. People with a lot invested in a dying or dead subject have strong incentives to misrepresent the situation (see, for example, the famous Monty Python Dead Parrot sketch). Loeb implicitly compares the current situation with string theory to that following the financial crisis, which was worsened by the ratings agencies assigning AAA ratings to debt not far from default.
Senior scientists might seem the people best suited to rate the promise of research frontiers. But too many of these physicists are already invested in evaluating the promise of these speculative theories, implying that they could have a conflict of interest or be wishful thinkers. Having these senior scientists rate future promise would be akin to the ‘AAA’ rating that financial agencies gave to the very debt securities from which they benefited. This unseemly situation contributed to the last recession, and a long-lived bias of this type in the physics world could lead to similarly devastating consequences — such as an extended period of intellectual stagnation and a community of talented physicists investing time in research ventures unlikely to elucidate our understanding of nature — a theory ‘bubble’, to borrow from the financial world.
The problem of how to get scientists and academics to rigorously evaluate what works and what doesn’t is a difficult one. In particle physics, success has led to making progress harder to come by, so just noticing a lack of progress at the rate of earlier times is not enough. String theorists are right to point out that developing ideas to the point that the theory can be compared usefully to experiment could be a difficult goal that may take a long time to get to. They’re wrong though not to acknowledge the fact that they’re not getting closer, but rather farther and farther away. And misrepresentations about the state of a subject can victimize young students and researchers, induced to devote crucial parts of their lives to something not worthwhile.
I’m rather skeptical about Loeb’s faith in mathematical models to provide objective guidance. The AAA ratings assigned to dubious mortgate-backed securities were the product of mathematical models, defective ones. If you let me design the model, I can come up with one that will justify whatever conclusion I want. In the end, outcomes will depend on the quality of the judgment and decisions of those the community chooses as its leaders. Throughout academia, bad ideas live on, and good ones don’t get the recognition they deserve. At the same time, in many fields those put in positions of responsibility live up to them and often do a remarkable job of countering the forces promoting stagnation as well as providing a positive vision that drives real progess.
As for Loeb’s idea about a web-site where young scientists could go to get information about the health of a field, I remain skeptical about prospects for one that implements a mathematical model. However, a website devoted to honest and informed discussion about what is going on in a field and whether it is healthy, providing a place for students and others to listen to and participate in debate, helping them make up their own minds, seems to me an excellent idea…
Update: I just noticed that Loeb had a paper on the arXiv last year spelling out his proposal in more detail.
There is an interesting analogy here between analyzing stocks based on recent price movement (technical analysis), or based on earnings (fundamental analysis). In the long run fundamental analysis is what works, but in the short run in may not.
It seems like funding and publications are like short term price movements while validation by experiment is more like earnings. I’m not sure about hiring and promotions!
Evaluating the health of a research field? I agree with you, how do you do that. It seems to me great advancements in history were made when some egg head told somebody else “This can’t done.” Indeed, the best advice might be to tell a young physicist to go into the field that such a website says is dying.
The Nature article is behind a paywall, but from your quotes and summary there seems to be not inconsiderable chutzpah for the chair of the Harvard astronomy department to say both that senior scientists are not the best judges of research frontiers and by the way string theory is overfunded.
piscator,
The “chutzpah” label has been previously applied to Loeb in this context:
http://www.bautforum.com/showthread.php/120202-Physics-Chutzpah
For another sort of argument from him, about encouraging people to also spend time on some riskier non-mainstream work, see
http://arxiv.org/abs/1008.1586
I think it is ridiculous to think that young scientists need help in ranking fields of study. Typically, they’re 20+ and have been thinking a long time on it. Grad students I have known, including future string theorists, know perfectly well the prospects. Many go into string theory anyway, for other reasons (one being that it interests them). Maybe a bad decision but I don’t see a coverup of the truth.
“I advocate the need for a website operated by graduate students that will use various measures of publicly available data … to gauge the future dividends of various research frontiers. The analysis can … aim to alert the community of the risk from future theory bubbles. ”
Peter, you are not a grad student but I think you have already implemented to some extent his proposal for string theory.
I’ve discussed the problem of bubbles of nothing in academia here and here for example. If Loeb had read what I wrote he might have understood that his proposal isn’t going to solve the problem. The main problem is not that we need a better way to rate on the promise of a research field or project. The problem is that presently scientists have insufficient incentive to accurately rate it to begin with. I referred to this as “external pressures” that essentially prevent an accurate judgement. As a result, the system, as it is presently organized, doesn’t optimize for “most promising research.”
To see what I mean, consider the following example that I personally find the most troublesome, though there are other problems: It is hard if not impossible for a scientist to change fields after, say, the 2nd postdoc. At that time, you’ll be desperately looking for a permanent position and trying to get in grants. Both will only work if you have a track-record in some field, and that field is in addition considered promising. Taken together, this means researchers will go around and advertise their research field because they’re stuck on it. You can’t sell your publication’s keywords and buy new ones like you buy stocks. It’s your history and the way the system presently operates it basically *forces* people to create bubbles of nothing because otherwise they’ll end up unemployed!
You can find more details and other examples in my posts. More recently, in my post On the importance of being wrong. we discussed the problem that admitting on having misjudged the promise of a research project is not only not rewarded but actively punished in the community, and wishful thinking is one of the most common biases affecting science.
A “rating agency” isn’t going to do anything about what’s the actual origin of the problem. In fact, if people came to believe the ratings it would make the problem even worse, because it contributes to a rich-get-richer trend (if there are many people who work in a field there are many people who have all reasons to rate it up).
I think Bee makes a very good point. New graduate students have the freedom to choose their field, but tend to be relatively naïve when they do so (I know I was; I couldn’t possibly accurately evaluate the worth of, say, string theory, at the time). It’s practically impossible to change fields after a year or two of graduate study, because one must write a thesis before running out of funding!
There may then be some freedom to wander during the first postdoc or two, but there are overheads in time and effort associated with branching out, and one is always concerned about writing enough papers, lest the next job not be offered.
Speaking from a researcher from another field, I had my concerns who string theory develop. I think the comparison with an economic bubble is quite appropriate. One of the biggest problem faced by younger scientists are the need to publish and to be seen “doing”/”producing” something. The string theory “bubble” is self-reinforcing; the bigger the bubble gets, the urge to even make it bigger is even higher due to career pressure.
The biggest difference with an economic bubble is how the bubble could be burst. Real economic bubble burst as balance sheets and debt obligations implode, and people and institutions running out of money. I wonder what mechanism can burst an “academic bubble”. I am not sure research grant cuts can actually burst an “academic bubble” – in the ends… it is experimental/field work that cost the most, not theoretical and computer model work. The folks inside the bubble field won’t burst the bubble themselves, of course – for the interest of their own career.
What I think may burst the bubble is a major breakthrough in experimental physics that allow a highly competitive and creditable theoretical work to develop.
Bee,
Thanks for the links to a lot of good discussions of this topic. I agree that young scientists are in their most vulnerable position, with a lot invested in a particular field, and leaving it very dangerous, at the time they don’t yet have a permanent position, but need to get one soon. This is the time in people’s career that they are least likely to be able to change research topic, so it’s not surprising that few do.
What is more surprising is how few people are willing to give up on a failed research program and try something else AFTER they have a permanent position and (at least typically in the US), lifetime tenure. The whole point of the tenure system is to make it possible for people to make this kind of change, but few take advantage of this.
This shouldn’t be too surprising. The tenure granting process actively selects those who are either not particularly curious about topics irrelevant to their immediate research or are sufficiently disciplined to suppress any such curiosity. A young condensed matter theorist who devotes a considerable fraction of her time to work on quantum gravity is unlikely to obtain tenure. If a newly tenured professor had in fact been suppressing a burning extracurricular curiosity for years, it is likely that this habit would have become permanently become ingrained.
A better analogy IMO would be Technology Adoption and Meme Adoption. Both involve beliefs of what is true as well and they have many dynamics in common, specifically in term terms of human networks and in terms of nonlinear thresholds or tipping points of believe or “paradigm” acceptance.
Rather than “control” these, which generally isn’t possible, it may be enough (or better) to simply characterize and comprehend where an idea or theory is in its adoption curve. Not all technologies are adopted and so too not a ideas are adopted. And also some technologies simply are “ready” adoption.
In technology there is the “20 year rule” which basically is the empirically observed time between discovery of an idea and its broad acceptance. When a technology isn’t “ready” it typically takes a generational replacement to bring it up for consideration again so often technologies take 20, 40, 60, etc. adoption intervals.
I suspect theories in science follow a similar pattern for similar reasons.
Peter, Avi gave two talks one at KITP and another at CFA talking about these ideas in detail. Anyhow I know from personal experience that its very hard to work on independent ideas (without any sort of encouragement/feedback). I know someone who
has been working seven years tirelessly on a modification of GR, having refereed
publications and inspire of not getting paid a cent for his research, or getting
seminars/conference invitations
The irony is remarkable
Tenure is supposed to encourage independent thinking.
But anyone sould see what would happen.
That it would coerce people to think the way the tenure granters tell them to.
Was it always thus?
And was that the purpose all along?
And now, how do you get rid of a system that is so corrupting?
Most researchers do not recognise the simple fact that to admit “I was wrong” means that other fellows might not follow the same path, thus lots of funds, energy, time, etc. could be spared.
I was even considering setting up a “Wrong Nobel” fundation – rewarding those who have the courage and honesty to say aloud “I was WRONG”!!
Unfortunately, the EGO is stronger than any award… (especially with matematicians – sorry Peter :)))
Pingback: Should we have a ratings agency for scientific theories? « mathbabe
Bee said: “It is hard if not impossible for a scientist to change fields after, say, the 2nd postdoc. ”
I think it is hard to change fields even after a PhD (and this includes subfields of physics), unless one is going into the industry (then one can do whatever one wants to). For one, there is a steep learning curve, and two, there are already many PhDs in the new field who will be in the queue for post-docs.
The only way I can think of someone changing fields (or a subfield) is by getting a second PhD and this time in an area chosen with better judgement—born-again academic.
The obvious answer is that science must go back to recruiting the independently wealthy types like Boyle, Darwin, Noyes, etc. who have little need for external funding. Theorizing is cheap, so someone with a good inheritance could do good work without any need to deal with these institutional problems. Selection from such a narrow group would of course restrict the talent available, but a) you could keep the existing system alongside it and b) those wealthy pseudo-amateurs did a pretty remarkable job over a long period of time.
The implementation barrier is that being an independent scientific theorist is no longer a “cool” thing for the idle rich to aspire to.