Meltdown at CERN

Besides the meltdown in the financial markets, perhaps of more concern to physicists is a meltdown at CERN, specifically in the connection between two magnets during powering tests being conducted at 11:17 last Friday the 19th. Here’s the daily report from that day on the LHC beam commissioning site. By the next day it had become clear that it would be necessary to warm back up the sector to deal with the problem, leading to this press release. It was immediately clear that doing this would take at least a couple months, and make it impossible to have physics collisions at the LHC this fall. Today, a new press release confirms this:

The time necessary for the investigation and repairs precludes a restart before CERN’s obligatory winter maintenance period, bringing the date for restart of the accelerator complex to early spring 2009. LHC beams will then follow.

The press releases refer only to a “large helium leak” and the failure of an electrical connection, making the problem sound rather minor. I tried contacting a physicist at CERN to find out if they had any more information, but was told that he or she was under instructions not to discuss anything about what had happened beyond what was in the press releases, and that CERN was specifically concerned that information might show up in blogs. This policy seems to be being unevenly enforced, since today the Everything Blog carries the following report:

I was in a meeting at CERN when someone ran to the front of the room with a computer, then after letting the speaker finish and setting up the computer to project the press release/e-mail (agonizing moments: it was a Mac), they let us know. I was worried that the sector with the helium explosion had collapsed like an old mine— there were rumors going around that this is a weak point in the tunnel. (Of course there would be such rumors.)

I have learned a few more graphic details about the event in the last few days. First off, it was two tonnes of helium, not one. But I’ve also learned that this was a more explosive and dramatic event than I had imagined— helium is fortunately an inert gas, but the temperature gradient caused it to explode violently, probably causing physical damage to the nearby components. And now that section of the LHC is an ice tunnel, maybe with stalagtites hanging down and a Yeti moaning in the distance.

Up until now, the policy at the LHC has been to be quite open about the commissioning process, with detailed technical information provided on web-sites that were freely accessible. For instance, there’s detailed news about how beam commissioning was going up until the accident available here. Going forward, it will be interesting to see how CERN deals with the problem of letting not-so-good news out to the public. Just about one year ago, I wrote about some earlier LHC problems, and it has always worried me a bit that having a huge LHC publicity onslaught before the machine was actually ready and working might not have been a good idea. Going through the often painful process of solving the problems likely to show up in a project of this scale may provide a different education of the public than the one people were hoping for.

Update: Yesterday there was a LHCC meeting at CERN, broken up into an open and closed session. The slides from the talks are available here, and give a wealth of information about the state of the machine and detectors, and what they were able to accomplish during the short period that they had a beam. There’s no info about the accident in the slides, but video of the talks is here, and at the end of the talk by Lyn Evans (9-18min into the video) he gave a report on the accident and answered questions. The problem has been traced to an electrical fault in the magnet busbar, and bizarrely occurred during the test of the last circuit of the last sector that was being commissioned for 5 TeV operation.

Evans says that the machine will be down until “early spring”. Before the accident, the plan had been to bring the LHC back up in early June, after various work needed on the injection system. The current plan is to try and get this work done instead this fall, so that they can start up more quickly in the spring.

This entry was posted in Uncategorized. Bookmark the permalink.

27 Responses to Meltdown at CERN

  1. Pingback: An agorà of education and scientific communication ? « A Quantum Diaries Survivor

  2. Eirik says:

    Obviously this is the effect of the anthropic principle! In all the universes that the LHC were functioning and crossed it’s beams, black holes were created which swallowed the earth. Since we are all alive and see the LHC malfunctioning, this is evidence that miniature black holes can be created at LHC. This is really the first experimental data the LHC have produced. It’s amazing how much new good science this anthropic prinicple makes possible!

  3. wb says:

    The failure was in an interconnect between two magnets that caused a fire. Ultimately more than 20 of the superconducting magnets quenched. The good news is that the quench protection systems operated properly at a scale never tested before. This gives good confidence about magnet protection. The unknown question is why the interconnect failed – a change badly done connection,a design flaw? What needs to be done is to warm the sector and inspect. Based on what one sees,one may decide to warm everything and inspect the entire machine.

    This not such a simple decision. Remember the rf-fingers that had problems. These fingers have a potential for damage every time the machine is cooled. Therefore one tries to avoid thermal cycles unless necessary

  4. Observer says:

    I wonder how cold it got when the helium expanded?

  5. Coin says:

    I am expecting serious questions in the press and worldwide demonstrations for accountability over the question of whether it is possible that the operation of the LHC could, in fact, create Yetis.

    Remember: There is no scientific theory that can definitively prove that the LHC will not result in the production of Yetis.

  6. calvin says:

    It is being rumored that the LHC in it’s test run has already produced evidence of susy. In fact, it produced the susy partner of a black hole, which instead of swallowing up the world, swallowed up Wall Street. In fact, the susy black hole – also known as sblack shole – a.k.a. Black-Scholes, for convenience of pronunciation, with the ‘c’ added to confuse people – has been around for a while.

    The rest is silence.

  7. csrster says:

    Eirik, that may be the best comment in the history of this blog.

  8. Marco says:

    … it will be interesting to see how CERN deals with the problem of letting not-so-good news out to the public…

    Concerning this point, yesterday during a collaboration (ATLAS) meeting we were formally asked not to give to the press anything that is not a CERN official statement (and this is somehow bounding at least for the CERN employees, we sign something about information disclosure with the contract). I guess this says something about the way CERN plan to control the information leak in the next months…

  9. DB says:

    I can understand that they need to inspect the failed parts and precisely determine the cause, and that premature speculation on blogs by Cern employees could be used irresponsibly by the media. But then they need to be completely open and transparent. As soon as is practicable. Just as they were when the faulty Fermilab components failed last year.

    I have no reason at this stage to believe they will act differently in this case.

  10. Jack Lothian says:

    The media will do what they always do. Thinking that one can control the media is extremely foolish. As word gets out that CERN is trying to “control” the media’s access, they will get more aggressive & hostile. From the media’s viewpoint this is a declaration of war & they will now go to greater lengths to root out the real story.

    I see this as a serious management mistake. Some people higher up are getting very nervous about their job & prestige and they are panicking. It is a normal human reaction under such conditions to try to control everything. Clamping down like this occurs all too frequently when the shit-hits-the-fan. It usually makes things worse but we do it anyway because it makes us feel safer.

  11. Peter Woit says:

    Marco,

    Thanks. The problem with the “nothing to the media except what is in a press release” policy that the LHC collaborations are instituting is that, taken literally, it basically shuts down all blogging by LHC scientists. Blog postings are, unavoidably, communications to the media, since they can read them. If the only thing you can put in your blog are CERN press releases, why bother blogging about your work?

    Funny, but this policy either wasn’t in place or wasn’t enforced until there was some bad news. De facto, the policy seems to be that LHC bloggers can discuss good news as much as they want, but not bad news, Also, the collaborations are mixing one legitimate problem (their conventional concerns about discussion of preliminary data), with a very different one, that of the public perception of how well the LHC project is going.

    I think it would be appropriate for CERN to make sure that bloggers working there clearly distinguish between when they are speaking for themselves and when they are speaking officially for the lab, but beyond that I don’t think it’s a good idea to do anything other than encourage the dissemination of accurate information about how things are going.

  12. Pingback: LHC: Hawking v Higgs « Antimatter

  13. Pingback: This is not sensitive information! « The Everything Seminar

  14. Jim Pivarski says:

    No I am not releasing more than what is publicly available! (This is Jim from the Everything Seminar.)

    Please see http://cornellmath.wordpress.com/2008/09/24/this-is-not-sensitive-information/

  15. insider trading says:

    I totally agree with Jack Lothian. This is indeed a management mistake, the way I see it. Compare it with a totally open view: everything open from day one after the incident. Where is the trouble ? They show what they are doing. If they do the best that can be done, things are okay. Instead, they really look like they have something to hide.

    I stay anonymous but I am a CERN experimentalist. By the way, beam operations will not restart before the start of spring, most probably April. Then there will most probably be a longer-than-planned run at 10 TeV.

  16. Will says:

    Peter, Why are you always so negative? Why?

  17. Peter Woit says:

    Will,

    This isn’t a really good example to accuse me of negativity. There’s no way to deny that this is a discouraging piece of news. Writing a positive posting about it would be kind of idiotic. As for whether I’m negative about attempts by people at CERN to shutdown discussions on blogs by physicists that they might find inconvenient, I guess I’m a pretty negative guy when it comes to that.

    On the whole, one goal of this blog is to put out accurate information, especially in cases where there’s a lot of hype going on. I guess that can be thought of as a “negative” activity, but I think it has its value.

    Personally though, I can assure you that I’m mostly a quite cheerful and optimistic sort, very happy with life. Maybe I compensate for this with the blog….

  18. Sara says:

    @ insider trading: but do they really have anything to hide?

  19. Pessimist says:

    What if it emerges that CERN management cut some corners so they could have “Big Bang Day” on schedule?

    Barring that, the whole incident has been poorly handled. If these guys had a clue, they would have called the BBC back to inspect the damage. It’s rumored to be quite spectacular, and pictures will undoubtably hit the web. Instead of a media bonanza, with the public engaged in the discovery process, it could become a sleazy scandal. Perhaps Big Science needs a good house cleaning.

  20. amba says:

    Moral: do not announce your pregnancy until you’re past the danger period for miscarriages.

  21. chris says:

    well, i would call this a first successful LHC prediction 🙂

    http://arxiv.org/abs/0802.2991

  22. Steve Myers says:

    I’m confused: didn’t any one test for faults before power up? Was that too difficult to do? I have been involved in many startups of high power & complicated equipment, malls, office buildings, and power was never applied without extensive tests. Phase to phase, phase to ground, bad connections, etc. faults should be caught in initial tests and inspection. Clearly there’s something I don’t get about this accident.

  23. Peter Woit says:

    Steve,

    From what Evans said, this happened when they were ramping up from a current corresponding to 5 TeV to one corresponding to 5.5 TeV. This was late in the game, after extensive testing, some kind of problem that only showed up at high currents.

  24. Pingback: Where the heart beats « A Quantum Diaries Survivor

  25. Pessimist says:

    The official report is out.

    https://edms.cern.ch/file/973073/1/Report_on_080919_incident_at_LHC__2_.pdf

    Thirty magnets out, 6 tons of He lost, soot in the beam pipe, ++

    The good news is ” the official inauguration ceremony that will take place [ ] in the presence of the highest representatives from the
    CERN Member States, representatives from other communities and authorities of the countries participating in the LHC.”

  26. Covariant says:

    From the report

    “These forces displaced dipoles in the subsectors affected from their
    cold internal supports, and knocked the Short Straight Section cryostats housing the
    quadrupoles and vacuum barriers from their external support jacks at positions Q23,
    Q27 and Q31, in some locations breaking their anchors in the concrete floor of the
    tunnel. ”

    Wow

  27. Benni says:

    At least, from what is in the report, it seems that one can take appropriate measures that such a chain reaction of events won’t happen again.

    I wonder if the whole LHC must be warmed up when they make the necessary modifications on the other magnets to prevent such an accident in the future,

Comments are closed.