4m Blanco Glycol Chiller system: history, recent events, present status and plans Brooke Gregory, Esteban Parkes, Chris Smith 22 May 2012 Background Information: Original configuration (from ~1975) of two glycol cooling loops: Floor loop: 10 T chiller (for cooling M floor; later its use was expanded to cool hydrostatic bearing oil, primary mirror as well) Building loop: 30 T chiller (air conditioning for computers, console room, other working spaces, etc.) Chiller system upgrade: (Jun- Oct 2011): Floor loop: 30 T (refurbished and moved from the Building Circuit) Building loop: 40 T (new Trane chiller replaced 30 T unit for cryocooler compressors, air- conditioning for Coude instrument lab and cleanroom, in addition to existing demands for computers, console room, etc.) Glycol plumbing: Glycol from each of the two chillers is circulated to the telescope via two nominally independent circuits, each with its own pump. The utilities building, which houses the pumps, also provides a system of valves and interconnecting pipes which permit: Re- routing the glycol so that a spare pump can be brought into service to replace the pump in either circuit, Connecting the circuits in series so that they may be served by a single pump and a single compressor, providing backup (with reduced capacity), in the case of a chiller failure. DECam and the glycol system: The DECam instrument is cooled by a continuous stream of liquid nitrogen pumped in a closed loop from a storage vessel. The nitrogen vapor produced in the cycle is re- liquefied in the vessel by cryocoolers. The cryocoolers use pressurized helium as the working fluid, also operating in a closed loop. The helium compressors recompress the helium returned from the cryocoolers. The compressors run continuously and are cooled by glycol from the Blanco glycol plant. Consequently, continuous operation of the glycol plant is important for continuous operation of the instrument. (The liquid nitrogen system is, however, tolerant of interruptions of the glycol system of several hours, so, in the case of failures, time is available to reconfigure the system, switch to backup compressors or chillers, without interrupting the operation of DECam.) This document reports a recent failure of the glycol system, analyzes the causes and identifies areas where the current system can and must be increased in reliability.
Analysis of the Events of Mar/April/May 2012 28 Mar 40T chiller: This chiller has two completely independent refrigerant circuits, each with two compressors. The two circuits are activated simultaneously only when cooling demand requires them. On this date, low refrigerant pressure was detected in circuit 2. The demand in this period was relatively low and the second circuit was not required for operation but steps were taken to get the circuit recharged. We did not have the refrigerant or the charging manifold when this occurred. We had asked the air conditioning contractor Creavi to quote on periodic maintenance on 19Mar. When we detected the low- pressure we insisted with Creavi and told them of the more urgent need for the recharge. Their quote for the repair work (US$4500) was excessive, so we opted to acquire the refrigerant and manifold ourselves. We received the material on 9Apr. One of our technicians was instructed to recharge the system, but hadn t gotten it done before 17Apr due to other duties. Apart from this background problem, both chillers were delivering the cooling required at the time. 17Apr - At about 22:30 [UT], the 40 T chiller had been working well controlling the temperature of the glycol at about 5 C. At that time the helium compressors were turned on by Herman Cease. Immediately, (within a few minutes) the glycol supply temperature started to rise. After about half an hour the compressors were turned off. See the second plot in the technical appendix. The near simultaneity of the chiller shutdown with the compressor turn- on is clear evidence that the load represented by the compressors caused the shutdown. Presumably this is because the chiller was working on only one circuit. As long as the load did not rise above the limit that could be handled by the single circuit, the chiller continued to operate reliably. Once that limit was exceeded, the chiller failed, not simply to be able to maintain the control temperature but by shutting down. Subsequent misbehavior of the chiller controller suggests that the controller may have responded incorrectly: there is no obvious reason why the chiller should have shut down. Steady operation of the 40 T chiller was obtained late in the day of 19Apr and compressor operation began. Shortly after noon on 20Apr, >12 hours after starting the operation of the DECam cryocoolers, during which time the supply of glycol and the operation of the cryocoolers was normal, an increase in the glycol temperature indicated that the chiller had failed again. Based on experience since that event it is becoming increasing clear that the 40 T chiller controller has become unreliable. 18Apr - The backup procedure in this case is to replace the 40 T chiller. The 30 T chiller is switched from the floor glycol loop to the building glycol loop. That was undertaken. However two unrelated problems emerged: a.) In part due to recent changes in the glycol systems and transitions in staff responsibilities, the Telescope Operations staff did not have the knowledge to reconfigure the valves to switch to the other
chiller. Enrique Figueroa was contacted for information, and a former CTIO glycol system technician, Eudocio Guerrero, was pulled from another project on the on the mountain to make the switch. b.) Once connected to the other loop, the 30 T chiller failed to run. After unsuccessful attempts to get help from the Creavi tech who had worked on the refurbishment of the 30 T chiller, the problem was traced to the glycol flow sensor. This sensor should have been detecting flow in the circuit flowing through the 30 T chiller. The pump was on, so Guerrero guessed that the flow sensor was connected to the wrong glycol circuit. This turned out to be the case: turning on the pump for the 40 T chiller permitted the 30 T chiller to start! Once the 30 T chiller was working, it was observed that its capacity was lower than expected. It was unable to reach its 5 C setpoint, staying at about 8 C. This is low enough to work but was indicative of a problem. 23Apr Enrique Figueroa examined the status of the 40 T chiller, out of service since its failure. He observed the Code 8 boot error with which the chiller had failed. (Repeated attempts to recover from the Code 8 boot error by resetting had failed to resuscitate the chiller.) At this time, Figueroa did a reset and the machine started normally on Circuit 1 and ran for 2 hours without failure. Circuit 2 was still without refrigerant and prevented from running. He and Guerrero proceeded to recharge Circuit 2. They disabled Circuit 2 to prevent interaction with the operation of the other circuit. They evacuated the refrigerant volume of Circuit 2 (28 inches of vacuum) and then installed the full contents (11.3 kg) of the new cylinder of refrigerant (R- 410A). (They observed that when the pressure was lowered in Circuit 2, it caused the operation of Circuit 1 to halt though there should have been no interaction between the Circuits. This was a surprise.) After recharging, the refrigerant pressure was more than sufficient for continuous operation, but a total of 14.5 kg of refrigerant is specified in the manual. An additional supply of refrigerant has been ordered. 24Apr Meanwhile, engineers from Climacor were invited to the mountain to diagnose both machines. (Climacor is a company, with a presence in La Serena, which is used by Gemini for chiller maintenance.) They found the 40 T chiller to be operating normally, and diagnosed the 30 T chiller as having a small refrigerant leak. The observatory is arranging to have Climacor return to correct the problems which are limiting the performance of the 30 T chiller Conclusions: The root cause of the failure of the 40 T chiller on 17Apr was almost certainly an intermittent fault related to the repeated faults seen in the control system since then. The Observatory is in contact with Trane and Climacor to identify the best course for implementing a repair (probably a replacement of the control module).
Whatever the cause of the failure of the 40 T unit, the second 30 T chiller is capable of providing the cooling required by DECam and is intended to be available as a backup in the case of a failure of the 40 T chiller. The attempt to switch to the 30 T chiller was hampered by several conditions: A previously unidentified wiring error in the (recent) refurbishment and installation of the 30 T chiller caused it to fail to start. The staff present at the time of the failure was not fully trained in the required reconfiguration of the glycol lines. The performance of the 30 T chiller was lower than normal. Though it did not play a role in the current failure, the 40 T unit was itself in a less than normally robust condition because of the loss of coolant in a second, redundant, chilling circuit. This condition was not repaired in a timely fashion for reasons of lack of tools and material (refrigerant), and lack of staff to make the repair. Improvements required: In the absence of in- house staff doing maintenance & repairs, establish contact with locally available (La Serena) contract technical expertise. Rapid response capability is important. Train of mountain- resident staff in backup procedures. Test (periodically) backup procedures for responding to failure. Notwithstanding the present incident, the glycol system is very reliable. Therefore it is important to stage failures in order to test the capacities of the backup systems and to test and/or refresh the readiness of the staff in implementing backup procedures. Provide adequate material and tools for critical repairs. Status: 11May 40T chiller: Working normally, Circuit 2 recently recharged with refrigerant. Control module is evidently very unreliable. Though the unit is now running, restarting is difficult at best. The Observatory is in contact with Trane and Climacor about repair. 30T: Working normally but at less than normal capacity. The Observatory is arranging with Climacor for repair. Plans: Complete repair/maintenance on the 30T unit to bring it up to full capacity. Timeframe: ASAP Complete repair/replacement of the 40T unit control panel. Timeframe: ASAP Identify local profession AC equipment contractors to do periodic maintenance of both 30 T and 40 T chillers. Trane, Creavi and Climacor are candidates, although Climacor has local representatives who
may be able to respond more rapidly. We have a reasonable quote in hand from Climacor, but AURA regulations require at least three quotes, which we are pursuing. Timeframe: ASAP Draw up detailed operations and maintenance program for system by mountain staff Timeframe: by July 1 (including documentation and training) Including: Assign staff from TELOPS/ETS (rather than Facilities as has historically been the case) to 1. Oversee routine maintenance of the glycol system 2. Startup and operate the glycol cooling system. 3. Implement standard backup procedures to assure continuous or near continuous supply of glycol for the critical cryocooler and computer facilities, in the case of a component failure. These procedures are to respond to the cases of A failure of a pump in the glycol circulation system A failure of a chiller in the glycol refrigeration system Train staff: Start- up procedures Chiller failure recovery Pump failure recovery Maintain a complete stock of maintenance equipment: Stock of refrigerant R- 22 (??) and R- 410A Scale for weighing refrigerant in charging operation Charging manifolds Vacuum pump Spare control module for 40 T Trane chiller? Timeframe: August 1
Technical Appendix: Record of supply and return glycol temperatures (from DECam (SISPI) log file) a) Compressors operating with 40 T chiller. There is a gap in the SISPI record from ~13:00 18Apr - 21:00 19Apr, but according to Esteban's cry for help at 9 am 20Apr, the system had been working with the compressors since the day before. (There is a gap in the data from ~19Apr 2:00 until 19Apr 22:00.) b) Failure (of chiller?) causes compressors to shut down. c) Abandon tests (not clear exactly what precedes this). d) 8 C glycol, compressors not running (30 T chiller cooling glycol). e) Switch back to 40 T (after recharging and successful restart). f) Operate compressors with glycol from 40 T. g) Chiller fails again, and is successfully restarted and compressors are restarted. h) Compressors turned off, 40 T working, but cooldown is abandoned. i) Another chiller failure? This record starts at the beginning of the warmup that starts at 7:30 pm on 17Apr and also affects the temperature in the Coude (witnessed by Observatory sensors). The temperature rises off scale to 30 C! The DECam glycol sensors show the same temperature rise. The compressors were apparently not operating during this warmup, but were turned on after restoring the 40 T chiller and before the restoration of the data logging graphed here. With the
compressors off, there is no differential between supply and return. The warmup was caused by the initial failure of the 40 T chiller.
Zooming in on incident in the evening of 17Apr: