Piping
Trust Us, it’s P91
At one of the world’s most spectacular power-desalination facilities, the systemic weld defect discovery resulted in a soul-crushing realization that some contractor’s self-interest has no moral boundary. If they think they can get away with something, they will certainly try, regardless of the impact to a client/owner and even themselves. Even more troubling, there really is little financial gain with some of these actions.
What started as an investigation into welding defects not properly identified in QC documentation, evolved quite dramatically when a P91 (9% Cr) hot reheat steam piping failed radially at a fabricated wye (fitting) near a steam turbine. Luckily, the failure was not catastrophic, being about 12” long, and allowed a prompt orderly unit shut down with no injury.
The fitting was cut out, and boat samples of the crack area were sent to a contractor lab and owner lab for metallurgic analysis. In the meantime, the head scratching started. It was clear early that the weld records had issues, this was visible internally. However, while waiting for metallurgy results, a... Read more
Savings from less rework and use of less skilled welders. Maybe $1mm. Cost to reweld all the large-bore in a huge power plant…??
Repair cost to the EPC contractor was immense, but the cost to the Owner was far greater due to the extended outages which persisted many years while O&M was trying to run the facility. This was well over a $100mm problem to the Owner the 1st year alone.
Weld Record Anomalies
Our default perspective is positive, and we prefer to be able to accept a story or data (documentation) as reasonably true even if we don’t believe all of it. We became gob smacked as a seemingly curious paperwork question morphed into extreme seriousness, the deeper we dug.
As an introduction, a review was performed of a very large facility shortly after COD. One aspect of the review was welding code compliance (ASME). This review was performed for all Section 1, BEP, and NBEP field welds for all the units/blocks. This started as a documentation review.
Initial findings were curious. Of the 25,000 large-bore field welds, only one (1) documented repair (R1) was indicated. Otherwise, the records showed no defects in any visual or radiographic inspections. Please keep in mind, the labor in this location was less than average and the work conditions extreme. In a western context, a good site welding program reject rate is perhaps 1-2%. In this situation, 4% would have been an overachievement, but 0%. This did not smell right. But all the paper appeared fine, except…
In one... Read more
Savings from less rework and use of less skilled welders. Maybe $1mm.
Repair cost to the EPC contractor was immense, but the cost to the Owner was far greater due to the extended outages which persisted many years while O&M was trying to run the facility. This was well over a $100mm problem.
Trip To Work
For owners building plants, it is important to have a team that is engaging in construction oversight at the field level.
These are a few examples of a problem that we see often, which should not exist. Further, at least for electrical components (think conduit) it is a code violation to obstruct walkways and traffic areas. OSHA also addresses the issue with regulations. If this is not enough, it should be understood by most anyone, that plant workers need to be protected and nearly every power asset owner has a corporate statement of commitment to health and safety practices.
If labor is not knowledgeable enough to understand this concern in developing regions primarily (but not exclusively), then certainly contractor supervision or quality control should intervene before the work is complete.
In these photos, the plants are past COD, in a few cases over 18 months. These are also high traffic areas.
As a stopgap, O&M groups should walk the plant and add these items to an active punchlist to be sure they get resolved.
At EPR, this is an example of an item our... Read more
None, or perhaps a small savings.
Usually not large but can be left unresolved and an accident can occur. Accidents have two aspects; financial cost and a moral cost.
"Old School" Steam Drains
Keep in mind, this is a new plant in operation about one year, in a location where quality plants are built routinely... Yes, this is a NEW plant built to contemporary codes and standards by a large international EPC contractor.
During EPR's review of the Facility, this system is perhaps the single most compelling example of inferior design in the plant. While not the most cost intensive, the effects are broad reaching. The open trench design belches hot condensate on steel, electrical hardware, and other equipment leaving the plant to look ready for decommissioning. O&M personnel are left to perhaps unknowingly assume that facility condition is not important to the owners; how could it be, look at this? In so many ways this is not consistent with the Facility purchased.
The condensate collection system of trenches and transfer sumps has experienced critical deterioration. Serious failures in the concrete infrastructure include trench through-cracks, cementitious erosion, advanced concrete spalling and scaling, sub-grade undermining, differential settlement, and accelerated... Read more
Perhaps $500k.
Unknowable, but very significant. Violates Environmental permit, among other problems.
Dirt... and Fuel Gas
As background, one critical system in a power plant is fuel gas. Typically, every system is cleaned to a condition that is nearly spotless. This is especially true of fuel gas because it feeds the turbines and duct burners. EPR evaluated a plant that was physically very large. The fuel gas header was 1.5+ miles long, inside the plant downstream of the supplier custody-filters. The system changed elevations, had countless fittings, and pipe size changes. Clearly, this is a difficult system configuration to clean well for an inexperienced crew.
Prior to EPR being involved, roughly 50 warranty claims were raised against the EPC contractor that enumerated roughly 300+ failures of valves, flame scanners, regulators, and really anything in the system with a seat or rubber O-ring. The EPC contractor refused to address the issue on-the-whole with a proper RCA (root-cause analysis), instead dealing with each item as if unrelated.
The attached photographs tell the story. This system is so full of dirt and contamination that it's not defensible to indicate proper, or perhaps any, commissioning... Read more
No Savings. Contractor went through "motions" of cleaning system, just didn't!
Just in terms of Owner down-time for needless repairs this is millions of $'s. Parts, labor, etc...
Failed Expansion Bellows
At a new plant with large-scale desalination operations a recurring failure was noted. Dozens, probably hundreds, of failures of 24” rubber expansion bellows occurred due to incorrect installation.
The summary conclusion in this case is that the contractor provided no training or oversight to piping personnel responsible for installing this engineered device.
Operation of the systems created continuous piping vibration (normal levels) and the environment was quite hot, even the working fluid (seawater) was quite warm so the bellows was probably incrementally more relaxed. However, failure was certainly due to flange-to-flange dimension busts, misalignment of bolt holes, out of square flanges, and an assortment of creative abuse.
Where flange-to-flange distances were not to specification (narrow) the bellows nearly universally failed due to bolting impinging on the rubber outer layer of the bellows.
This defect could have easily been spotted by engaged contractor supervision, or by an owner’s representative, but neither addressed the problem which is easy to see. The... Read more
None.
Unknown. Device mortality will be a continual O&M cost for the life of the plant, unless piping changes are made during outages.
GRP Joint Fail
EPR evaluated a very large facility that had one of the world’s largest desalination operations at around 250,000,000 gallons per day. An extraordinary facility in terms of scale and elegance of design. But separately, there were quality concerns. The photos are about 12 months after COD, so the plant was effectively new. For context, the seawater and freshwater ‘loops’ in this plant were miles long and complex configuration of GRP (glass reinforced plastic) commonly known as fiberglass.
GRP is an excellent product but requires attention to detail when installed. Joints need to be ‘layed up’ with a carefully proportioned two-part resin over a clean and prepared substrate.
In this plant, chunks of GRP started to appear in the process screens. On further evaluation inside drained circuits and elsewhere via borescope inspection evidence of GRP delamination and shedding at the joints was discovered. If there were a few instances of the problem, it would have been noted and dismissed, but this condition appeared in varying degrees in all the units and interconnecting piping.
The... Read more
None.
Unknown. Consider the primary revenue for this plant is water production and a unit outage costs several $X0,000/day. This could get expensive.
Piers, Sort of
Sometimes people ask us how new plants end up completing so badly. EPR is not typically engaged to develop a well-studied opinion about the details of execution failure, so we tend to stay away from speculation and focus on the task of Verification of Mechanical Completion. However, we can suggest with a high degree of confidence that plants that exit construction/commissioning in poor condition are almost never resulting from engineering failures. Yes, sometimes there are oversights and errors in the plans and specifications, but it is rare enough and usually minor enough to allow us to broadly conclude execution problems are at the site.
This is one example. The drawings indicated a pier design with a spread footing to effectively keep the pier in a fixed position when loaded from pipe stress. However, the site decided to precast piers with a greatly reduced footing area, which contravened the design.
This has resulted in about 300 piers that have settled and been pushed out of plumb by piping forces. Further, the depth of cover is not suitable (<250mm) compared with the original... Read more
Some, but relatively minor. ~$25,000.
The cost of replacement is small compared to the damage to rotating equipment and other components.
Rope Hangers
Piping, particularly hot services (Steam, Feedwater, Condensate), are engineered systems. The stresses induced on the piping, fittings, and valves are controlled through purposeful routing, hanger locations, and hangar properties. This is very important to the longevity of system components. Thermal cycles and other transients accumulate stress-induced damage with time leading to failure, if incorrectly installed. Plant reliability and personnel safety are dependent on proper implementation of the engineer's design.
Given their status as “engineered components” it is hard to reconcile the lack of attention-to-detail observed in the installation of supports/hangers in many recently inspected plants. In some cases, the contractors have been so sloppy as to not even finish installing the required supports/hangers.
After a support/hanger is installed, the contractor must also follow up to assure it's correctly setup (commissioned). Usually, this necessitates a QC step to assure the installation is correct, followed by documented verification of the “hot settings”, and then... Read more
Contractor: No savings; hangers usually purchased; lost/not installed.
Contractor: Usually not very costly. Owner: Can be costly, because of the outages caused by even small piping failures/leaks.
Piping Protection
In this instance, the facility was combination of power and desalination on a scale that made it one of the largest in the world. It was routine for staff to use vehicles during the normal course of business due to its size.
In many plants fire systems are usually the subject of some scrutiny related to protection from vehicle damage. However, in this facility there was an extensive amount of process piping and dozens or perhaps hundreds of locations where underground piping emerged near roads to interconnect to above ground processes. This piping was glass reinforced plastic (GRP), or fiberglass. While a great material, it is fragile when subjected to impacts.
After a couple instances of non-destructive damage, the issue of protection was evaluated. In this case the piping was designed to ASME B31.3 and with some additional EPC contract language, it became clear bollards or other means of protection needed to be installed.
B31.3 refers to safety inherent in the piping by virtue of the materials of construction, methods of joining, and history of service reliability. Further,... Read more
Some savings, perhaps on the order for $250,000.
For plants where these processes are a material part of revenue, piping damage can take down a whole block for many days. In this instance one block outage (desalination) costs about $350,000/dy.