The process of restarting a semiconductor facility

But this was no ordinary snowstorm. Several hours into the day, we received notice that our electricity would soon be turned off so that the resource could be redirected for the health and safety of our community facing widespread outages. From there, we faced a significant curtailment of natural gas and water supply to our facilities.

Without basic utilities – power, gas, water – a wafer fab cannot maintain its basic safety systems in order to sustain a safe environment for our employees and our community. Any interruption to these, even for a fraction of a second, causes the fab systems to pause and a full recheck of our equipment and product to ensue. But the termination of all our utilities for an indefinite length of time? That required a complete shutdown of both our facilities and an evacuation of our team (except the critical facilities team members), marking an extraordinary event without any precedent in our 30+ years of operations in the Austin area.

Now that we’ve resumed initial operations, I can look back on what happened that day in February and share my first-hand experience of what the past several weeks have been like for the NXP team, the damage we saw on site and what it’s like to restart production at our facilities after a highly unexpected idle period.

To back up a bit, let me position this in the appropriate context. Two days before we were forced to shut down, the state of Texas and the neighboring states experienced a catastrophic winter storm of a ferocity that no one expected. For the first time in the history of the state of Texas, all 254 counties were under a winter storm warning. Ultimately, this storm resulted in significant loss of life, left millions without power and half the residents of the second most populous state in the U.S. without access to clean running water. The costliest natural disaster in the state’s history, this was truly the storm of a lifetime for Texans.

The storm of a lifetime for Texans – a storm of complexity for NXP

In Austin, our facilities create semiconductor wafers that are shipped to other NXP facilities where they’re tested, assembled and finally sent to our customers. The Austin fabs serve multiple end customers and market segments, with the largest being major parts suppliers to the automotive and industrial and internet of things (IoT) industries.

To make our products, each of our facilities use hundreds of pieces of highly sophisticated semiconductor processing equipment in addition to thousands of critical and complex infrastructure components that ensure our facilities are running properly. We know that our products are used in safety systems in cars and in many cases are crucial to the function of the end product such as automobiles and medical devices.

When we do idle our facilities for scheduled maintenance, we follow a painstaking process to ensure that all our equipment and all our wafer work-in-process (WIP) is protected. Normally, the process to safely idle one facility takes about a week (and the process of restarting takes an additional two weeks). What happens when we are forced to shut down without that careful process? Think of the angry, urgent warning messages you get when you force a USB drive to eject from your computer, alerting you that it wasn’t safely removed. Maybe your files were damaged and you need to check them all. Maybe the drive itself is now unusable.

Now imagine that at a much greater scale, across thousands of pieces of equipment and tens of thousands of WIP wafers, and you have a better idea of what it was like to fully shut down our fabs in an urgent crisis and restart them only weeks later.

At NXP, we anticipate and prepare for emergencies through meticulous, careful contingency planning. We consistently revisit and evaluate our plans to make sure they are up to date, although no one ever anticipated an event like this winter storm. Here’s how those plans worked on February 15 and in the weeks that followed.

Keeping NXP employees and our community safe

Our top priority is always the health and safety of our employees and the surrounding communities. The first thing we evaluated was the well-being of our teams at the fab. Not only were we all dealing with an unexpected situation at work, but we all have families and homes in the surrounding areas that were impacted by the storm. I’m proud to report that every member of our team remained safe throughout the shutdown process.

In this crisis, the first thing we needed to do was ensure that the massive bulk tanks of chemicals, gases and other materials we use in shaping silicon were kept in a safe state, regardless of the duration of this shutdown. When we lost electricity, backup generators were immediately enabled to support the stability of these materials as well as the safety systems for our critical facilities and cleanroom space infrastructure.

Next, we were informed that our supply of natural gas would be reduced to close to zero in support of critical shortages for nearby neighborhoods. NXP then implemented its contingency plan for this scenario and switched over from gas to diesel fuel. We also have a nitrogen contingency plan where we can self-generate nitrogen or convert liquid nitrogen to gaseous nitrogen, which is critical for our operations.

For any planned or unplanned shutdown, our safety protocol requires a team to be onsite 24/7 to monitor safety and ensure the shutdown did not lead to any issues with the machinery. We had a group of about 20 dedicated individuals at each facility working non-stop, despite the cold and darkness, with many of them staying overnight on cots we keep for just that purpose. These team members – the front line of ensuring the safety of our site – make me proud to be a part of NXP.

Careful planning to meet unprecedented challenges

In late February, utilities came back online at our facilities—the first step of getting the fabs back up and running. The second step was to stabilise both facilities; a multiday process of recovering both sites’ safety systems and ensuring that processes like heating and cooling are working properly. Throughout this process we conducted a variety of safety checks including evaluating our integrated air measurement system, which is a very thorough, multiday procedure to evaluate the gas elements in the air. Our facilities team tirelessly reviewed each alarm in our sophisticated control room each system was operating properly and enabling a safe workplace for our engineering and operations team members.

On February 27, we were able to bring more staff back onsite, most importantly into the cleanrooms, the heart of our facilities. Even as I write this in mid-March, validating the safety of the facilities is still an ongoing process. This is how thorough and careful we are when planning to restart production.

Only after we’ve verified safety can testing begin on every piece of equipment in the fabs – just under 1,000 of them per factory. We power up all tools in a safe way, inspect all wafers in our cleanroom, our WIP, and assess any physical damage. There is a major troubleshooting component to powering up the tools because they are so sophisticated and sensitive. Restart of the tools must meet our quality standards, meaning measuring particles on each piece of equipment before certifying the tools are ready to run our sensitive production.

To put this in context, we measure for particles 1/1000th the width of a strand of human hair. A single particle can cause us to undertake significant troubleshooting steps. It takes many days just to power them back up, and then several more days to make sure they are production-ready using our advanced process control systems.

Consider also these tools operate in cleanrooms at temperatures in a range of +/- one degree. A long period of exposure to uncontrolled environmental conditions, including a record low temperature well below the cleanroom target temperature, caused the kind of major damage you can see in the photograph below:

Beyond damage to manufacturing equipment, the storm damaged components of our facility infrastructure that were never meant to be replaced under any circumstances. These are components that are meant to last the life of the facility, and as such, there is no simple way of removing or repairing them. These are unprecedented challenges that most semiconductor manufacturers like NXP have never had to face.

Our teams developed solutions quickly to meet these challenges. A case in point is the replacement of damaged air handler units. As shown in the images below, we had to physically cut out the damaged units, remove a piece of the roof, and using a crane brought on site for this purpose, remove multiple 1,000 lb. units through the holes we just made. All of this with a tolerance of less than 12 inches, making this major mechanical replacement a high-risk endeavor for the team.

There are no shortcuts to functional safety and manufacturing quality

Only once we’ve determined that the tools and facilities meet production quality requirements are we ready to start running production for our customers. In this phase, the product is reviewed and confirmed, the tools are up and running, and we are finally able to ramp production towards normal pre-storm levels.

Restarting a fab is a time-intensive, full-team effort. Not only because of the massive scale of what we’re trying to do, but also because of how carefully we need to constantly inspect everything inside our facilities. All of us at NXP are eager to get the facilities running at full capacity and are making good progress towards that goal. Our processes are safe, tested and proven. We did not, nor will we ever, take shortcuts.

The professionalism, dedication and round-the-clock work over this period demonstrated by our teams on the ground in Austin is truly admirable. As we stabilise fab operations over the coming weeks, we will step back and assess what we’ve learned and identify opportunities to further harden and protect our facilities and equipment from similar conditions in the future.

I’ve worked in semiconductors for nearly 30 years and am constantly learning, but this past month has taught me and the entire NXP team more than we ever could have expected. We saw the importance of putting our community first, realised the value of constantly stress testing our backup plans and validated the importance of placing product quality ahead of speed of recovery.

The values exemplified by our wafer fab team are (in order):

• Safety
• Quality
• Delivery
• Cost

And all of this is empowered by our people.

Never have these values been more important than in response to this unprecedented weather event in Austin. Through it all, our team reaffirmed that they are the most dedicated and talented in the industry, and I am truly humbled by their commitment and resilience.