If you've ever been hobbled by a broken bone or endured an MRI, you’ve probably experienced the products made by Mallinckrodt Pharmaceuticals.
The US$5 billion global company produces specialty pharmaceutical products and diagnostic imaging agents. As the nation’s leading supplier of prescription pain medication, Mallinckrodt sells its products to major wholesalers and retail drugstore chains.
Its focus on pain control has helped to transition the company from solely being a drug provider to a manufacturer of equipment used to identify the source of the pain.
Located on the company’s 40-acre campus in St Louis is an active pharmaceutical ingredient (API) plant that makes ingredients used for narcotics and other final saleable products. The FDA-validated facility is more than 145 years old and employs approximately 450 people.
The plant’s control equipment was installed in 1985 and was costly to maintain. Mallinckrodt needed to upgrade the ageing technology and bring the plant to a common control-system platform. Cost and operational efficiencies were more important than ever; the company went public in 2013 and is now accountable to its shareholders. Ultimately, a control system upgrade meant that Mallinckrodt could evolve its business and realise improved uptime.
For system operators, one of the most challenging aspects of relying on an outdated system is the limited support. Of all engineering employees at the plant, only two understood the programmable logic controller (PLC) technology running the equipment. Service and vendor support were minimal, as was the availability and cost of spare parts. The result? Longer and more costly downtime.
“Cost avoidance was a huge motivator in updating our system,” said Don Geers, Senior Maintenance Engineer, Mallinckrodt Pharmaceuticals: “The downtime associated with hardware failure was a major concern. We wanted to avoid these issues by putting in a newer control system that’s easier to maintain with improved parts availability.”
Taking a virtual approach
The old technology did not allow engineers to perform predictive maintenance. The daily production reports would pinpoint problems, but only after they had occurred. With reactive maintenance as their only solution, unplanned and longer downtimes were inevitable.
Realising the cost of downtime is greater than the cost of change, Mallinckrodt decided to upgrade its control system. Replacing the brains of a manufacturing process in a 24/7/365 facility requires careful planning. The team wanted to limit transitional downtime and make sure the new system would run efficiently — and at full capacity — as soon as it was brought back online.
The team opted for a virtualised architecture. Mallinckrodt decided to complete the upgrade by replacing the legacy PLCs with a PlantPAx distributed control system from Rockwell Automation built on an existing VMware virtualisation infrastructure.
The virtualised PlantPAx platform enables plant-wide multidiscipline control and eliminated the number of physical servers needed to run the batch and HMI software. This greatly reduced the maintenance required for servers and workstations.
The system design was completed in two phases to minimise downtime. The first phase was handled by the Mallinckrodt maintenance team, with engineering from Rockwell Automation and local systems integrators — Stone Technologies and Syntronix — who were involved in developing the programming standards.
Once finished, Rockwell Automation completed the documentation and the functional and design specification. Then, the team built, tested and validated the programming code. This standard code was to be used on the next phase, as well as any new projects being implemented in the facility.
During the development phase, Rockwell Automation used a duplicate system in its Cleveland facility to help troubleshoot and finalise the programming before start-up. Mallinckrodt retained that service to call on technical support engineers for future assistance … should issues or system changes arise.
In phase two, updates were performed on the functional specifications utilising the new programming standards. Leveraging the existing production line documentation helped to reduce project risk. It also helped to ease the transition to the new system as the plant engineering, quality and validation team members were already familiar with the documentation. This reduced both time and cost during system testing and validation.
Factory acceptance testing (FAT) was completed at the Mallinckrodt facility with the help of local system integrators and Rockwell Automation. It took three months to complete the FAT.
Then the system was installed in the operating unit where multiple water batches were run to ensure that all system components operated properly before adding the expensive chemicals used to produce the pharmaceutical and imaging ingredients. Once testing proved successful, Rockwell Automation and Syntronix engineers completed the start-up on site.
As part of the PlantPAx system, reporting and historian software was implemented for improved production intelligence. The software collects, tracks and records key process data to pinpoint production trends, allowing for proactive maintenance activities.
Now equipped with validated, electronic reporting capabilities, Mallinckrodt is better equipped to meet regulatory reporting requirements and streamline overall operations through daily electronic production reports.
In addition, Mallinckrodt upgraded its HMI operator terminals. New HMI graphics were redrawn from the legacy HMI at the facility and consisted of 140 screens of process graphics for the new API line. The goal was to keep the operations interface similar to the existing equipment in the plant so that operators could easily transition to the new system when it came online.
In the end, commissioning only required five weeks, minimising system downtime for the install, thanks in part to smart engineering, migration tools and comprehensive functional testing.
Production benefits
“The new PlantPAx system has provided greater diagnostic/troubleshooting information into the control system hardware,” said Geers. “Having a common platform for control and information has also allowed our maintenance personnel to transition from expensive, reactive maintenance to proactive maintenance that was not possible with the old technology. This will reduce our downtime further as we move forward.”
The redundant virtual infrastructure, located in separate buildings, provides an added layer of protection, preventing a catastrophic failure in one location from shutting down the entire system. In addition, a virtualised platform consolidates islands of information.
In the event of a system failure on the data centre, the Mallinckrodt team will use the back-up. They can also run the back-up during scheduled maintenance outages, enabling the production line to keep running.
If a failure occurs, Mallinckrodt estimates that virtualisation will save at least eight hours — and as many as 24 hours — to restart the system and get the line back into service. With three separate facilities involved — one acting as the main power supplier for all other buildings — virtualisation provides a quick disaster-recovery option.
From a maintenance perspective, the control-system upgrade was deemed to be a success, but the benefits extend far beyond the maintenance aspect. Mallinckrodt's environmental health and safety (EH&S), plant floor, production and leadership teams now have access to continual process data via the PlantPAx process historian.
This allows them to more easily meet regulatory reporting requirements, access production information in real-time and initiate action if required.
In conclusion
In fact, operators and managers across the campus are benefiting from new levels of access to system data. Data from the historian software is accessible to anyone in the facility with Excel spreadsheet software and can be analysed from any building on campus.
EH&S professionals can now analyse real-time data to help ensure hazardous chemicals are not released into the environment. Operators entering production data on the HMI can now generate an electronic batch record, reducing the risk of error in compliance reports.
Daily production reports provide engineers and managers with correlated process data, collected from batches completed during several days, to quickly identify trends and institute changes to streamline production or to pinpoint potential bottlenecks. This enables Mallinckrodt to predict impending problems, respond faster and keep consistency and quality high.
“When Rockwell Automation originally approached us about the upgrade, their team consisted of engineers who work with life sciences customers,” said Geers. “Having a dedicated team that understood our pharmaceutical syntax and vocabulary reassured our decision-making team and allowed us to meet an aggressive timeline with minimal disruption and problems.”
Although it may have been a maintenance issue that prompted the control system and reporting upgrade, the entire operation has reaped the benefits associated with a better operating system and improved access to production data. Mallinckrodt now has the ability to build tools for all engineering departments and to benefit from a control system infrastructure that empowers the entire team.