A leading life sciences company with a portfolio spanning cell biology, DNA, RNA and protein analysis, as well as drug development, human identification and molecular diagnostics, needed to eliminate downtime and data loss in its manufacturing operations by selecting a more reliable and easy-to-maintain compute infrastructure for its batch manufacturing operations.
The company had been relying on hyperconverged server clusters and software-based fault tolerance to manage process automation and the data required for US FDA regulations.
Consulting with RoviSys, a global systems integrator specialising in automation, the company standardised on Rockwell Automation PlantPAx software running on Stratus edge computing for three manufacturing sites on the company’s campus.
RoviSys deployed fault-tolerant edge computing platforms for process automation that dramatically simplified the life science company’s IT infrastructure. Using Stratus platforms, RoviSys deployed the control architecture in a week — from factory acceptance testing (FAT) to site acceptance testing (SAT) and validation while reducing TCO by 15–20% (compared with the prior cluster solution).
Since the Stratus deployment, the life sciences company has experienced no downtime and is able to service the compute platforms without dedicated IT support.
Server clusters insufficient for life sciences manufacturing
The company had been relying on server clusters to provide computing and data collection for its critical manufacturing processes. The team faced a number of costly challenges, from component failure to unplanned system outages, that resulted in data loss or downtime.
This impacted their ability to pass FDA audits and GMP reporting and led to crippling manufacturing disruptions and product waste. If a cluster server failed, the team often did not have immediate support and knowledge to rebuild the node requiring additional recovery time.
Additionally, the team needed to be able to track their VCAP (version change, audit control) process to map system changes to specific batches and vice versa.
Initially, RoviSys considered a hyperconverged cluster based on Cisco HyperFlex that would reside in a corporate data centre and would be managed by the end user’s team. However, that choice entailed several hurdles. The solution required a virtual SAN with 10 GB networking across the customer’s campus, which needed a lot of expertise and bandwidth.
In addition, the highly customised and complex nature of the solution not only needs extensive licensing costs, but would also require a team of highly trained people.
From a practical standpoint, this allowed too much margin for error in the setup and management of the cluster system.
Simple, protected, autonomous edge computing built for OT
To provide a simpler and easily serviceable availability solution, RoviSys recommended the Stratus ftServer and Stratus ztC Edge compute platforms. In contrast to the more complex hyperconverged cluster that required significant integration work, Stratus platforms are ready off-the-shelf to be installed quickly.
The operational simplicity was a key factor making the Stratus platforms a solution that the OT team could use without needing specialised IT skills. Installation and end user training took about 2 hours — instead of at least 2 days for the HyperFlex solution. The team also calculated the end-user’s total cost of ownership (TCO) to be 15–20% lower with Stratus.
The Stratus platforms provided an exceptional level of data protection with 99.999% availability and no risk of costly downtime while enabling stronger cybersecurity. Owing to the hardware-based fault tolerance, there was no failover delay or data loss.
In the event of component failure, the customer’s operational team could easily swap out plug-and-play components with no impact to the batch application, process disruption or data loss. With the uninterrupted operation of the applications and VMs running on the system, validation testing can be completed quickly, saving significant time and resources.
In comparison, with a cluster node failure, the team would have faced multiple steps to replace and validate a new node — from physical replacement and physical/logical validation to confirmation that all VMs across all nodes were functioning.
The entire process for a manufacturing site might require as many as ten people on a customer’s validation team plus two or three consulting resources to run the testing and take days or weeks.
Production control architecture with Rockwell Automation and Stratus Technologies
For production control, RoviSys architected Rockwell Automation and Stratus solutions for three facilities — a large-scale protein manufacturing site, a site producing DNA and RNA reagents and a small molecule site feeding other manufacturing processes.
For the two larger sites, the team matched Rockwell Automation PlantPAx, including HMI, historian, batch, Asset Centre, Vantage Point, Thin Manager, Engineering Workstation and SQL, running on a Stratus ftServer 6910 and 2900, respectively.
The small molecule site did not require 5 nine plus reliability and ran the PlantPAx suite on a Stratus ztC Edge 110i platform.
For network security, the platforms were segmented based on Rockwell Automation’s recommended cybersecurity practices.
The team set up seven VMs per production line to consolidate software workloads. The efficiency also translated into significant savings on software licenses, for example, costing only $600 for VMware for ftServer as compared with thousands of dollars per node in a cluster solution. Similarly, the team was able to realise cost savings with Microsoft data centre licenses.
Shorted FAT/SAT cycles and quick validation
For deployment with the Stratus platforms, RoviSys accomplished the crucial FAT/SAT testing and validation processes in hours versus weeks with a cluster solution. The team was able to plug and unplug nodes and verify immediately failover without disruption, including running a library of Linux routines to test that various components and low-level software were up-to-date as part of the FAT/SAT process.
The rack-mountable, fault tolerant and self-healing design of the Stratus platforms enable them to be deployed in production for a lengthy time. The team forecast a 15-year duty cycle for the platforms and concluded that with Stratus’ longevity, durability and streamlined FAT/SAT verification, the life sciences company would have a top-performing, high-availability solution for the long haul.
OT/IT convergence, shorterned FAT/SAT cycles and fault tolerance a win in life sciences
Working with their life sciences end user, RoviSys quickly deployed Stratus edge computing and transformed the reliability of their customer’s manufacturing operations, VCAP process and GMP reporting. The solution dramatically simplified IT infrastructure and enabled the OT team to deploy and maintain new infrastructure without on-site IT support.
Importantly, RoviSys’ key to success was running a successful proof of concept (PoC) and using a solution that reduced FAT/SAT and validation testing from weeks to days. With the trust built from this success, RoviSys has expanded its end user engagement for additional projects.