The primary objective of this article is to provide an overview of the comprehensive topic of Factory Acceptance Tests (FAT) and Site Acceptance Tests (SAT). It begins by defining the core differences between the tests, moves on to the stages of test planning, defines the participants in FAT tests, proposes a team division and outlines their responsibilities, and concludes with a description of the most common problems encountered during tests, along with pointing out good practices to smoothly progress through all test stages.

At the outset, it is essential to clarify the basic differences between FAT and SAT tests: FAT tests are conducted at the supplier’s premises, while SAT tests are carried out at the client’s location after the prior approval of FAT tests and the delivery of the device to its final destination. According to Annex 15 to Eudralex Vol. 4, SAT tests may include tests conducted during the FAT stage, but they are not obligatory and serve as a complementary measure.

Why is it worth conducting FAT tests?

The FAT protocol is usually prepared by the supplier; however, the client’s company can also create the protocol. FAT is primarily developed for large and intricate devices, such as autoclaves, lyophilizers, or entire production lines encompassing machines for washing, sterilization tunnels, and aseptic filling stations. Nevertheless, smaller devices (e.g., laboratory glassware washers) may undergo preliminary testing by the supplier before being dispatched to the client. Before commencing any FAT tests, the client company should review the proposed FAT tests by the supplier, approve the protocol, and, if necessary, incorporate its own detailed requirements based on the user requirements specification. This is crucial because the vendor is likely to formulate the FAT protocol with specific tests and acceptance criteria (e.g., the device halts only after detecting 8 consecutive uncapped vials, as opposed to, for instance, 3 – such test stipulations impact, among other factors, the final machine performance directly tied to the quantity of capped vials per unit time, such as per hour, thus allowing compliance with acceptance criteria for overall performance, yet potentially resulting in losses during production due to excessive rejects of vials already with the final product, such as dispensed vaccine, to ensure the device passes acceptance tests at the supplier’s factory). Therefore, it is advisable for the recipient company to prearrange an agreement with the seller to have the flexibility to add/modify or expand tests, including refining the acceptance criteria proposed by the supplier, to thoroughly evaluate the device’s operation in accordance with the specific process requirements. It is also recommended for a representative of the contracting (company) to participate in the tests, serving as a witness to ensure that any modifications to FAT mandated by the company are considered, and that the tests meet production needs.

Furthermore, the execution of FAT tests brings various advantages, including:

• Enabling the manufacturer to identify and resolve issues before delivering the device to the client. This helps save time and costs associated with addressing problems at the client’s premises.
• Fulfilling FAT requirements may ensure that the client receives a product that meets their requirements and expectations.
• Enhancing customer satisfaction through the fulfillment of requirements.
• Reducing downtimes by identifying and addressing issues before delivery. Properly conducted FAT should contribute to reducing downtimes that could occur at the client’s site, thereby exposing them to potential future losses, such as failing to meet potential contracts or the need to dispose of expired products.

The approach to Factory Acceptance Tests should commence with the development of a testing plan. It’s worth noting that, regardless of how precise the planning by developers or software engineers may be, employees/operators often provide the most accurate assessment of new equipment. Their input is crucial in test design.

The involvement of all stakeholders in creating and reviewing the steps used for testing device functionality and failure conditions is also a crucial risk mitigation technique. Providing testing documentation to machine operators, quality personnel, and even production and sanitation staff, as well as process engineers, increases the chances of identifying all functions requiring testing. If the FAT test plan uncovers deficiencies in the design, it gives the team the opportunity to make changes at minimal costs. Implementing changes post-development can prove to be excessively costly. Placing an emphasis on early-stage testing ensures that both the software and the device are developed with a focus on both proper functionality and operator safety.

The Launch phase can be challenging and full of unforeseen obstacles. Without a well-executed FAT, the team may end up debugging the device system post-implementation, which can be time-consuming, frustrating, and costly. Adding new features after device startup often leads to poorly thought-out code (operating system) and a less flexible design. It is much easier to create and test potentially challenging scenarios in a simulated, rather than an already operational, environment. Eliminating potential issues through a properly executed FAT plan early on provides the greatest chance of success during implementation. It saves money on startup costs, eliminates the stress associated with deployment, and increases the likelihood of completing the project on time, thereby avoiding often significant penalties resulting from contracts or commitments.

FAT Planning

Planning is essential, and creating a comprehensive FAT plan is the first and potentially most crucial step. This plan serves as an action plan throughout the entire testing process. The FAT plan, also known as the ITP (Inspection and Test Plan), outlines all preparations to be made before the test and the procedures to be used during the test. A properly adhered-to test plan should help uncover any errors that were not detected during the device design stage. The test plan, along with specified acceptance criteria derived from the User Requirements Specification (URS), should be sent to the supplier as part of the Request for Quotation (RFQ). If the criteria are not defined, the machine may not operate according to the expectations of the future user.

When conducting FAT testing, it is crucial to remember that performing tests at this stage (at the supplier’s facility) is much easier and less costly than skipping tests that may initially appear too difficult, time-consuming, or expensive to execute, such as leak tests. It is important to keep in mind that the future user may not have the necessary tools to troubleshoot similar issues. The guiding principle should be that if testing in the FAT phase appears difficult and costly, what will it be like during the unit testing or regular usage phase.

The initial scope of FAT, to be supplied by the manufacturer, is defined during the customer’s order quotation phase. This plan includes all applicable specifications, standards, and drawings from the customer. The FAT scope assists in assessing the acceptability and reliability of the delivered equipment. The test scope is communicated to the customer in advance to fulfill both the requirements and often the specific expectations of the customer.

Before commencing FAT tests, it is essential to gather a comprehensive set of reference documents, including:

1. Customer scope and specifications
2. Drawings (General Arrangement-GA, Piping & Instrumentation P&ID, etc.)
3. Data sheets
4. ITP (Inspection and Test Plan)
5. Applicable codes/references
6. Checklists and procedures specific to FAT
7. Certificates specified in the agreement and calibration certificates

All devices used for data logging during the test should be verified beforehand for the calibration date required by the manufacturer or in accordance with customer specifications. The list of instruments and accessories used in the tests should include the device name, a description of its basic functions, model, manufacturer’s name, and serial number. Before initiating tests, it is advisable to check the fundamental functions of the tested device, such as power on and off; the behavior of the HMI interface during startup and shutdown; the appearance of relevant dialog windows; the accuracy and timeliness of date and time; the results of clicks on individual buttons or peripheral devices such as printers; and the signaling of external alarm indicators.

Responsibilities and Roles During Testing

FAT participants should act as representatives for both the vendor and the user. The collaborative team should include the following roles:

• User engineer in charge of system specification
• User and/or vendor personnel responsible for software
• User and/or vendor personnel responsible for hardware integration
• User support and/or supplier personnel

One of the team members, typically a user engineer, should be selected as the FAT team leader. The team leader should present a list of vendor requirements in written correspondence before the scheduled FAT. The number of participants depends on the technology as well as the size and complexity of the tested device. Factory Acceptance Tests are conducted by individuals representing the department or area for which the system or equipment is being purchased. It is recommended to involve personnel responsible for machine operation, such as operators and maintenance staff. Each FAT/SAT team should consist of individuals with direct operational experience, as they can provide valuable and practical feedback about the machine.

The testing process should encompass, at a minimum, the following aspects:

• Declaration of FAT location and dates
• General approach
• Description of the FAT checklist format
• Specification of hardware and software versions to be tested during FAT
• Verification of the tested equipment configuration
• Personnel safety issues

List of amendments

Documentation of testing procedures and results is key to achieving the intended test goal. One of the most efficient methods to ensure that all findings are documented is to maintain a list of corrections to record any errors detected during the test. Each item on the list should be numbered and marked with the date of discovery and a signature, described in a clear and consistent manner, and after corrective actions are taken, annotated with the date and initial.

Assignment of Roles in FAT Testing

Scenario 1: The supplier takes charge of all activities, including hardware integration, software configuration/programming, test execution, and corrective measures. Members of the user team serve as witnesses for all tests.

Scenario 2: Responsibilities are the same as in Scenario 1, except that members of the user team conduct the tests, and the supplier addresses any identified issues.

Scenario 3: Responsibilities are the same as in Scenario 2, except that members of the supplier team assist the customer in conducting the tests.

Scenario 4: The user is responsible for software programming/configuration. Tests are performed by both the user and the supplier. The user is accountable for resolving issues found in the software/configuration.

Scenario 5: The user is responsible for software programming/configuration and conducting the test. The supplier is responsible for addressing any hardware issues.

Scenario 6: This is an example of a BPCS system, whose primary function is to control process operations. The user purchased the equipment, integrated the system independently, and performed programming/configuration. The user is responsible for conducting the test and resolving any issues encountered during testing.

As evident from the aforementioned examples, there are several options for dividing responsibilities during acceptance testing. The scope of responsibilities should be clearly defined before the commencement of testing.

If possible, the FAT team should be divided into two groups:

1. One group conducting the tests.
2. Another group addressing issues identified by the first group

The advantage of this method is that FAT can continue almost seamlessly across one shift, with issues being resolved in the subsequent shift. This requires overlapping working hours of both teams to ensure timely and accurate information transfer. Each team member should be assigned specific tasks during the tests. The FAT preparatory document should define these tasks as detailed as possible.

FAT Schedule

The FAT schedule should include a daily list of activities, identifying each element in the system to be checked on a given day. The schedule should be structured to allow for the execution of all previously defined tasks within a predetermined time frame.

Additionally, the schedule should allocate sufficient time for one day per week, typically on the weekend, when testing will be suspended. This day should be utilized to reduce the list of corrections to a reasonable size. It is the responsibility of the team leader to utilize this time for that purpose.

Test Equipment Inventory

The inventory of test equipment should encompass all essential items for conducting 100% of the planned tests. Additionally, it should account for equipment necessary for troubleshooting, such as a directional microphone for detecting potential leaks. Instances may arise where, post-delivery and installation of a device, there is a leak, characterized by an inability to maintain the specified pressure. The user team leader must ensure the availability of required tools, including seemingly basic ones like screwdrivers (using torque wrenches is recommended for attaching covers to prevent cover cracking due to excessive tightening; it’s imperative to verify the correct tightening of screws connecting various parts of the housing, especially during SAT tests).

Test Procedures

A detailed test procedure should include, at a minimum, the following elements:

• Input simulations at 0%, 50%, and 100% input signal.
• Monitoring of the output signal at 0%, 50%, and 100% output level.
• Description of the typical testing method for each classification of programs in the system (e.g., interlocks, special computational blocks, etc.).
• Description of controls performed on graphic displays: these should cover graphical layout, color specifications, text, touch areas, paging functions, point addressing, and any system-specific elements.
• Description of the method for checking all other aspects of the system (e.g., visual checks, trends, logs, system failures, etc.).

The following types of tests are most commonly performed:

• Verification: documentation, visual, operation of individual components, operation of the machine as a whole.

Document Verification involves reviewing all documents required by the buyer. This fact should be specified early in the purchasing process. It is a necessary step as this documentation will be needed during operational and maintenance activities throughout the device’s lifecycle until its eventual retirement in the future.

Visual verification encompasses all physical checks of the machine in line with agreed-upon requirements:

• Data from the machine’s nameplate
• Specifications of individual components, both mechanical and electrical

Mechanical Inspection:

• Verification of P&ID
• Verification of electrical components, such as motors, sensors, PLC controllers, HMI interfaces, etc.

Verification of individual component operations involves checking the functionality of specific systems rather than the entire operation:

• Verification of safety and alarm systems
• Electrical system, such as motors, temperature systems, and the operation of Variable Frequency Drives (VFD)
• PLC controllers and their modules
• HMI along with its software
• Recording, storing, and displaying data

Machine operation verification

The final step involves verifying the operation of the machine as a whole, or potentially the entire production line. It is standard practice to replicate real plant working conditions. To achieve this, materials (packaging, samples of production material) used in the buyer’s facility are sent to the manufacturer in advance. . Sending materials in advance significantly aids in determining machine performance based on real parameters and checking if any detected deviations can be easily corrected. Additionally, changeovers are conducted for all formats to assess machine efficiency.

Finalization of FAT procedures

The outcomes of all tests are recorded, and a comprehensive final report is generated based on these results. Any deviations or discrepancies identified during the FAT are recorded and should be corrected before the machine is shipped to the end user. The acceptance document may specify whether open items on the list of issues need to be resolved/fixed before shipment or can be addressed after the device is delivered to the end user. The user should have the right to return to the factory to recheck any elements that are supposed to be rectified in the factory. For user convenience, the user may waive this right. 

If the test fails, corrective actions are defined, as well as a schedule for their implementation and the dates of the next FAT test. If the test yields positive results, the machine’s shipping process to the buyer is outlined, along with the schedule for the Site Acceptance Test (SAT). The FAT protocol is signed by authorized representatives of both the manufacturer and the buyer, with copies retained by both parties.

Most common issues during FAT and their remedies

Here are four commonly identified issues during Factory Acceptance Tests, along with the corresponding remedial actions.

1. Low manufacturing quality

Quality department representatives should regularly coordinate actions with project engineers and production representatives through, for example, periodic meetings. This ensures that technical drawings (electrical, mechanical, pneumatic, and process flow diagrams) are adhered to during device assembly. Before the Factory Acceptance Test, calibration certificates for instruments should be prepared, and equipment and material data sheets should be reviewed to deliver high-quality machines with the performance customers can trust and expect.

2. Non-compliance with regulatory or sanitary specifications

Customers often specify the standard level in the production of ordered equipment. Equipment for bioprocessing, boilers, and pressure vessels can obtain certificates from the American Society of Mechanical Engineers (ASME) through compliance assessment. In Poland, this process is overseen by the Office of Technical Inspection (UDT). For products, systems, and services in the electrical and electronic realm, manufacturers typically adhere to the standards of the International Electrotechnical Commission (IEC). To ensure compliance with design specifications, collaboration with industrial hygienists or health and safety specialists is recommended.

3. Lack of adequate machine safeguards and labels

Machine guards should be inspected before and after work, both before and during FAT with customers. It is the responsibility of the relevant personnel to ensure the availability and currency of installation, maintenance, and operating instructions, recommended spare parts lists, and user-friendly training materials for operators and maintenance staff. Labels and Lock Out Tag Out (LOTO) procedures should be clearly defined to comply with the standards of the Occupational Safety and Health Administration (OSHA) in the United States or the Polish equivalent (BHP).

4. Performance falling below expectations

The manufacturer’s quality representatives should present the step-by-step FAT procedure, including all records of the agreed FAT protocol, for the customer to review and approve. During the actual factory acceptance test, the desired equipment functionality should be confirmed through a series of validation tests based on agreed acceptance criteria and a list of issues (compiling discrepancies for modification).

Best practices during FAT testing

1. Communication with the customer

Upon the customer’s arrival at the factory, the manufacturer or supplier should discuss details that need to be defined before commencing acceptance tests. Explaining project corrections, such as change orders, demonstrates an understanding of the customer’s production process and the ability to meet their expectations.

2. Review of a detailed test plan

The test plan is a tool used to clearly illustrate the schedule and sequence of tests or inspections based on equipment order specifications and functional requirements. When reviewing the test plan with the customer, all necessary reference documents or acceptance criteria should be defined.

3. Conducting a safety-related risk assessment

Project managers should identify and eliminate threats in the testing area before commencing acceptance tests.

4. Equipment Operation Demonstration

After approving the FAT protocol, proceed to functional and regulatory tests—from the startup process to maintenance procedures. During the device testing stage, customers may ask questions to verify how the machine will operate post-installation in their facility. Be prepared to conduct ad-hoc tests if any discrepancies are revealed.

5. Implementing additional adjustments

Once the defects have been corrected and additional adjustments to the equipment have been applied to the user’s requirements, an official acceptance statement should be signed by the manufacturer’s or supplier’s quality manager, chief project engineer and production manager, as well as the customer’s representative, chief operator and maintenance coordinator (e.g., Maintenance Director). The client will conduct the on-site acceptance test (SAT) following proper installation in the permanent workplace. SAT additionally verifies the equipment’s performance relative to other systems on-site.

Object-oriented Acceptance Tests

After completing the Factory Acceptance Tests (FAT), receiving, and installing the equipment at the final destination, it’s essential to conduct Site Acceptance Tests (SAT). Successfully passing FAT should not hinder the execution of SAT, as these tests will reveal whether the transport and installation have affected the device’s functionality. Even if the seller is located nearby, damages may occur during transportation, particularly with equipment that includes high-efficiency air filters.

Filters can be mechanically damaged or improperly installed even at the customer’s site. As one can easily infer, not reverified during the SAT phase, an air filter may lead to non-compliance with cleanliness requirements according to Eudralex Volume 4 Annex 1: Manufacture of Sterile Medicinal Products. Additionally, instruments can collide, connectors may loosen, and parts may fall or get lost during separate transportation. Companies undergoing FAT should also conduct SAT tests to ensure that all elements are still in place and functioning as intended. These tests are also carried out based on a written protocol, which must be reviewed and approved by the customer before testing begins.

After receiving the device, the receiving company (manufacturer) performs a verification of the device in accordance with the order requirements. Subsequently, the manufacturing company prepares and approves the SAT protocol for both small and large units (often the same basic tests conducted in FAT). The SAT test, although similar to FAT, is usually a simpler testing protocol. Its purpose is to demonstrate that the received device was not damaged during transport and that all components are still intact. The next step is the commissioning.

Sample SAT scope

SAT may include the following tests:

1. General visual inspection
2. Visual inspection of major components
3. Verification of pressure and ventilation settings
4. Verification of functionality/locks (mechanical and programmatic)
5. Checking protective devices and locks
6. Operator training
7. Providing completed FAT protocol and documentation package including:

• Completed FAT protocol
• Maintenance and operating instructions
• List of recommended spare parts
• List of recommended cleaning agents
• Compliance certificates
• Technical drawings
• Material certificates/data sheets
• Instrument calibration certificates
• Welding process qualification

Eliminating manufacturing defects at the FAT stage, when the device is at the OEM, allows better control of the project schedule and budget.  

On-site SAT acceptance testing, which includes function testing, is virtually a necessity. This avoids delays in the commissioning and handover of the device . This is obvious due to the ever-increasing number of technology types, their implementations and integrations.

SAT is a cost-effective solution that is better than troubleshooting after the fact.

Employing a consultant is advantageous for both end-users and clients, as well as manufacturers. This is because consultants ensure that the new device meets all contractual requirements of both parties. It also helps in addressing any operational issues before the equipment reaches the client’s installation site.

Krzysztof Dziedzic
Professional Validation Specialist