As electric utilities strive to modernize and handle increasing complexities within their operational frameworks, the role of Advanced Distribution Management Systems (ADMS) becomes critical. These three stages delve into the multifaceted process of ADMS testing, beginning with initial considerations, progressing through the meticulous execution of test cases, and culminating in the final validation before deployment.
Stage 1: Understanding the Need for Rigorous ADMS Testing
At this initial stage, we spotlight the crucial testing phase of ADMS. Ensuring that ADMS meets its business, technical, and functional requirements is a multifaceted process, encompassing various stages that stretch from its inception to the final deployment and commissioning at the customer’s site. This phase is essential for laying the groundwork for a successful ADMS implementation, tailored to navigate the unique challenges electric utilities face today.
EnerNex specializes in guiding utilities through this critical testing phase, equipped with the expertise and insight to ensure that every requirement is met, setting the stage for seamless deployment and integration.
Stage 2: Delving into the Testing Phases
As mentioned in stage 1, rigorous ADMS testing is a comprehensive and ongoing process that extends from initial development through to final deployment and commissioning at the utility’s site. This involves several testing phases tailored to the ADMS’s complexity and its integration challenges within the existing utility systems:
- Functional / Unit Testing
- Specific to ADMS product build.
- Internal to ADMS provider, rarely shared with customer.
- Performed during DMS development on Product Development System (PDS).
- Critical if ADMS required extensive customization and/or enhancements not available in base product.
- Pre-Factory Acceptance Testing (Pre-FAT)
- Performed by ADMS provider on PDS using internal templates/data (Fantasy Island). Alternatively, testing is conducted using customer’s converted DB and displays, if available.
- Validates functional requirements and test cases for upcoming FAT.
- Often covers comprehensive test cases limited to scope of DMAS product delivery.
- Encountered defects as often logged and prioritized for fixes by ADMS provider, not shared with customer.
- Factory Acceptance Testing
- Performed by customer with assistance from ADMS provider on PDS, often on ADMS provider’s site and with customer’s data/network model.
- Validates that contractual requirements are met by the delivered ADMS product.
- May include various FAT cycles if ADMS product delivery happens in multiple official releases.
- Resultant higher severity/criticality defects 9Level 1&2) must be resolved prior to upcoming SAT.
- Pre-Site Acceptance Testing (Pre-SAT)
- Performed by customer with supervision by ADMS provider, often on the Quality Assurance System (QAS) with customer’s data / network model and converted DB and displays, once shipped to customer’s site.
- May include: Smoke testing (checks/confirms eh stability of ADMS product build and connectivity of the critical functionalities; Regression Testing (retests defects fixes from FAT)
- Site Acceptance Testing (SAT)
- Performed by customer with ADMS support from ADMS provider, often on the Production System (PRD_ with customer’s live data and operational network model, on customer’s site.
- Often validates end-to-end To-Be business process workflows.
- May include field verification, with point to point testing of RTUs and supervisory controls.
- Make sure that all ADMS components, data flows, interfaces and communication paths function as designed/required.
- Resultant higher severity/criticality defects (level 1&2) must be resolved prior to official end and sign-off on SAT.
- Customer & ADMS provider to reach a consensus on prevailing lower severity/criticality defects (level 3&4).
- Site Integration Testing (SIT)
- Performed by customer with support by ADMS provider as needed, on PRD with customer’s live data and operational network model, on customer’s site.
- Tests ADMS’ interfaces to other operational and enterprise systems based on related requirements: 1) Data quality, Transformation, Flow, Messaging, Scheduling, and Timing. 2) Event triggers.
- Often uses other manually initiated or automatic script-based tools that validate the interfaces
- User Acceptance Testing (UAT)
- Performed by customer with support by ADMS provided as needed, on PRD with customer’s live data and operational network model, on customer’s site.
- Performed by end/business users to validate the final ADMS product meets the business requirements.
- Validates end-to-end data quality for end-to-end business processes.
- Occurs after all major integration tests are complete and all related business functions and business events have been integrated and tested.
- Performance Testing
- Performed by customer with support by ADMS provider as needed, on PRD with customer’s live data and operational network model, on customer’s site.
- Most often, the ADMS provider drafts and shared the performance test strategy and procedures with customer.
- Confirms that ADMS performance meets or exceeds pre-determined/required acceptable levels: ADMS application outputs, processing speed, data transfer velocity, network bandwidth usage, maximum concurrent users, memory utilization, workload efficiency, command response times, etc.
- Availability Testing:
- Performed by customer with support by ADMS provider as needed, on PRD with customer’s live data and operational network model, on customer’s site.
- Ensures availability of ADMS critical functions for specified times, such as: 1) System uptime > 98% 2) Each ADMS server/device uptime > 98%
- Duration of Availability Testing – often at customer’s discretion, for instance test could be conducted for the period of 1000 hours.
- Other Testing:
- Regression Testing: validates correct operation after software is modified.
- Loading Testing: could be part of Performance Test, to validate ADMS’ adequate performance under prescribed loading requirements specific to customer’s operating conditions, such as Storm vs. Sunny-Day modes.
- Security Testing: identify security vulnerabilities and validate CIP compliance to NERC Information Security Policies.
- Operation Readiness Testing: validates the readiness of ADMS to move into production environment, e.g., testing of local and geographic fail-over, system, OS and application monitoring, and operational support processes.
Stage 3: Test Case Setup and Execution Lifecycle
In any of the ADMS testing phases covered so far, designing, and setting up meticulous test cases is crucial and often complex. Each test case is crafted to validate a specific ADMS functionality or feature, aligning with either a business or a technical requirement. From simple functionality checks to comprehensive end-to-end process evaluations, test cases are the backbone of ensuring ADMS systems meet operational demands. Conceptually, an ADMS test case would have the following components:
- Title and General Information: Includes IDs, timing, priority, software version, and other relevant details.
- Description and Setup: Outlines the purpose, expected outcomes, and setup requirements for the test.
- Entry and Exit Criteria: Defines what triggers the test and the conditions for its completion.
- Dataset and Steps: Details the data involved and the step-by-step procedure for execution and verification.
- Result Status (as seen below): Outcomes are categorized as Passes, Passed with Minor Errors, Failed, In Progress, Did Not Run, or Blocked, each requiring specific follow-up actions.
- Clean Up: Describes actions necessary to return the system to its normal state post-testing.
A typical execution cycle for an ADMS test case is depicted in the following illustration, assuming that the ADMS has a Program Development (PDS) and a Quality Assurance (QAS) components of systems:
CONCLUSION
As we wrap up our exploration of ADMS testing, it’s clear that each phase of the process is not just a step toward implementation, but a stride toward transformative change in utility management.
We have demonstrated the depth of expertise and attention to detail involved in ADMS testing, from initial setup to final validation. As your partner in energy solutions, EnerNex is dedicated to advancing the filed of innovations that not only meet today’s challenges but also anticipate tomorrow’ opportunities.
For expert insights and guidance on ADMS implementation, connect with Aniss Fradi, Principal Consultant.
Follow up on LinkedIn and Twitter to stay updated with our latest news.