Quality at speed with Lights-Out Testing for DevSecOps

Why continuous testing? Complex technological landscapes and constantly evolving cybersecurity threats have significantly increased the need to perform more regular and iterative security scans on enterprise applications, especially for larger organizations. For example, cybersecurity testing should now be included as part of your company’s regular test automation coverage in a CI/CD (Continuous Integration and Continuous Delivery) model. And that is where the Lights-Out Testing (LOT) method comes in.

An automation feasibility assessment is the best way to agree on the level of automation coverage for functional and security verification. However, the automation scripts must be executed without manual intervention and this means using LOT. This virtually abolishes the need for human involvement by automating the execution of tests scripts.

How do you get started with LOT?

Testing is focused on processes that run and control critical operations. LOT is about mitigating the risk of maximum coverage in the shortest time. This ensures that business can continue running with a high level of quality and efficiency, even when the underlying technologies and processes change.

The LOT approach should be established from the early project lifecycle in a shift-left approach. This adoption begins from the unit test execution phase and continues in subsequent phases.

The use of LOT in one of our ERP implementations reduced execution effort by more than 75%.

How does LOT work?

The most important step is to strategize and identify the right toolset to orchestrate your LOT execution. For example, the toolset could comprise Requirement Management, Impact Analysis, Test Automation Scripting & Orchestration, Process Modeling, TDM, and Smart Reporting. These tools are either commercial or open source.

As soon as the toolset is ready, the automation feasibility check and automation script creation should be conducted. The scripts need to be developed using a modular-based approach, which eventually forms an end-to-end script and is easy to maintain. Once the environment is ready, the scripts should be set up in LOT execution with the aid of an orchestration tool.

Things to consider

It is important to consider the following key elements before adopting LOT:

• Choose the tools correctly based on the application landscape (we recommend performing a test yools assessment)
• Ensure tools are properly integrated to avoid multiple LOT issues
• Describe the defect logging process within your LOT approach
• Take a module-based approach to scripting to increase re-usability
• Tie up the test automation scripts in a logical way to perform end-to-end flows
• Make sure the scripts have some ‘wait time’ if multiple application/systems are involved in end-to-end flows
• Set up you test data using record set or import record sets
• Set up required credentials, based on the roles for each application.
• Most importantly, identify the off-peak time (mostly at night) to schedule the LOT

Identify causes of failed test cases

For the completion of the LOT, a causal analysis process must be executed for failed test cases. There are various reasons behind the failed scenarios, all of which need to be validated:

• Test data
• New enhancement
• Script problem
• Environment issue
• Functional defects

For functionality issues, a defect must be raised via the LOT setup with detailed evidence collected in a highly systematic approach. This evidence can be used as training and onboarding material in the future.

The final report should be generated using a Test Management tool.

The benefits of good Test Management

A Test Management solution integrated as part of Quality Engineering with ADMnext, will deliver significant benefits, including:

• Speed to market: Execution lead time is significantly reduced.
• Upsurge in quality: Off-peak testing capable of running multiple times in many environments improves test quality.
• Ease of test execution: Test execution can be performed on demand
• Productivity: With more time to focus on script development instead of script execution and failure analysis, the QA team’s productivity increases.

Certain Key Performance Indicators (KPIs) are used to measure the automation speed: Under Test Preparation; Automation Test % Coverage, Automation Progress, tests for Automation Execution include Test Execution Productivity and Defect Leakage.

So, In DevSecOps ERA, the continuous testing using Lights-Out testing approach can be a factor to succeed in big transformations using speed and quality.

As Marc Benioff says “Speed is the new currency of the business”.

About the author – Shashank Kallole

A Management Consultant at Sogeti in Finland, Shashank is a thought leader who shares his papers at international conferences. His specialization is test automation in the SAP landscape. Shashank has helped many leading organizations in the Nordic region to define the vision and strategy needed to deliver quality.