Introduction functional-testing-introduction

Learn about the quality gates available in the AEM as a Cloud Service deployment process, the different types of functional testing built in, how you can contribute and how you can make best use of them in the context of an overall testing strategy.

Overview

The following diagram provides a high-level overview of the available pipelines in the context of an overall testing strategy and the AEM as a Cloud Service deployment process.

AEM Cloud Service deployment quality gates

Purpose

The purpose of the AEM Cloud Service deployment pipelines is to facilitate robust and secure deployments at various stages of your development and AEM product release lifecycle. These pipelines incorporate multiple quality gates at different levels to ensure the integrity and safety of deployments for both your AEM application changes and AEM product updates.

Adobe provides several built-in quality gates, while others require your intervention for implementation and configuration. These quality gates are versatile, with some being applicable at various stages of the lifecycle and even integrable into your own development setup and CI/CD processes.

The built-in quality gates primarily validate the functionality of the AEM product within the context of your AEM application. In contrast, the custom quality gates you set up are designed to verify that your application’s critical features and user interactions perform as intended. Collectively, these two sets of quality gates work together to ensure robust and secure automated deployments for both your code modifications and AEM product updates.

It is important to note that these quality gates are not intended to be a comprehensive testing framework for your entire testing strategy. The AEM product is subjected to extensive testing before entering the AEM cloud service deployment process. Similarly, your application should already be of high quality before it reaches the deployment phase. This approach ensures that the quality gates focus on their primary objective of safeguarding the deployment process, rather than being a substitute for a full testing regimen.

Quality Gates

The following diagram provides a detailed view of available quality gates and their use in the overall testing strategy and the AEM as a Cloud Service deployment process.

AEM Cloud Service deployment quality gates

Summary Customer Provided Quality Gates

Unit Tests
Custom
Functional Tests
Custom
UI Tests
Customer
Validations
Manual
Testing
Production Pipeline
Yes
Blocking
Yes
Blocking
60m Timeout
Yes
Blocking
60m Timeout
No
No
Non-Production Pipeline
Yes
Blocking
Opt-In
Blocking
60m Timeout
Opt-In
Blocking
60m Timeout
No
No
Adobe Internal Validation
Yes
Blocking
Yes
Blocking
60m Timeout
Yes
Blocking
60m Timeout
No
No
Customer CI/CD
Yes
Yes
Yes
Yes
Yes
Customer Local Developer
Yes
Yes
Yes
Yes
Yes

Unit Test

You are encouraged to provide the unit tests for your AEM application, which are the foundation of every testing strategy. They are intended to run fast and often and give early and fast feedback. They are tightly integrated into the developer workflows, your own CI/CD and the AEM cloud service deployment pipelines.

They are implemented using JUnit and are executed with Maven. See core module of the AEM Project Archetype for an example unit test for AEM and getting started.

Code Quality

This quality gate is configured out-of-the-box and runs static code analysis on your AEM application code.

See Code Quality Testing and Custom code quality rules for more information.

Product Tests

Product functional tests are a set of stable HTTP integration tests (ITs) of core functionality in AEM such as authoring and replication tasks. Adobe provides and maintains them out-of-the-box. They are intended to prevent changes to custom application code from being deployed if it breaks core functionality in the AEM product.

They are implemented using Junit, are run using Maven and use the official AEM Testing Clients. The product test suite is maintained as
an open-source project, follows best-practices and can be considered a good starting point for the implementation of your tests.

Custom Functional Tests

Like the product tests, customer functional tests are HTTP integration tests (ITs) and are as well implemented using Junit, are executed using Maven and built on top of the official AEM Testing Clients.

NOTE
Custom functional tests are run in the production and non-production (opt-in) pipelines which are used by your AEM application changes deployments and AEM product push updates and are therefore a key contribution to help ensure proper functioning of your application and increase release safety. The customer functional tests are also executed in internal pre-release validation pipelines for each customer, which helps provide early feedback.

To keep pipeline runs efficient, we recommend focusing on key features and main user interaction flows. A run time of ~15 minutes or less for functional tests is recommended. Full functional test suites that do not fit in this quality gate are recommended to be ran as part of general customer validation pipelines during the customer’s development flow.

See open-sourced product tests or the it.tests module of the AEM Projects Archetype for examples.

See Java Functional Tests for more information.

Custom UI Tests

To maximize risk control for your customer-specific development, Adobe strongly encourages you to capture critical UI tests into AEMCS. They are intended to be kept rather limited in number, but with the highest impact on your customer experience.

The tests are packaged in a Docker image - designed to be as volatile as possible (with support for Cypress, Selenium, Java, and Javascript). They follow the same characteristics and purposes like the custom functional tests.

NOTE
Custom UI tests are executed in the production and non-production (opt-in) pipelines which are used by your AEM application changes deployments and AEM product push updates and are therefore a key contribution to help ensure proper functioning of your application and increase release safety. The customer UI tests are also executed in internal pre-release validation pipelines for each customer, which helps provide early feedback.

In order to keep pipeline executions efficient, we recommend focusing on key features and main user interaction flows. Full UI test suites that do not fit in this quality gate are recommended to be executed as part of general customer validation pipelines during customer’s development flow.

See open-sourced example tests or the ui.tests module of the AEM Projects Archetype for examples.

See Custom UI Testing for more information.

Experience Audit

The experience audit quality gate is performing Google Lighthouse audits against the customer’s webpage.

This quality gate is provided by AEM out-of-the-box, but is not blocking the deployment pipelines. By default, an audit against the root page (/) of the publish instance is performed. You can contribute by configuring up to 25 custom paths that are considered for audits.

See Experience Audit Testing for more information.

Customer Validations

The customer validations quality gate is a placeholder for the customer’s own testing strategy and effort, executed before the customer’s application changes reach the AEM cloud deployment pipelines.

Here you can choose the tools and frameworks you prefer. In contrast to customer function tests and custom UI tests, there are no AEM as a Cloud Service related limits, and we therefore recommend to perform long-running functional and UI testing here.

While you are free to choose any tool and framework, we recommend you align HTTP-based integration tests and UI tests with the tools and frameworks available in the custom functional tests and custom UI test quality gates. We recommend integrating Rapid Development Environments (RDE) in your local testing strategy to test as close as possible to AEM cloud environments.

Manual testing

The Manual testing quality gate is a placeholder for customers who do manual testing. AEM cloud pipelines do not support manual testing and therefore this needs to happen as part of your own local testing strategy.

For manual testing, it can be useful to integrate with an additional AEM Cloud Service development environment.

recommendation-more-help
fbcff2a9-b6fe-4574-b04a-21e75df764ab