Show Menu
TOPICS×

Understanding your Test Results

Cloud Manager for Cloud Services pipeline executions will support execution of tests that run against the stage environment. This is in contrast to tests run during the Build and Unit Testing step which are run offline, without access to any running AEM environment. There are two types of tests run in this context:
  • Customer-written tests
  • Adobe-written tests
Both types of tests are run in a containerized infrastructure designed for running these types of tests.

Code Quality Testing

As part of the pipeline the source code is scanned to ensure that deployments meet certain quality criteria. Currently, this is implemented by a combination of SonarQube and content package-level examination using OakPAL. There are over 100 rules combining generic Java rules and AEM-specific rules. The following table summarizes the rating for testing criteria:
Name
Definition
Category
Failure Threshold
Security Rating
A = 0 Vulnerability
B = at least 1 Minor Vulnerability
C = at least 1 Major Vulnerability
D = at least 1 Critical Vulnerability
E = at least 1 Blocker Vulnerability
Critical
< B
Reliability Rating
A = 0 Bug
B = at least 1 Minor Bug
C = at least 1 Major Bug
D = at least 1 Critical Bug E = at least 1 Blocker Bug
Important
< C
Maintainability Rating
Outstanding remediation cost for code smells is:
  • <=5% of the time that has already gone into the application, the rating is A
  • between 6 to 10% the rating is a B
  • between 11 to 20% the rating is a C
  • between 21 to 50% the rating is a D
  • anything over 50% is an E
Important
< A
Coverage
A mix of unit test line coverage and condition coverage using this formula:
Coverage = (CT + CF + LC)/(2*B + EL)
where: CT = conditions that have been evaluated to 'true' at least once while running unit tests
CF = conditions that have been evaluated to 'false' at least once while running unit tests
LC = covered lines = lines_to_cover - uncovered_lines
B = total number of conditions
EL = total number of executable lines (lines_to_cover)
Important
< 50%
Skipped Unit Tests
Number of skipped unit tests.
Info
> 1
Open Issues
Overall issue types - Vulnerabilities, Bugs, and Code Smells
Info
> 0
Duplicated Lines
Number of lines involved in duplicated blocks.
For a block of code to be considered as duplicated:
  • Non-Java projects:
  • There should be at least 100 successive and duplicated tokens.
  • Those tokens should be spread at least on:
  • 30 lines of code for COBOL
  • 20 lines of code for ABAP
  • 10 lines of code for other languages
  • Java projects:
  • There should be at least 10 successive and duplicated statements whatever the number of tokens and lines.
Differences in indentation as well as in string literals are ignored while detecting duplications.
Info
> 1%
Cloud Service Compatibility
Number of identified Cloud Service Compatibility issues.
Info
> 0
Refer to Metric Definitions for more detailed definitions.
You can download the list of rules here code-quality-rules.xlsx
To learn more about the custom code quality rules executed by Cloud Manager, please refer to Custom Code Quality Rules .

Dealing with False Positives

The quality scanning process is not perfect and will sometimes incorrectly identify issues which are not actually problematic. This is referred to as a "false positive".
In these cases, the source code can be annotated with the standard Java @SuppressWarnings annotation specifying the rule ID as the annotation attribute. For example, one common problem is that the SonarQube rule to detect hardcoded passwords can be aggressive about how a hardcoded password is identified.
To look at a specific example, this code would be fairly common in an AEM project which has code to connect to some external service:
@Property(label = "Service Password")
private static final String PROP_SERVICE_PASSWORD = "password";

SonarQube will then raise a Blocker Vulnerability. After reviewing the code, you identify that this is not a vulnerability and can annotate this with the appropriate rule id.
@SuppressWarnings("squid:S2068")
@Property(label = "Service Password")
private static final String PROP_SERVICE_PASSWORD = "password";

However, on the other hand, if the code was actually this:
@Property(label = "Service Password", value = "mysecretpassword")
private static final String PROP_SERVICE_PASSWORD = "password";

Then the correct solution is to remove the hardcoded password.
While it is a best practice to make the @SuppressWarnings annotation as specific as possible, i.e. annotate only the specific statement or block causing the issue, it is possible to annotate at a class level.

Writing Functional Tests

Customer-written functional tests must be packaged as a separate JAR file produced by the same Maven build as the artifacts to be deployed to AEM. Generally this would be a separate Maven module. The resulting JAR file must contain all required dependencies and would generally be created using the maven-assembly-plugin using the jar-with-dependencies descriptor.
In addition, the JAR must have the Cloud-Manager-TestType manifest header set to integration-test. In the future, it is expected that additional header values will be supported. An example configuration for the maven-assembly-plugin is:
<build>
    <plugins>
        <!-- Create self-contained jar with dependencies -->
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-assembly-plugin</artifactId>
            <version>3.1.0</version>
            <configuration>
                <descriptorRefs>
                    <descriptorRef>jar-with-dependencies</descriptorRef>
                </descriptorRefs>
                <archive>
                    <manifestEntries>
                        <Cloud-Manager-TestType>integration-test</Cloud-Manager-TestType>
                    </manifestEntries>
                </archive>
            </configuration>
            <executions>
                <execution>
                    <id>make-assembly</id>
                    <phase>package</phase>
                    <goals>
                        <goal>single</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>

Within this JAR file, the class names of the actual tests to be executed must end in IT.
For example, a class named com.myco.tests.aem.ExampleIT would be executed but a class named com.myco.tests.aem.ExampleTest would not.
The test classes need to be normal JUnit tests. The test infrastructure is designed and configured to be compatible with the conventions used by the aem-testing-clients test library. Developers are strongly encouraged to use this library and follow its best practices. Refer to Git Link for more details.

Custom Functional Testing

The Custom Functional testing step in the pipeline is always present and cannot be skipped.
However, if no test JAR is produced by the build, the test passes by default. This step is current done immediately after the stage deployment.
The Download Log button allows access to a ZIP file containing the logs for the test execution detailed form. These logs do not include the logs of the actual AEM runtime process – those can be accessed using the regular Download or Tail Logs functionality. Refer to Accesing and Managing Logs for more details.

Local Test Execution

As the test classes are JUnit tests, they can be run from mainstream Java IDEs like Eclipse, IntelliJ, NetBeans, and so on.
However, when running these tests necessarily, it will be necessary to set a variety of system properties expected by the aem-testing-clients (and the underlying Sling Testing Clients).
The system properties are as follows:
  • sling.it.instances - should be set to 2
  • sling.it.instance.url.1 - should be set to the author URL, for example, http://localhost:4502
  • sling.it.instance.runmode.1 - should be set to author
  • sling.it.instance.adminUser.1 - should be set to the author admin user, e.g. admin
  • sling.it.instance.adminPassword.1 - should be set to the author admin password
  • sling.it.instance.url.2 - should be set to the author URL, for example, http://localhost:4503
  • sling.it.instance.runmode.2 - should be set to publish
  • sling.it.instance.adminUser.2 - should be set to the publish admin user, for example, admin
  • sling.it.instance.adminPassword.2 - should be set to the publish admin password