Software Testing Overview for Drupal

Software testing challenges and why it is important

A faulty application might cost a lot of money to clients and users. As a result, you might have to compensate financially and even lose trust of your clients. With a thorough process of QA testing, you can foresee any potential bugs or issues that might occur in the future. In many cases, it saves a lot of money and time for all parties. 

Drupal-powered websites have specific Drupal-only contexts to keep in mind in the QA process. Whether you are a qualified developer, project manager, or quality assurance engineer, we will help you understand how to save your time without sacrificing quality.

Custom code can be a nightmare to keep stable

Finding a bug is not the trickiest part. Usually, you have a task to complete and then some sort of quality control over it to make sure everything is fine. It is where you are more likely to find some bugs.

Should you request any fixing? What happens if the testing fails and you need to reopen the ticket and work on it further? After reopening and fixing it once again, there might be a chance for additional bugs to appear, so this is quite a complex thing.

Drupal brings 2 of the most significant challenges - estimating and sanity testing. These are common challenges of giving the right estimate and choosing the part of the product to test.

Let’s look at each problem in more detail.

Requirement details and task estimation

We prefer to create stories for huge tasks with features and break them down into small pieces to describe every sub-task separately. Of course, we can give the estimate based on our experience but only with regular tasks.

For not-so-regular tasks, we give a more detailed estimate. We describe 3 scenarios: optimistic, realistic, and pessimistic.  Estimation is a very important part of the delivery process, because it allows planning releases and client’s time to verify the outcome of the release. Taking into account the Drupal context for accurate estimation is important to provide details on what should be done, so the developer knows what to do and QA knows what to test. 

That is why QA often begins work on the task before any development to make sure the task has all needed details and covers all of the possible cases. The acceptance criteria should always be provided, and we wait for our QA to ask clarifying questions and offer suggestions on the user experience.  

Let’s say we received a not very detailed request from the client for a new feature for ads on the website. Imagine the user story, including details: 

• ad placement options;
• allows admins to prioritize ads;
• visual effects like color, etc. 

With this information, converted into detailed task description, QAs can give a more accurate estimate rather than with a ticket saying “New feature”. Below is the example of the task that clearly defines what should be done and gives enough details about behaviors and how they should be tested.

Read the articles to learn more about different approaches to QA estimation: 

1. Estimation Techniques - Testing describes seven approaches to testing estimation:
   1. Experience-based testing estimation technique.
   2. PERT software testing estimation technique.
   3. Percentage distribution.
   4. WBS.
   5. UCP Method.
   6. Wideband Delphi technique.
   7. Function point/Testing point analysis.
2. Software Test Estimation Techniques: Step By Step Guide gives a good overview of the testing procedure from a business perspective with scenarios for applicable roles and best practices. 

Also, we suggest estimating the worst-case scenario when you don’t have enough time to dig into tasks or use any techniques.

Sanity testing

You can spend a lot of time if you decide to check all the functionality during a sanity test. The best start is to design a test plan. 

A test plan is a document outlining the scope, approach, and schedule of intended testing activities. It helps to choose resources for QAr to perform effectively. You will also understand how many times to test and what approach to use.

Basic steps of the test plan

1. Analyze the product. Who are the visitors of the website? What are use cases? How should it operate? What is the stack? 
2. Design a test strategy. It will help envision the volumes of work and plan risk mitigation strategies. 
   1. Pick the scope for testing.
   2. Choose the type of test (Unit test, API test, Integration test, etc.).
   3. Register risks and issues/bugs.
   4. Assign deadlines and a team behind the test. 
3. Define test objectives. What devices to test? What is the primary goal of the test? 
4. Define test criteria. Set up suspension and exit criteria (appropriate run and pass rates). 
5. Define Drupal core features. Define the list of Drupal core features that should be tested.
6. Plan resources. Both human and system resources. 
7. Plan a test environment. The testing team needs to consider the traffic volume, hardware/software behind the website, and any extra configurations for browsing the website. 
8. Schedule & estimate in man-hours. 
9. Determine test deliverables.
   1. Before testing deliverables (test plan, test cases, test design).
   2. During testing deliverables (test data, logs, simulators, test scripts, etc.).
   3. After testing deliverables (test reports, release notes, test procedure guidelines, defect report). 

The test plan guides our thinking. It is like a rule book to follow.

When we talk about CMS-based websites or Drupal, in particular, we should keep in mind some things. Drupal does not stand still and develops constantly. We need to stay tuned to its updates all the time. At Five Jars, we have test cases for QAs to follow with new Drupal updates. 

For example, with the release of a new version of Drupal, we make sure that the admin can create all common nodes, and users can see them without any problem. We check that basic Drupal functions work as before and there are no regressions in the admin part of the website along with the front-end.

Admin panel plays a vital role in Drupal. When you have a new version and check all functionalities in the developed for the customer module such as a “schedule module” for example , everything may seem good. However, it’s easy to fail to spot bugs instantly. The logs in the admin panel can help you detect abnormalities that you have overlooked.

Visual regression

Failure to visually validate can severely compromise the user experience and, in many cases, lead directly to lost sales. Without visual testing, the bugs are prone to slipping through even on a well-tested application.

Drupal is a stable platform. You can forget frequent crashes or blockers and prevent them with visual regression. We can provide this testing manually, but it will take a lot of time. For these scenarios, we use special tools to make changes smoothly, such as Diffy, Lambdatest, Kobiton, and AyeSpy.

These are amazing helpers with visual regression. We will describe some of them in another article.

As for devices and browsers with different versions, what should we choose? How do we understand if there are enough devices and browsers for testing or not?

It’s a good question. Drupal has requirements for browsers. It supports the latest release of each of the latest two supported major versions of:

Desktop browsers:

• Google Chrome
• Firefox
• Safari
• Microsoft Edge
• Opera

Mobile browsers:

• Safari for iOS

The latest supported release of the latest major version of:

Desktop browsers:

• Firefox ESR

Mobile browsers:

• Chrome for Android
• Chrome for iOS
• UC Browser (Drupal 9 and below only)
• Opera Mini (except for 'extreme data savings' mode)
• Samsung Internet

It is necessary to know because, by default, Drupal core does not accept bug fixes to unsupported browsers. Committers and the security team will assess the risk case-by-case and make exceptions to this rule when needed.

What about choosing specific browsers and devices? 

We suggest to:

• check which browsers and devices your visitors use (Google Analytics or similar tools can help you).
• check which versions are trendy nowadays.
• interview your customers about it. 

Automated testing

We recommend investing  in automation.

Automated testing is a software testing technique performed with special automated testing software tools to execute a test case suite. On the contrary, manual testing is performed by a human sitting in front of a computer, carefully executing the test steps. 

Automated testing allows us to test more, faster, and more effectively. You can forget routine manual checks and focus on things that are more important to test manually.  

But with great power comes great responsibility.

The average test plan for a commercial-grade application will have between 1,000 and 10,000 test cases. The biggest challenge is to determine which components of the test plan to test manually and which components to test automatically. Set realistic expectations. Do not automate everything. We are smarter than machines. Humans can see patterns and find failures intuitively.

First, decide on the type of testing:

• Automated unit testing;
• Automated UI testing;
• Automated API testing;
• Performance, load, and stress testing.

We perform testing in different layers to confirm the quality and reaction of code on edge cases:

• Unit tests;
• Kernel tests;
• Functional tests.

Also, we should not forget about auto tests that ship with Drupal. It comes with various automated testing options to choose from. Most of those are part of the core testing framework. It means that we can start simply testing already if our testing module is enabled. Below is just a simple example:

For example, using the following command we can run a core test:

$ ddev ssh
$ cd web
../vendor/bin/phpunit -c core core/modules/action

Successful execution with  looks like this:

PHPUnit 9.5.23 #StandWithUkraine

Warning:       Your XML configuration validates against a deprecated schema.
Suggestion:    Migrate your XML configuration using "--migrate-configuration"!

Testing /var/www/html/web/core/modules/action
....S..                                                             7 / 7 (100%)

Time: 03:25.204, Memory: 8.00 MB

OK, but incomplete, skipped, or risky tests!
Tests: 7, Assertions: 229, Skipped: 1.

HTML output was generated
http://localhost/sites/simpletest/browser_output/Drupal_Tests_action_Functional_ActionListTest-1-51362070.html
http://localhost/sites/simpletest/browser_output/Drupal_Tests_action_Functional_ActionListTest-2-51362070.html
http://localhost/sites/simpletest/browser_output/Drupal_Tests_action_Functional_ActionListTest-3-51362070.html
http://localhost/sites/simpletest/browser_output/Drupal_Tests_action_Functional_ActionListTest-4-51362070.html
http://localhost/sites/simpletest/browser_output/Drupal_Tests_action_Functional_ActionUninstallTest-1-80829434.html
...

This is a simple example of how drupal helps us with automated tests. Describing all benefits can be a part of another huge topic.

Functional testing

The purpose of functional tests is to test each function of the software application by providing appropriate input and verifying the output against the functional requirements. Here we want to talk more about unit testing and how it can be useful in Drupal. 

Most objects of a system depend on other objects (depended-on objects). Objects contain other objects and delegate work to them. The main objective of unit testing is to isolate written code to test and determine if it works as intended.

Drupal uses a testing framework called PHPUnit. PHPUnit delivers fine-grained testing. Its main purpose is unit testing. There are other Drupal frameworks built on top of it, such as KernelTestBase, BrowserTestBase, and WebDriverTestBase. These allow you to perform different types of tests using Drupal. For instance, BrowserTestBase focuses on performing HTTP requests against the site under test in a consistent way.

Drupal uses 2 different test runners: run-tests.sh and phpunit.

Drupal has standards about setting up tests, allowing test runners to find tests and specify different types of tests.

Here’s an example with the BrowserTestBase framework.

Why this framework? 

BrowserTestBase gives you a way to test web-based behaviors and interactions.

1. BrowserTestBase fills a need to perform top-level, request-based tests of integrations between various subsystems in Drupal. 
2. Generally, BrowserTestBase tests should not require site-specific themes or configuration, although it's possible to write tests with those requirements. Other tools, such as Behat, could be used for site-specific behavioral tests.
3. BrowserTestBase replaces the legacy Simpletest-based WebTestBase. If you are migrating away from Drupal's Simpletest, you should find it relatively easy to convert WebTestBase tests to BrowserTestBase. 

Performance testing

Some websites are more popular than others. It means they have more active users. Sounds good, huh? It does, but you need to have your application ready for the traffic volume growth.

Performance testing focuses on three metrics: scalability, stability, and speed. Unlike the usual quality assurance, we're looking for performance bottlenecks instead of bugs. Drupal has performance requirements for websites. Sticking to these requirements is a part of the QA process, too.

Besides, Drupal has requirements for a server for running the core software.

Disk space

The base files for the core software take up about 100 MB on the web server. You will need more space if you install additional modules or themes, host media, backups, and other files generated and uploaded to your website.

PHP

PHP 7.4 or higher PHP 7 version. PHP must be set up with a minimum memory size of 64MB. Running multiple modules on your website or using memory-intensive PHP-based command-line tools (such as Composer) requires more memory.

Web server

Apache is the most commonly used web server. Alternatives include PHP Local Server, Nginx, and Microsoft IIS. Pick the one that compliments your goals.

Database

MySQL - 5.7.8 (MariaDB 10.3.7, Percona 5.7.8) or higher with an InnoDB-compatible primary storage engine.

PostgreSQL - 10.0 or higher with the pg_trgm extension.

SQLite - 3.26 or higher. Temporary local demo websites use SQLite, distributed as part of PHP. It doesn’t require installing separate database software, but make sure your version of PHP has the minimum SQLite included.

We won't talk about performance testing deeply, but you can check how it works in our company in the article Website Performance and User Acceptance Tools.

Found a bug in Drupal Core? Learn how to report and get it fixed 

As we know, every system, app, and website has bugs. Sometimes we can even find them in Drupal Core. It is the best chance to help the community to refine the product. Drupal wrote some tips to make a good bug report and here nothing add. I just share these links where you can find usefull information about this.

Conclusion 

The QA process is a really important and difficult part of the development and you should not undervalue any of the testing activities. They are part of a successful project. I hope reading this article would make your project better and help with understanding the important aspects of working with Drupal.