What is Black Box Testing?
Black Box Testing, as defined by International Software Testing Qualifications Board (ISTQB), is either functional or non-functional (hyperlink) testing, without reference to the internal structure of the component or system.
Guru99 expands on this statement, and defines it as testing without looking at the internal code structure, implementation details and knowledge of internal parts of software.
This testing type is retrieved solely from technical requirements and product specifications. Its main purpose can be defined as checking whether the software is performing as expected by users and indicated in the specification documents.
What is White Box Testing?
White Box testing, also known as glass box, clear box testing, code-based testing or logic-driven testing, is defined by ISTQB as testing based on an analysis of the internal structure of the component or system. In general, White Box testing is considered to be more low-level testing, and is derived from expected internal functioning of the system.
We have also given an overview of the differences between White Box and Black Box testing types types below. However we recommend going through testing types within each category and their techniques see where these differences originate from.
Black Box Testing Types
Black box testing can be seen as testing from customer’s point of view, where the tester has an understanding of the expected output, without awareness of how that output is achieved by the system.
The general categorisation of testing into functional and non-functional also applies within Black Box testing. See our article on functional vs non-functional testing types (hyperlink) to read in more detail about them, their types and examples and differences between them.
Functional Black Box Testing
Functional testing in short tests whether each function of software meets the specified requirements.
Some of the main functional testing types include the following:
- Smoke Testing
- Regression Testing
- User Acceptance Testing
Example: Testing whether clicking on a button leads you to a page it is supposed to direct to.
Non-functional Black Box Testing
Non-functional testing tests all the non-functional requirements of a system, which comprise of all tests that do not fall into functional ones.
Functional testing types that could be tested via Black Box techniques are:
- Usability Testing
- Load Testing
- Stress Testing
Example: Page load and response times could be tested as part of non-functional testing methods.
Black Box Testing Techniques
Even when a developer has an idea of what needs to be tested, it can be quite tricky to come up with test cases in a systematic and structured way. Furthermore, the process has to ensure that the tests are going to cover the majority of the functionality and expose system’s potential weaknesses.
Now that we have briefly covered the types of testing that are considered Black Box testing, we are going the techniques used to develop the tests.
Equivalence partitioning, also known as Equivalence Class Partitioning, is a technique that eliminates the number of redundant test cases, where outputs are the same, and which would not reveal new or genuine defects of a system.
In other words, its goal is to reduce the test cases, that taking the same input set data make the system behave the same, i.e. lead to the same outcome when testing the system.
Therefore, when executing a program, inputs that give the same result are identified and classified into groups.
This technique is perhaps one of the most commonly used ones within Black Box Testing.
This article also lists several examples that might give you a better understanding of this technique.
Boundary Value Analysis (BVA)
Boundary Value Analysis is a testing process that checks correct execution of a program given valid and invalid input at extreme ends or boundaries of input domains. So, rather than focusing on the values in the ‘centre’ of input, it test the extreme valid inputs, as well as those falling outside those boundaries.
Equivalence Partitioning and Boundary Value Analysis are linked to each other, and are usually tested together. The boundaries are based on the classes defined in the equivalence partitioning, and are tested using BVA. Thus, Equivalence Partitioning is applied before Boundary Value Analysis
For example, extreme ends on input data, such as start and end, lower and upper, as well as so called ‘just inside’ and ‘just outside’ data sets are tested in boundary testing.
This technique is particularly useful when there’s a large set of possible inputs for the system, and thus creating individual test cases for all possible data is not feasible.
Decision Table Based testing
Similar to the techniques mentioned above, Decision Table Based testing is categorising input to the system, and expected outputs or system behaviour according to these inputs. However, in this technique, different input combinations and their corresponding output are mapped into a table. This table is sometimes called Cause-Effect table.
A good example would be of a table for an action of uploading an image on a website, where images are the inputs and system behaviour is an output. The image file has to be of .jpg format and of size less than 3 MB.
This table easily showcases the system behaviour when the input file does not match the specification.
|Condition||Case 1||Case 2||Case 3||Case 4|
|Size||Less than 3MB||Larger than 3MB||Less than 3MB||Larger than 3MB|
|Output||Image Uploaded||Input too large||Wrong format||Input too large|
Graph-Based Testing Methods
Graph Based Testing is similar to Decision Table Based Testing, as it maps input and output of a system. The graph drawing depicts links between the inputs (causes) and output (effects) triggered by the inputs.
Production of a Cause-Effect Graph can be helpful in generating test cases, when you think about the software test design and available input and output conditions.
This technique is also useful when trying to understand the performance of a system functionally, as it visualises actions performed and the flow from inputs into outputs.
Error guessing, as the name suggests, is quite a subjective technique. This definition puts it nicely, and mentions that it relies on the tester’s intuition and experience to identify defects that might be harder to catch using more formal testing methods.
This method can be more effective when several testers combine their knowledge and experiences of similar systems in coming up with potential issues of the application.
Its drawback is, of course, the subjectivity and total dependency on the knowledge of the tester.
White Box Testing Types
White Box Testing encompases testing types which are used to evaluate specific block of code, software package and/or to evaluate usability and functionality of an application.
However, in doing so it takes into consideration how the functionality is written, and often tests the particular parts of an implementation. In other words, the implementation of some functionality of a system is known and targeted in these types of testing techniques.
Some of the particularly widely used testing types that fall under the umbrella of White Box Testing include the following.
Unit testing is considered to be the lowest level testing method, where individual units or components are tested. Typically automated, unit tests are written and run to verify that small sections of an application meets its requirements, and its output matches the expectations.
This article gives a good detailed description of what unit testing is, as well as some examples.
Security Testing can sometimes be viewed as a separate branch of testing, as it can be done in many ways and by following a variety of structural approaches.
In general though, its aim is to find potential threats and check the validity of the system in regards to data stored and passed between components. For example, security testing can find how well a system is protected from unauthorised access and code damage.
If done improperly, security testing can undermine a company's reputation and safety of the stored data. Hence, methodologies employed within this type need to be quite sophisticated and thorough. While some black box methods can be used to test parts of a system, knowledge about the internal structure and code is required to perform the tests in-depth.
Testing for Memory Leaks
Another quite sophisticated testing that can be categorised as White Box testing is searching for and detecting memory leaks in an application. Memory leaks are one of the biggest contributors to low application performance.
More often than not, specialists in memory management are able to tackle these issues by testing at low to higher levels.
White Box Testing Techniques
One of the main techniques utilised within white box testing is called code coverage. Code coverage aims at finding the areas of a system that has been tested yet, and measures the degree to which the program has been tested already. The process also involves creating test cases that increase coverage, and continuously measuring the coverage throughout the process. The information gathered includes major characteristics of the running program and its code coverage, which is generated in a form of a report at the end of the process.
There are quite a few levels of code coverage available to developers. However, we are focusing on the three most prevalent ones, namely statement, branch and path coverage techniques
Other coverage methodologies not included in this article include Toggle Coverage and FSM Coverage.
Statement coverage is a process that involves execution of all statements of the source code at least once. You can imagine that the total number of total statements to execute can get high very quickly, so the process also tried to cover these statements with the minimal number of tests.
The process is used to calculate the total number of executed statements out of total present statements in the source code, and is based purely on the structure of the code.
Generally, statement coverage tries to uncover the following cases:
- Unused Statements
- Dead Code
- Unused Branches
- Missing Statements
Branch Coverage is used to make sure that every possible outcome is tested. The name is derived from the fact that the process aims at executing all “branches” from each decision at least one time.
For example, if the outcome of a process is a boolean variable, the functionality has to be tested for both True and False results.
Due to its nature this method also helps uncover sections of the code that don’t have any branches, thus reducing redundant test cases. Another positive consequence this method brings is that it can help test cases which would not have been tested using otherwise.
To design test cases, Path Coverage uses a program’s control flow graph, designed to identify execution paths within the program. A method called Cyclomatic Complexity is used to determine the number of linearly dependent paths.
Finally, a test case gets generated for each path resulted from the according control flow graph and calculated cyclomatic complexity.
Differences between White Box and Black Box Testing types
The main differences are easy to identify from the clear definition of these testing types. When testing using Black Box methods the implementation and internal structure of a system are left in a ‘box’, without looking much into it. Thus, the testers does not care much about the details, but rather on the functional outcome of the system. While in White Box testing methods, internal structure is exactly what is being tested.
Below are given the main differences, as well as what they imply in terms of effort and focused effort within developer teams.
In this article we have given definitions of two categories of testing - White and Black Box testing, as well as testing types that fall into these categories and techniques of developing test cases for each.
Although each category has its own benefits and drawbacks, it does not mean that one is preferred to another. Both of these types are essential to complete system testing, and are applied at different times in a variety of ways.