Crowdsourced Software Testing: What It Really Is?

Dunya Habibi on 06 April, 2020

In the past decade crowdsourcing as a term has been making its way to headlines of news, articles and blog posts. Companies and start-ups claim this approach to be the ultimate driver for effective and scalable innovation.

Even though crowdsourcing has established itself quite clearly, it is usually the case that the term is misused. So what is crowdsourcing exactly?

Although the main topic of this article is crowdsourced testing, it is important to clear the air about crowdsourcing in general, as it forms the bases of crowdtesting as a concept and its characteristics. Hence, here we first discuss what crowdsourcing is and its brief history. Then, we connect the concepts of crowdsourcing and crowdtesting, and how the benefits usually obtained by the power of the crowds apply within testing.

What is Crowdsourcing?

The term crowdsourcing was coined in 2006 by Jeff Howe in his article for Wired magazine called “Rise of Crowdsourcing”. A few years later, Howe also wrote a book called “Why the power of the crowd is driving the future of business“ that elaborated on his points made earlier in the article, and expanded the topic further in detail. This book to this day remains one of the most recommended books when it comes to the topic of crowdsourcing.

However, since 2006 the Internet has not stopped growing in the slightest, further flourishing every year, and yielding innovation within technology and the ways people use it. The emergence of crowdsourcing and its success is largely attributed to the so called “Web 2.0” era, which also allowed it to further develop in the following decade.

In general, a divide in crowdsourcing definitions can be clearly seen.

Jeff Howe, considered by most as the ‘founding father’ of the field, defined the term as ‘ways in which the power of the many can be leveraged to accomplish feats that were once the responsibility of a specialized few’. He highlighted how the crowd is not only wise, but also creative, talented and stunningly productive.

Several years later, the industry very much needed a less ambiguous definition. Daren Brabham gave such a definition in 2012 in his book, appropriately named “Crowdsourcing”.

The shorter version we are listing here goes like this:

Crowdsourcing is a type of participative online activity in which an individual, an institution, a nonprofit organization, or company proposes to a group of individuals of varying knowledge and number, the voluntary undertaking of a task. The undertaking of these tasks of variable complexity and modularity, in which the crowd should participate bringing their work, money, knowledge, and/or experience, always entails mutual benefit.

The important distinction here is that there is an organisation that proposes the task, and the interplay between this organisation and the crowd yields mutual benefit.  


What is not crowdsourcing?

To showcase what this distinction means in practice, it is important to distinguish pursuits that do not count as crowdsourcing per se. Contrary to common belief and media articles, the following examples are not considered as crowdsourcing.

Although throughout history there have been many instances of problem-solving in a collaborative manner, crowdsourcing is a new phenomenon that relies heavily on technology.

Crowdsourcing is not merely any activity that engages an online community. It involves a task posed by an organisation and an online community, and mutual benefit for both.

Type Example
Open Source Projects Mozilla Firefox
Commons-based peer production Wikipedia
Market Research CocaCola flavour selection campaign

Not every online community and their efforts are considered crowdsourcing.

What is crowdsourced testing?

It is finally time we talk about what crowdtesting is how it fits within crowdsourcing. Although lengthy, the paragraph above gives an important introduction to what crowdsourcing is, and what its characteristics are. These aspects of crowdsourcing apply to crowdtesting, and lay the ground for the tendencies and benefits of the latter.

Given the definition above, it is clear how testing, organised in a certain way, can be considered as a crowdsourced activity.

This certain way of testing, organised to utilise the power of crowds, is what Buglance provides as a platform - a platform, where companies and teams submit their software applications, and the community of Buglance tests and reports bugs.

Firstly, this is considered as a crowdsourced activity as there is an organisation, in this case a company or a tech team, that submits a task for the crowds. The tasks in general terms entail making sure that the application at hand does not have defects, or more realistically, has as few defects as technically possible.

Secondly, a bug-free software and the uncovering of defects of an application is the common goal for both parties: the company and the crowd.

This activity also leads to mutual benefit. Obviously, the company benefits from finding bugs in the submitted software, and tech teams within the company can get to fixing bugs as soon as they are submitted.

The individual testers, however, get financial rewards based on the complexity of bugs they have reported. Buglance has set up the system and its pricing in a way that makes it financially appealing to both the community and the tester crowds. The company pays ‘in bulk’ a certain amount, according to the pricing plan it has chosen, and gets thousands of testers essentially for the price of one. We will expand on this point later in the article.

And testers, on the other hand, spend as much time as they would like on a problem, and get financial compensations based on the difficulty of the problem and the amount of effort required.

The platform has shown that a win-win scenario is more than possible, and its model has proven to work extremely well. By now, the tester community of Buglance has grown in numbers to more than 40,000 testers from more than 150 countries.

More importantly, in just a year the community has reported more than 250,000 bugs in software applications submitted from all over the world.

To showcase how crowdtesting enables efficiency and how the numbers mentioned are even possible, we will outline some characteristics that crowdsourcing in general, and crowdtesting in particular, have.

Note: it is also important to distinguish between crowdsourced and outsourced testing. softwaretestinghelp.com has a great article about crowdtesting, which showcases differences in these approaches.

Crowdsourced testing characteristics

We are going to list some characteristics crowdsourcing brings to testing, as well as some additional ones, which are byproducts of such approaches.

Scale

It makes sense that in organising the crowds to perform a task, more often than not, the number of people involved has to become relatively large. The scale with which the problems are solved enables organisations to effectively tackle issues, without the need to hire thousands of people. Individuals from all over the world can start engaging with an issue at hand, and offer their solutions and opinions on it.

When it comes to testing, scalability means a lot more than just the number of people involved.

Usually, a software application gets tested on a variety of areas. Those might include graphical user interface (GUI) testing, security testing, functional testing, etc. With crowdsourced testing, there is no need to do the testing sequentially and/or partially, due to time and resource limitations.

When an application is submitted to Buglance, preferred testing types can be selected during the setup. This way, the application can be tested within those types in parallel by hundreds of real users. For example, in a car hire mobile application a group of testers could be reporting bugs on the GUI, while another group is testing the payment process in the app.

Speed

High speed is another attribute of crowdtesting. Due to the number of people involved and the fact that all activities are performed independently and online, tasks get completed at an extremely high pace.

In regards to software application testing at such speeds can lead to many benefits. Especially, when small and medium sized companies are trying their best to deliver their product on time, and quality is of utmost importance.

Another positive consequence of the speed and crowd’s involvement is that by outsourcing their testing, teams can focus on the development of the product and its new features.

Instead of coming up with test cases, setting up the tests and running them, software developers can effectively work and fix reported bugs.

Here in Buglance, we often find tech teams very surprised, when they start seeing bug reports within the first few hours of submission. As a result, the whole process of setting up and testing itself is usually cut down by ten times.

Diversity


Diversity might be one of the most defining characteristics of crowdsourcing. It is usually said that the Internet and crowdsourcing has broken down stereotypes of expertise, proving that innovation does not necessarily come from officially qualified experts.

Since the qualification is not any longer a necessity for anyone to take part in an activity online, the crowds usually consist of highly diverse individuals. People with different backgrounds, cultural identities and technical skill sets take part in these activities, usually in their free time to get financial benefits and/or emotional satisfaction.

In crowdtesting diversity is reflected in several ways. Those also can be categorised into human and environment diversity.

Human diversity simply means what was mentioned before - people with diverse set of skills and backgrounds come together to test software. This is particularly important as developers usually get stagnant in their approaches to testing and have what is called ‘developer bias’. Meaning, they assume certain user flows, which prove to be overly simplistic and perfect.

Crowdtesting allows more explorative testing, whereby real users can find cases of which engineers would have never thought of themselves. In fact, this has been the case for most start-ups Buglance has facilitated testing for, and engineers are left surprised by the reported bugs.

Human diversity, also characterised by geographical variety lets software to be tested within many countries. Testers can report mistakes in the content of an application, and issues related to compatibility. For example, if an application is a web based one, certain countries might have restrictions on certain library or browser usages.

Environmental diversity is a crucial feature of crowdtesting. If you read through user feedback on mobile application stores (e.g. Google Play, App Store) you will see that the majority of negative reviews are related to certain device ranges. It is hard to make sure that when an application is released, it is compatible with outdated or new devices, and their libraries.

Therefore, by testing on thousands of devices, tech teams can ensure that the number of devices on which an application causes an issue is minimised.

Note, that not all crowdtesting platforms allow for this feature.

In Buglance, the environmental diversity is easily showcased by the number of devices accessible through the platform. Currently the platform holds more than 10,000 unique devices. In fact, when setting up an application for testing, you can also choose a range of devices to test on, their models and brands.

Cost

Crowdsourcing is well-known for its ability to reduce costs. It does so by splitting the task at hand into smaller chunks, and distributing it throughout the community.

Testing might not be on the list of things that a start-up would like to decrease its expenses on. However, product development and staff salaries certainly are on this list.

It is estimated that testing can take up anywhere from 20 to 50 percent of development time, depending on the type and complexity of software being developed. And an average salary of a single software developer in the United States is around 87,000.

A team of developers usually consists of 4 to 9 developers, bringing the annual expenditure on software engineers to anywhere between 350,000 and 800,000.

And this brings us to the total annual amount spent on testing:

  • Minimum of 70,000
  • If you have only 4 developers and a very simple application
  • A whopping maximum of 400,000
  • If you have around 9 developers in your team, working on a very complex user facing application

It becomes clear then, that decreasing the amount of time a team of engineers spends on testing can have a significant impact on a company’s bank statement.

Furthermore, because crowdtesting lets engineers focus on the development of a product, and reveals more defects that a normal testing procedure would, it also increases user trust. In the long run, this would also have its effect on the company's reputation, and ultimately the revenue.


Real Users

The crowd being a community of real users does not necessarily translate to immense benefits within all industries, however it does so for testing activities.

There is no doubt, that traditional testing methods and outsourced testing tools can help extensively test an application. However, what these methods are commonly missing is the real user interaction.

It is therefore one of the defining characteristics of crowdtesting, that thousands of real users can help test software both functionally and non-functionally in real environments.

It is important to note, that crowdtesting and other testing methodologies have their benefits and drawbacks and are more suitable for specific types of testing. As mentioned above, crowdtesting can speed up testing endeavours and detect a higher number of bugs more quickly. However, it does not replace traditional testing methods completely. A number of functional testing methods with rigid structure and test cases might work better for some testing purposes.


In our attempt to explain what crowdtesting is, in this article we have first tried to determine  its core concept, crowdsourcing. We have considered the commonly misinterpreted cases of crowdsourcing, and why they should not be labeled as such.

Following the foundations of crowdsourcing concepts, we then explored its benefits and how they apply in testing. In this case, these benefits are reflected in many ways, besides the obvious.

To see for yourself how the magic of crowdsourcing applies in testing, you can request a Demo from us, and get access to the crowd of 40,000+ testers and 10,000+ devices from around the world.