Finding the right balance between automated and manual testing

Author: Ryan Yackel, Senior Sales Engineer, QA Symphony

The truth about automated testing
It’s easy to fall into the trap of thinking you can set up an automated test once and have it run on its own after that. However, the truth is that automated testing is not as 'set it and forget it' as it often appears.

In reality, you need to regularly maintain the source code for all of your automated testing scripts, which includes updating the code alongside application updates. Failing to maintain the source code can lead to false testing results.

Another common misconception about automated testing is that it’s entirely tool-based. While the tool does matter, automation is still about people. That’s because a test automation tool will not do all the work for you - you still need testers who are knowledgeable about automation to operate that tool, develop the scripts and maintain the source code. A linear, 'record and playback' approach with non-technical resources will never be maintainable.

Balancing automated and manual testing
These misconceptions aside, automation remains a very valuable capability in the testing world. So how much testing should you automate?

The fact of the matter is, you simply can’t automate everything. Saying that you can automate everything assumes that you can test every single bit and piece of code, which you can’t do. From a test coverage standpoint, 100% coverage is a pipe dream. It’s just not possible.

Even if you could automate everything, that would not be the best approach. Neither is automating all of the testing that you will undertake. This is for two reasons:

1. Maintenance. The more tests you automate, the more source code you have to maintain, and that can become something of a rat’s nest. In turn, missed maintenance can lead to issues such as false error reports, that you’re unaware of. In contrast, a human tester would be able to identify issues in test and user experience alignment to remedy a mismatched setup that would lead to false error reports.
2. The human aspect. In general, automation removes the extremely important human element from testing. Beyond the issue identified above, manual testing allows for more accurate testing of real-world scenarios than does automated testing, and that human perspective is not something you can afford to sacrifice.

Top areas to introduce automated testing
If you can’t (and shouldn’t) automate everything, what testing should you automate?

More often than not, you’ll want to leave the more complex areas of an application to a manual tester because there are more things that can go wrong. For example, if you are trying to automate an entire end-to-end flow between multiple applications and different technology stacks, the script is more likely to break. Having a human at the steering wheel can make it easier to identify those wrong turns.

Beyond that, here are three ways to approach the decision of what to automate:

1. Risk-based approach. Identify high-risk areas of your app and automate a smoke test, a simple test with a high impact. For example, if 90% of your users have the same type of user profile, you might want to automate a test for logging in with that type of profile since any issues would impact 90% of your users. The risk of failed login for the remaining 10% is not high enough to warrant an automated test.
2. Conversation-led approach. Most context-driven manual testers are subject matter experts who know the systems they test inside and out. Having these testers consult with automation engineers to identify areas for automated testing, such as smoke testing or testing against other applications, can go a long way toward understanding where automation can add value (and where it might not add value).
3. Exploratory testing approach. Exploratory testing can often feed into conversations about automation. That’s because during exploratory testing, you gather and record knowledge and issues. You can then use this information to decide where it makes sense to automate testing.

With the release of qTest 8.1, qTest eXplorer now has the ability to record and identify objects in your web application. This allows testers to then export the script and then modify it from Selenium (C# or JAVA) and Protractor. 

Measuring the value of automated testing
Last but not least, as you automate tests, you need to measure the value of that automation to ensure it’s providing the results you want and returning a value greater than manual testing would provide.

For example, if you run an automated test 100 times and it passes every single time, is that test really providing any value? If the results are indeed accurate, then the test may not be a valuable one period, unless it’s a high-risk scenario. However, if the manual test finds more defects, you have to ask what’s more valuable: the time-savings that come from automating a library of tests or uncovering actual issues by running manual tests?

That’s not to say that automated testing isn’t valuable because it most certainly can be; but it’s not a blanket solution. Rather, it’s an approach that you need to take strategically and revisit regularly.