Test automation tools: An experienced insight

We have prepared a three-article series on test automation for everyone interested in the quality of digital projects. The first article helps you identify how to achieve your goals through test automation. The second article will provide an introduction to Flutter automation testing, catering to Flutter testers and developers. The third and final article in the series delves deeper into Flutter test automation, specifically tailored for Flutter developers.

Here we will look at the early consideration of automation tools, and how these tools can be leveraged to enhance the achievement of your goals, advance your missions, and effectively tackle the challenges you seek to resolve.

It’s high level content aimed at the whole team, whilst it won’t provide you with specific solutions it will hopefully give your team some thoughts and ideas to discuss to empower good decisions with regards to automation.

Don’t jump in, just yet.

Automation can at times get labeled with almost magical expectations that it is “the” go to solution for quality problems.

The reality is generally very different. Up front quality evaluations can often highlight both automation opportunities and flag other opportunities to improve things that may be better suited to the problem you are trying to solve. When it comes to product automation we suggest a few fundamental principles to consider before getting started.

Some of these may seem initially counterintuitive but they are based on a lot of real experience and projects in this field so please bear with us. We believe the concepts could be invaluable to our customers particularly when it comes to getting everyone’s expectations aligned.

Basic principles before we start.

Your decision to leverage automation follows the same consideration as any other tool; what problem are you trying to solve, what are your goals and the expected value you want it to provide. 

With specific regards to test automation, if your primary goal is cost or effort savings it is an amber flag that you have other issues in your development or testing process. We can help you evaluate that risk.

Test automation scripts are really good for verifying known risks; test cases either pass or fail.

New products have far more unknown risks, even when automation is fully in place. It makes sense to maintain a holistic testing view that also includes other needed testing areas and development practices. Automation will not replace this good practice but is a necessary extension.

Potential goals of automation

Consider your automation goals in advance of any other decisions. Automation can be used in many areas of software development. The following list includes a few of those ideas even though we won’t dive into all the areas within this article.

These goals not only impact your decision to leverage automation but will also help with other factors such as who will automate, the level you automate at and what tools you will select to assist you.

Layers of automation

The consideration of layers is very important because it asks the question, “At what level is it most efficient to catch a specific type of issue should an issue of that kind exist?” The unit and api layers tend to have narrower goals;  to get the rapid feedback loop from test results to the team.  It has great value in daily development work as developers quickly learn how well the code works. Ideally these tests are quick to create and run but also easy to maintain. 

Some of the ideas behind the Automation pyramid align with these key attributes. The pyramid also indicates the track to find a root cause for a single feature error. One should start tracking root causes for errors from the lowest level of a pyramid and only then move upwards. We don’t recommend blindly following this model but it is potentially useful for considering if it’s a good match for your current project.

The pyramid is very specific to test automation and does not apply to your overall testing model.  For a much broader viewpoint in risk management the holistic approach  is recommended.

User facing end to end flows

The layer is the most complex layer to automate at, it has more dependencies, higher maintenance costs and a slower feedback loop than the lower levels. At this layer good development architecture practices are needed with automation, as are basic coding skills.

Mobile App automation can further complicate things compared to web apps, as there are less tools available and it’s often less testable. For example on a web application often it’s a straightforward case of directing a tool to a url, whereas on mobile apps you may need to consider things like emulators, debuggable builds, WebView’s outside of the application and native device configurations. It remains a very important opportunity for automation though particularly when it comes to the significant device variation risk. When it comes to the end to end layer and finding issues, hands-on risk based testing will often offer much more value. 

Common automation models

Here are two example models we have leveraged from both where our own team is building the product and where our automation engineers step in to help out our customers teams.

Efficiency focused team model

This model really suits new products where close team collaboration is at the center of development. It’s designed to keep the automation code even at the end to end layer as close to the whole team and product code as possible.

The combination of high levels of collaboration, whole team contribution and building in parallel to product code make this a highly efficient approach to leveraging automation. 

It can be challenging initially for testers as some coding is required and we’ll often pair developers and testers together on automation to get the most out of this model. Our next article about Flutter automation will go into more detail on this approach.

Regression or complexity risk model

There are also many products/services out there which have run into regression issues and over time it has become detrimental to the team.  Automation can be used to primarily reveal the issues and to cover this risk.

It’s often worth optimising the efficiency of the Agile team model by utilising test automation so that it frees the developers from fighting regression risk fires on a daily basis.

In the hands of full time automation engineers the end to end coverage will often go broader and deeper. As it is distanced slightly from the developers and the code, there is the potential to utilise more tester friendly tools. Once broad coverage of regression risk is in place it can be easier for the team to start addressing the root causes of the risk and consider a more optimal model going forward.

Wrap up

We hope you have enjoyed reading some of our thoughts on automation and perhaps will take a few ideas that can promote active discussion before you jump in. It’s a great tool to add to your software development toolbox and help teams from many angles. We have a lot of experience in this area so get in touch for a chat and we can explore some options together. Or stay tuned for our next article that focuses on Flutter test automation.

Ready for the second part of the article series? Read it from the blog of Rebel App Studio by Codemate!