Scoping Test Automation Part 2: Post Release

A distinction between test automation during development and post-release.

Published: 2017-03-30 23:00:00

If you haven't read part 1 yet, you can find that here: Scoping Test Automation Part 1: In Development

Test Automation Post-Release

Post-release, deployment, the software under test is now packaged. We are beyond maintaining any code base. Instead we are maintaining the delivered software itself along with it's specific configuration and any dependencies required to keep it running. Beyond unit tests, beyond any automated feature tests built and run within the code. Those automated feature tests can't help us now. We are more interested in acceptance of that software. A lot of the tests run against that software now will be "black box".

The delivered system will now have users wanting to check that it meets their specific requirements. QA teams will now want to run a raft of test cases ensuring that given inputs to that system have expected outputs via any user interfaces, export files or the database. Test automation in this scope therefore seeks to mimic these interactions, the interactions of users with the system.

Test automation at this level is external to the delivered system. Given inputs to the system must be provided via APIs, message protocols, the system's database, pretty much any interface available. The only interface not available is the code upon which that system was built. Similarly, any interaction with the system can only be via external touch points with that system, likewise for validation of any expected results.

Where automated feature tests (within the code) could interact at the code level, automated functional tests can only interact with the system at its boundaries. We don't need to rely upon any frameworks here, any tool that can interact with an API, a user interface or a database will do. The only other requirement would be that that tool allow us to write some instructions or logic to tell it how to perform those interactions and how to interpret results found. We could just as well write our own code to do this or employ any number of other tools already built for this purpose.