Are you ready for continuous integration?

Test automation would seem to be a pretty technical excercise. Don’t forget there’s a lot of communication involved too. This article describes what to tackle first and answers the following questions:

You’ve decided you need test automation. Awesome! Next phase: Get cracking! You’ve read what considerations to keep in mind when creating a successful automated testing strategy. So you’re good to go. Or are you?

Test automation is the process of automating tests. A set of checks distilled from test cases that are a translation of acceptance criteria. Clear user stories are the first prerequisite for test automation.

Level 1.) Write clear acceptance criteria

Testing is communication. It’s the art of translating acceptance criteria into test cases and then executing those. It’s about proving to the product owner he got what he asked for. Note the word “proving”, not “telling” nor “convincing”. It’s really not a technical exercise and it shouldn’t be. It’s a conversation in which sometimes questions arise. Scenario’s the team hasn’t thought of. Or scenario’s the product owner thought were in scope, but according to the team, weren’t. Some are easy to fix, some aren’t, some should be fixed and some shouldn’t.

Update your definition of ready, if you haven’t done so already and only get started on a story if it has clear, scoped acceptance criteria! Know what to test. Test automation begins in refinement sessions. Create clear, testable acceptance criteria. It takes practice to get it right.

Level 2.) Write test cases

You’ve got your stories sorted. Great! Next step: Creating test cases out of the acceptance criteria. These must be behaviorally described test cases, which — combined with their results — result in an effective way to communicate about the systems state.

You’ll probably need multiple test cases to cover a single acceptance criteria. You’ll need them do cover the edge cases. The Gherkin language can be a good way to describe behavior instead of technique. Execute them and use them to communicate with the product owner.

Level 3.) Agree on what to build and test. Agree on how to communicate about it.

Nothing can prove that a system works. There’s no such thing as “works”. You’ll have a description of the systems known behavior at most. Stakeholders will decide whether that’s sufficient or not based on that.

Development teams cannot start building before having an understanding of the clients wishes. In many development teams the developers think they understand and start coding. The resulting product will be tested by someone else at a later point in time. The tester will then mark the work item “done” and the team moves on. That approach has “assumption” written all over it. Would it not make more sense to have the development team and the stakeholders describe the test cases together?

Level 4.) Test first

Write these test cases before building the product. Note that common testing techniques got lots of “D”’s in them: ATDD, TDD, BDD. Each one of these abbreviations has a “D” in it that stands for “Driven” and one that stands for “Development”. The development process is driven by these tests.

Describe the systems new behavior collaboratively in the refinement. The behavioral documentation of the system are the test cases and vice versa. Create living documentation by doing so. Start testing the right things before testing it right.

Level 5.) Automate them checks

Note that nothing of the above mentions anything about automation. And I hate to break it to you: It can’t be done. Testing can’t be automated. Test automation means you’ll automate testing with some tool or framework. And as James Bach says: “Tools don’t test. Only people test. Tools only perform actions that ‘help’ people test.”. Checks can be automated at best. Look at test automation with a DevOps mindset: Automate everything!


You can’t automate tests if you don’t know what to test. You can’t start automating tests if you don’t know how to communicate about these tests either. Testing can’t even be automated in general.

Test automation should be called check automation. Only checks can be automated. Test automation is — if done right — a tool that will help teams to validate the systems behavior more often and faster.

Use test automation as a tool to do testing. Focus on testing the right things instead of automating tests. Use test automation as a tool to test faster!

software developer / consultant @