Pankaj Nakhat's Blog about Testing, Automation and Agile Testing

Agile is no longer a buzzword or an unknown territory in the industry. Agile has come forward leaps and bounds since last few years and has seen a great transition from a newbie to a widely accepted methodology.

Testing on agile project was a paradigm shift for the traditional testing roles. It needed a change in tester’s attitude from a laid back relay race oriented approach to an upfront involved role. No longer had QAs needed to get something over the line to know the quality, as approach was rather to do the things right first time. But it’s easier said than done, how does it happen in reality? Does it actually happen?

Agile is a philosophy which believes in delivering constant value to customers incrementally and frequently, which is based on simple values of communication and feedback. These two ingredients are the ingredients of a successful agile recipe.

A tester has an important role to play in agile projects throughout different phase of iterations. Iteration is generally a two week long activity where small requirements are analysed, coded and tested.

What QAs needs to do? I am outlining few of the basic QA activities in agile projects. Let’s simply break the QA activities in three phases in agile projects. However, this is no golden rule and it can be flexible according to the project situation. Agile QAs role is not limited to set of pre defined processes, as the methodology they are expected to play an agile role as per the situation.

Pre iteration : This is the time where requirements are analysed in detailed by the BAs and acceptance criteria are detailed out for that story. As QAs are immediate consumers of those requirements, so it is really important to verify these requirements early and often.

Story verification : Agile testing is all about giving feedback early, not necessarily only by testing the requirements, but doing requirements testing early on. What should one be focusing on while doing requirements testing?

Looking at requirements for Clarity and Testability - QAs really need to look at the requirement / stories upfront to figure out that requirement are unambiguous and testable.

Unambiguous- I think unambiguous in agile has little different context compared to typical waterfall project.

- Requirement should be small enough to make sense in the context.

- Acceptance criteria (Stories are generally broken in to acceptance criteria) should not be duplicate or overlapping from different stories.

However, doing that can be really difficult and can only be achieved with really good communication between Dev/BA/QA.

Testable: Testability aspect of the story requires QA to scan through the story to see what needs to be done to test the story. These factors can generally be.

- Finding hidden requirements

- Environment

- Test data

- Dependency on other requirement.

Getting these details early helps the story to be prioritised accordingly in the backlog, and allows smooth execution of the story in the iteration.

QAs also do participate in the iteration planning meeting to help estimating and giving testing perspective to the team to come up with developer estimate. QAs have big role to play in the iteration planning, as some of the implicit requirements gets escalated by

QAs.

a) QA Activities In The Iteration

Once QAs are happy with the acceptance criteria of the story they can work towards defining the acceptance tests for the story. Acceptance tests are requirements in terms of a test, so that it can be executed before hand to know what is expected out of requirements. These acceptance tests are generally automated and used by developers to drive development, Acceptance tests should not be covering too many corner case scenarios as this would create unnecessary delay in getting the story done by the developers and may end up in producing too many similar automated tests.

People often fail to understand that acceptance testing in agile projects is different from traditional projects, as in agile projects it happens before the software is delivered unlike traditional project where it happens at end of the software lifecycle. In agile projects acceptance tests are generally automated so that it can be run as regression tests.

Automation testing is really important for any agile project as frequent builds requires a short feedback cycle hence regression testing needs to be quick and accurate. Automation testing is different from traditional automation testing. In agile projects automation testing is practised by all levels be it Developers, QAs or BAs. It does not necessarily mean that everyone needs to be writing the test code. Involvement from everyone on automation testing increases the relevance of the tests and often helps in finding and implementing the right tests.

It’s always been debatable as who owns the automation tests in agile projects? And for me it’s more of a responsibility then a role. In my experience most effective automation tests are achieved when Dev/QAs work together on it.

Use Automation to the advantage not for sake of doing it

Automation in agile is quiet controversial, as people try to automated everything and end up in a trap of having a long feedback cycle. Automation is meant for giving early feedback on the latest code, and automation should be limited to what is worth Automating and what is not. Every automation test written has a cost against it. The cost can be seen as

Cost of continuously running it (Increased feedback cycle). This cost of automation should be compared against cost of not running it. So, a question needs to be asked what if a test is not automated. What we going to lose, what would be the cost of fixing the stuff around the code for which we are losing the coverage? Is it cheaper to test manually?

And it might not be straight forward finding the worth of a test. It’s a contextual decision and also depends on size of the project and number of people involved in it. Simply putting this in other words mean Longer feedback cycle = More people losing time in getting instant feedback.

The typical QA activities in the iteration are to continuously measure the Quality of the software. QAs participate in the story handover to the developers to make them understand the testing requirements of the story, so that developers are enabled for Test Driven Development(TDD) for their code, which is a backbone for agile development practices. Also handing over the acceptance tests and making developers understand about the testability aspect of the story often catches the most common defects even before they are delivered to QAs. These activities require a high level of communication with the developer and BAs to clarify things on the fly and making sure we are building the product right first time. QAs may also have to pair with developers work together on the story or tests for the story to get the better understanding of the requirements.

QAs try to resolve as much issue before hand by actively participating in the overall process. It also becomes imperative that once the story is delivered, it is tested properly with all the aspects and ideally on a proper environment. Once the QAs are happy with the stories/requirements, they sign off that piece of work in a typical manufacturing way and hand it over for further process in the line which generally a showcase to the business.

As a QA it is also important to think beyond the written requirements and to have a desire to break attitude, and that’s where QAs do exploratory testing to execute ‘out of the box’ scenarios and also do negative testing to make sure software is really robust. Exploratory testing is a not at all about executing a pre defined testing scenarios, it is more of exploring of software beyond test cases but at the same time keeping the focus around some specific requirements.

I have tried covering most of the aspects of the basic QA activities of a QA in an agile project, however it is not limited to these activities and QAs should play more collaborative role as and when needed to be able to provide more visible and transparent feedback.

I want to make few points quickly in favor of that.

1) No support for programming languages like Java, .Net so that it allows it to be used as an acceptance testing tool.
2) VBSCript doesn’t help in everyone’s participation in the automation process, which is a key in agile projects.
3) No IDE support : QTP’s IDE is no way near Ideal hence makes maintainable a tough battle.
4) NO support for standard IDE like eclipse //
5) Multi browser support is limited/
6) Cant execute scripts on Linux and Mac platforms.
7) No continuous integration support, makes it rally tough to be used as an acceptance testing tool.
And last but not the least : Bloody expensive