Tester Items

Software Products need their own unique approach to test adequately and correctly. Oftentimes, teams treat them like any other software (i.e. internal applications built for a specific client or team; not accessible by the general public; non-revenue generating) and that is the starting point of trouble.

Software Product Testing needs a custom test style and strategy to add value. Software Product development and sustenance is in itself a complex ecosystem and to thrive testers need to adapt.

Let me take a moment to explain why it is important and why I think Product development is complex, complicated, and composite, even at the best of times.

#1) Lack of control over user demographics, devices, environments, platforms, etc.: Software products unlike software built for specific stakeholders are not used in controlled and predictable situations. There are just too many factors to take into account.

#2) Foggy product vision: Product behavior and features are forever changing and the journey to maturity isn’t clearly visible. Or the product is growing too rapidly that it spirals out of control that teams don’t know what is happening.

#3) Aggressive timelines: Due to heavy competition in the software product market, things have to move at a breakneck speed and teams must stay a step ahead of their peers. Otherwise, they are sure to lose out to the competition.

#4) Fear of failure: Software products are usually innovative. So, their success is not always a given. This is the reason companies can’t go all out in terms of budget, technologies, infrastructure, etc. They often have to hold back to gain a certain amount of immunity from failure or to even break even stage.

#1) Product Introduction

Since this is the first time TrackFast would be going out into the market, the idea is to make a good first impression. So leave no stone unturned. Test everything and from every angle. In addition to that, lay the foundation for future testing.

A good test strategy at this point should include the following:

1. Tests that validate the short-term goals of TrackFast. “What does it need to be shipped correctly” should be at the forefront of the testing effort. Create End to end tests (front end, middleware, and backend) for thorough testing of every feature
2. Tests that compare TrackFast with the competition (ideally this is the job of product owners but as a tester, we can add our two cents. Also, this step is easier if the software has some peers already.
3. For example, It is easy to compare TrackFast with Bugzilla or JIRA or other legacy systems. But let us say I am creating an app that does something unusual like being able to predict when a baby is hungry or cranky :), it might be hard to find an application that you can use as a baseline)
4. Platform, browser, and device compatibility tests
5. Tests for ease of installation, set up and getting-up-to-speed
6. Tests for performance, security, and usability
7. Integration Tests if it interfaces with other systems. A simple integration example is that Defect tracking systems often interact with email clients to send notifications
8. Plan for regression– It is a good idea to flag or mark critical tests that you think will be a part of future regression cycles and think about automating them for future releases
9. Plan for known issues (are you going to be adding them to the backlog or handling them as CRs, etc.)
10. Flexibility to change when the product progresses to the next life cycle stage.

It could sometimes be a long wait before the product goes out, so use all the time you have to do as thorough a job as possible.

In this stage, though there is a piece of the product ready at the end of 2-4 week sprints, most often every sprint does not result in shipped code. Therefore, never consider last sprint testing ‘done-and-delivered’. Repeat critical tests with every sprint until release. With each sprint, test the entire product that you have until that point.

Stage #2) Product Growth

After the initial project introduction, if all goes well, expect an influx of activity because Product Growth is a fast-paced lane. You are now swimming along with the big sharks and unless you keep up, you get gobbled down.

Here, the releases get shorter, the improvements done to the software become more in number, and the extent of regression almost becomes unmanageable.

The product testing strategy should work with the pace that the software development is proceeding and should not become a bottleneck.

These can help:

1. Keep in mind the long-term goals of the project. It is not about getting it over with now. It is about living with the features and thriving with them.
2. Test Early- Consider TDD or BDD instead of deferring testing to the end with new requirements
3. Automate Regression and strengthen it– Create an automated regression suite in place so you are not left with untested landmines in your system
4. If your business/product owners want to get involved with testing, consider a business language-based automation tool such as Cucumber.
5. Keep usability and site design central to your testing. Because the more features we add, the cleaner the site should look
6. Perform performance and security testing when a major release has happened or there is a significant change made to the architecture. (New server brought in, etc.) Most software systems don’t need this with every release.
7. Keep in touch with the competition and know the product vision
8. Adapt pair testing, for immediate feedback and fixing. Include the product owner when possible
9. Plan for changes and known issues
10. Try to get your hands on the customer feedback and check if they are can be tracked as enhancement suggestions to keep the growth constant. (once again, this is not the primary responsibility of the QA team, but every one counts)

Stage #3) Product Maturity

Congratulations that your product has come this far. At this point, the features don’t change as often. The product team is going to be more focused on bringing more business or their marketing efforts. However, product development and testing need not and often don’t stop.

Therefore, the testing team can:

1. Work on maturing your test strategy. By this point, your regression suites, test design methods, and test management practices must work like well-oiled machines.
2. Focus on the finer details. Because overall the product works and is doing well, but as they say- ‘God is in the details’– find even the smallest of the problems that can improve the quality of the system
3. Consider customer feedback
4. Test Performance and security periodically
5. Take into account the new devices, platforms, and browsers that might have come into the market from the last time you tested
6. Test User Manual and FAQ pages because by now you have the time and you can afford to.
7. Experiment with a new product test tools, services, or a process because now you can.
8. Test the installation process with every release, however small that might be, and get statistics as to how easy or difficult it is for the end-user.

Whatever you do, don’t get complacent.

Stage #4) Product Decline/Circling back to Product growth

The product owners and businesses are smart these days and know very well that they can’t keep their products the same and expect the users to stay loyal. Things move too fast and so do products.

So, Track fast can’t sit back and relax. If it needs to have a continued market presence and stay the leader, it needs to evolve. Like it or hate it, Facebook started as a simple social network to connect people and it is a large software platform in itself integrating with a million other things and staying current.

TrackFast too has to evolve. After proving that it is a reliable and effective defect tracking system, it has to evolve or it will decline. So, the company XYZ decides to improve Track Fast by making it a general ticketing system that can be used to track any incidents or cases by the business other than IT/test teams (something like JIRA) and not just for defects in the software development process.

The wheel has made a full turn and you find yourself treating the system as a brand new one and follow the strategy we discussed in the Product Introduction section. Only now you are more experienced and familiar with the drill. But remember, with each new turn comes a new challenge. So stay sharp :)

What makes you a successful product tester?

1. Product testers must have a keen business sense, understanding of fast delivery development models, and should be ace testers who are not afraid to experiment with tools and become a bit of coder themselves if need be. These things can have a positive impact on any type of testing, but they are an absolute necessity in this type of testing.
2. Another important quality is that a product tester must believe in the product and genuinely want it to succeed. When I was a tester think that the software is total garbage, there is little hope that I will do anything to make it better.
3. Share the product/business owner’s vision. Unless you know where the product is going and how it is going to evolve, the testing will be super limited.
4. Cross-functional skills are beneficial– Know how to test the DB, how to take performance benchmarks, how to enable security certificates, how to deploy, etc. Be curious and explore.
5. Set no boundaries– don’t think that evaluating the user manual or checking the FAQs is not your job and a technical writer should take care of it. Well, they should and they will. But when you look at it as an insider as someone who knows the product inside out, your feedback is super useful.
6. Seek end-user feedback. The next big set of people who test after you are the real-time users. Know and understand what kind of problems they are facing. This helps you improve your test design so next time you know what to do to avoid those problems.
7. Work fast and be a decision-maker
8. Avoid technical debt. In a fast development and testing situation, it is easy to test exploratorily exclusively and lose the frame of reference for future releases. Don’t let this happen. Maintain skeletal documentation so you can track, trace and measure

The biggest difference between testing software built as a service and software built as a product is that – in the former, once the test strategy is arrived at, it is applied for all subsequent testing.

However, for a product, the test strategy has to change depending on the current life cycle stage the product is in and the changes to the market dynamics (new devices, new browsers, etc.). Product testing strategy needs to be much more flexible to change..

#5) Lack of actionable feedback: Since there are no stakeholders or business users or clients, so to say, it is difficult to understand what the end user may or may not like. Companies are constantly playing a guessing game and often have difficulty in bridging the gap between what they want for the software and what the customer wants.

These challenges affect all areas of product development, marketing, and sustenance- And they inherently impact product testing too.

To get ahead in the game, this type of testing has to take five key points into account:

1. Speed of development and releases
2. Short term and Long term product goals of the product
3. Extent and nature of competition
4. Target audiences and their environments
5. Requirements – Functional, performance, security, usability, configuration, etc.

Before we go into more details, let’s understand the product life cycle (This is a generic product lifecycle and not specific to software products but software follows a similar pattern):

A good Product Test Strategy/Approach should take into consideration the current stage of the product in its life cycle.