"How did QA miss this defect?"

Myth: The main task of QAs is testing and finding defects. Therefore, quality is the sole responsibility of QAs. When a defect is missing, QAs should be held accountable for the incidents.

Fact: Everyone from product managers to developers is in charge of making sure that features can and will work when they're handed off. There are many phases from requirement gathering, planning, coding, testing involved before features are handed off to the users. But QAs do have an especially difficult task: finding as many defects as possible at this final stage. However, if a defect is missing, it's because something might go wrong somewhere in the process, not only in the final steps.
What's behind the myth that "QA is responsible for missing defects"?

‍
In the traditional development process, testing happens very late. It's the only time where issues are found and reported. People rely on QAs to find and report all defects before a big milestone release. Of course, this leads them to think that if a defect is missed, there wasn't enough coverage.

Culture is another important factor. Developers and testers are two separate teams, and each has its objectives. Testers are supposed to report all possible errors created by developers' code. Having developers and testers as separate teams means no cross-functional collaboration and no responsibility for product quality.

Besides, quality is not included in many job descriptions of product managers or developers' roles as part of the responsibility. Hence, quality is often overlooked when people join and mistakenly believe that QAs will handle and be responsible for quality.

Anyway, let's have a more detailed look into why defects are leaked to production from different perspectives.
‍

Why were defects missing?

There are a couple of reasons why QAs missed defects and let users discover them:

‍
QAs are human

Of course, QAs are human, and humans aren't perfect. That's why defects happen from their mistakes, such as missing test cases to cover obvious acceptance criteria, using invalid test data, not configuring their testing environment properly, and so on. With this in mind, yes, QAs should be responsible for such mistakes. But we'll get further into what QAs need to do in the next section.

Lack of skills

The lack of training for new hires or junior testers regarding the product and its related systems can be one reason why they don't perform effective testing. If a new hire or junior does not have adequate knowledge and training, it's easy to miss an edge case or a dependency while testing a new feature.

On the other hand, many factors are out of QA's control.

Product complexity

Firstly, due to the complexity of the product and interrelated subsystems, it's hard to identify all possible scenarios in a short time. Additionally, because the product is complicated, it can be challenging for testers to mimic the production environment. This discrepancy in the testing environment sometimes causes issues that are only reproducible in the real environment.

Example: On the day the team deployed a feature to production, we experienced an awful performance issue where a SQL query crashed the production server because one of the columns was not indexed. It costs us time and effort to kill the query and roll back the servers to normal conditions. The query worked perfectly on the test environment because the database is much smaller than the production one.

One more factor related to the environment is when a user has installed something (e.g., an extension) on their computer that might block the code of application render on their browsers. If the volume of issues like this is high enough, this could be considered a missing defect.

Time constraints

The most common reason we all know is the time constraints that prevent thorough testing. Let's do the math here to understand the problem better.

Assume that an application has 100 test cases to be executed to ensure all features are tested before every bi-weekly release. Bird (short for Bird Eats Bug) wants to support all different OS such as Windows, macOS, Linux, ChromeOS, iOS, and Android. The app should be working on multiple browsers such as Chrome, Firefox, Safari, Chromium Edge, Opera, and more on each OS.

How many test cases to execute: 100 test cases * 6 OS * 5 browsers = 3000 test cases.

However, this is just the math to show the total number of tests to have the best coverage so that nothing is missed. That is not only impossible to apply in the real world; it's also not a wise usage of time and effort.
‍

How to reduce the missing defects?

It is important to emphasize that there is no bug-free product. Bugs always exist, and we should accept that testing shows defects and testing everything is impossible. Suppose everyone in the team understands these principles; it'll be easy to build a culture where quality is a shared responsibility and missing defects will not be a big deal and can be easily reduced.

We should keep in mind that having fewer bugs does not equal having good quality. Thus, the purpose of testing is to help reduce the chance of introducing bugs and provide confidence for the product to be released within a limited amount of time and effort.

Here are the list we could do to improve quality and reduce leaking defects:

• Apply shift left testing: QA should get involved earlier in the development cycle and testing should happen when requirements are being defined
• Coaching and training: QA should share testing approaches with developers to help them improve their local testing. Senior QAs should provide test case review to give constructive feedback for junior QAs. Ensure required documentation and test cases are clear, complete, and cover all the user scenarios.
• Team work: Perform thorough code review and test case review
• Improve estimation: Don’t be rushed. Give enough time for developers to develop and test their implementation. Give enough time for QAs to perform enough testing.
• Follow Agile practices which are reiterative and adaptive: Learning from failure and improve the QA process overtime
• With time constraints, always implement a wise testing strategy to minimize the tests that need to be run but have the best possible coverage: prioritize to test critical features which are the most common used by users and perform exploratory tests for the uncommon features. Communicate the issue and get buy-in from the team about what can be tested and what cannot be done so that it’s not a surprise when a defect is missing because it’s not been tested. By using this strategy, defects will be introduced and how many defects go into production depending on how much testing we can cover.
• Implement automation testing strategy: It is impossible to test everything manually within a limited amount of time, especially when more features are added. It’s more than proven that automation testing has helped improve testing effectiveness and efficiency. The most common practice in Agile testing is to build the automation testing pyramid which implements the automation at lower levels as much as possible

How we at Bird build quality culture

We want to offer the best product to our users so we release new features to production very often. At Bird, everyone is responsible for quality, from product managers to developers and testing happens on every stage of the development cycle, from design, coding to validation. QA plays quality assistant role and performs the last check before the release

Before feature are built

• Design will be shared by product managers to get feedback from everyone. There are a lot of discussions back and forth until everyone is happy with the final version and is on the same page. This could be considered as static testing because testing happens without actually executing any line of codes

While features are being built

• Developers start writing the code and perform testing by themselves in the local environment. When they finish, they make the pull request for code review by other developers.
• QA creates a test checklist and shares with developers how features will be tested when they are ready. By sharing the testing checklist, developers are aware of test scenarios will be performed by QA and they can test it by themselves

Once pull requests are approved

• Once pull requests are approved, it’s time for QA to do the final check. As you can see, the features would be nearly perfect when handing off to QA if everyone was testing their own work. QA is only the one who assists with test cases, asks the challenging questions and try to break the features at the end
• The most important thing is what if a defect is missing? Well, for us, we accept the fact that it will occur because of our frequent releases. What we have been doing is to work together to understand issues reported by users, trying to find the steps to reproduce the issues, identify the root cause, fix them quickly and release fixes as soon as possible to make our users happy then perform post-mortems and retrospective to learn from mistakes and don’t let it happen again
• One thing about the culture at Bird is that everyone “eats their own dog food” by using Bird for various purposes, from presentations, communication to reporting our own bugs during development. We try to step on users’ shoes as much as we can to not only catch missing defects but also understand our users' problems and create solutions to help them achieve their goals faster.

How can Bird help your team reduce missing defects?

As we've covered above, time constraints are the most common problem that prevents thorough testing. Bird is a time-saving tool that will help you (and anyone on your team dealing with bugs!) file an effective bug report and upload it in seconds to any bug tracking system you're using. Installing takes a few seconds, and you can take a screenshot of the issue or record a video to describe the steps to reproduce actual vs. expected results. More importantly, all the system information, console, and network requests will be automatically recorded while recording a video. Such information is super helpful for developers when it comes to debugging and fixing bugs.
‍

Last but not least, you can upload bugs straight to Jira, Github, Linear, or Trello directly from the Bird app. Bird frees up more time to focus on testing to find more issues rather than spending your time on reporting bugs. The more time you test, the higher your test coverage, and fewer bugs are introduced to production.

Conclusion

The question "How did QA miss a defect?" should be changed to "Why was a defect missing?". By changing the way we ask, the objective is entirely different, which is a missing defect. Therefore, the solution will prevent the defect from happening again, and no blaming game will occur. 

It's essential to build a culture where everyone is willing to make mistakes, learn from failure, and constantly improve.