Why PMs should do their own testing: An interview with Maria Martynova

Can you share some general thoughts on testing in companies?

For me, testing is not something that just one team does — everyone should be involved. “Be your own user” is a saying that creates a healthy environment. Everyone in the company should understand why we are building this particular product, how we can enhance the user experience and, of course, find bugs.

Quality, in general, is the responsibility of everyone in a company or organization. When PMs start working on features, they include expert opinions from QA and developers, among others. Everyone is working towards the main goal to create a better feature. PMs don't define features alone but together as a team. PMs bring a problem, and the whole team looks for a solution together. It seems unfair that if a solution is found together as a team, the issues that arise fall solely on QA.

“My experience is mostly with early-stage startups, meaning it’s really “Zero to One”. Therefore, testing becomes a normal habit for me rather than something that “needs to be done” — it comes naturally with my role.

PMs are the most knowledgeable about the product. Therefore, it’s logical that no one can test and ensure quality as well as PMs. What are your thoughts on this?

PMs should know the product end-to-end. They talk with stakeholders, designers, and combine all this information to build a product with the rest of the team. They usually know the most about the product, especially from a user’s perspective. I’m not a technical PM — I don't come with a developer background. This means that I will not be doing any automated tests or seeing how the code runs. I see these as the responsibility of developers and QA.

When testing user experience, PMs can spot small things like typos, buttons, colors, something that’s hard to automate, or spot problems that are not bugs but rather gaps in the user experience. When testing, PMs can also notice missing logic, where everything was built correctly, but what's missing is not visible until we've started building the product.

What is the role of PM in testing?

I see PM, QA, and developers' type of testing is entirely different. I think that QAs are responsible for setting up a framework: from how we should test things in general to how to automate tests. QA should also make sure that we ship high-quality products.

When it comes to developers, I believe that developers should test their code, run code through automated tests, have enough test coverage by creating more tests, and test on user experience. PMs need to make sure that what is built are the features we want to release and it works according to set expectations.

Most companies, especially startups (not only early-stage), usually don’t have QA. In this case, all the testing falls in the hands of PMs, developers, and everyone PMs can involve in testing. After such experience, you would appreciate any QA you can have and work with them as peers rather than pushing all the work to them.

What kind of problems have you personally seen regarding the QA setup within a company?

I've experienced several problems with how QA can be abused and ineffective in a company in my career. First, and personally the worst one in my opinion, is when QA becomes a bottleneck because developers don’t test anything and everything is pushed to QA. This results in everything rolling too slow, sprints are never done, and QA is blamed for everything. It affected the team morale and team members ended up not supporting each other. It truly contributed to creating an unnecessarily competitive environment within the company — and an unhealthy one.

Other problems I see are when QAs are not involved enough. I think QA should work alongside developers, PMs, designers — fully being part of the team, not separated from them. They should be included in feature requirements ASAP. I prefer to include QA once the first draft requirements are ready. This applies even when the draft may not contain all the details, but once we know why we want to build something. QA input is super important in helping the product team, bringing valuable insights into the user experience. This makes sense as they perform the most testing and plan which tests could be run. They also know whether something else in the product can be affected by this feature (together with developers).

What should PMs know about getting started with testing?

If you’ve never done testing and you’re a first-time PM, sit with your developers and QA, understand what type of test coverage we already have, what is missing, and whether or not we have a status page where one can see where tests are failing. This is done to understand what is there, as PMs need to understand what’s happening. It’s also important to educate yourself on different testing types (unit tests, automated tests, etc.). As PMs, we don’t need in-depth knowledge, but it’s preferable to know what developers and QA are referring to in discussions.  

PMs should write acceptance criteria as a part of feature requirements which should also be reviewed by QA (if you have one, otherwise ask developers for a sanity check), and probably a testing plan if the feature is complex.

I also prefer starting by doing, meaning that depending on how many people can test, I’ll try to at least test on iOS and Android phones (I primarily work with mobile apps). I’d go through different scenarios, try to “break” them by inserting some weird input in the input fields (e.g., adding emojis into the amount input field).

As a PM, I don’t see the need to test all scenarios, platforms, or phones. I’m not trying to be a security engineer that usually tries to break things in terms of breaking the code, or making sure that the code is impenetrable or any other efforts to alter the code — it’s not my area of expertise. I see PM’s role in terms of testing is to do manual testing to ensure that all features work as expected.

“I see PMs role in terms of testing is to do manual testing to make sure that all features work as expected”

How can PMs organize QA and testing processes within a company, big or small?

If there is a QA lead/person (depending on the company's size), they’re usually responsible for organizing this process. Of course, PMs are one of the stakeholders of this process, so they're usually involved in helping to shape it. This could include the process of how feature development starts until it goes live into production, criteria for going live, what happens if a feature does not pass through QA, and more.

If a company is just starting with this process, it’s better to start lean and build upon it based on adapting. Be agile. Each team is unique, so find what process should work for the team. Every company I have worked at always had different processes — and as it should be, always tailored to the team, the company, and its current needs.

How would you prioritize testing in the whole process?

Testing should be one of the steps in the workflow — there is no need to prioritize something that has to be there in the first place. At one point in my career, I’ve experienced the consequences of deprioritizing writing tests, testing, and having a dedicated/proper QA Engineering team (not manual testers, but people who have an engineering background, who can create and write tests).

There are usually three dimensions: doing things fast, cheap, or high quality. Usually, you can only have two of these. If you do things fast and don't want to sacrifice quality, you’re going to pay lots of money. Usually, and unfortunately, QA and testing coverage are the ones that are sacrificed, and some companies don’t realize until the consequences hit them. And once that happens, these companies would then have to dedicate several months of everyone’s work and time — not only QA but also PMs and developers to improve code quality, QA coverage, writing all tests, and fixing all bugs.

“There are usually 3 dimensions: doing things fast, cheap or high quality. Usually you can only have two of these. If you do things fast and you don't want to sacrifice quality, you’re going to pay lots of money.”

You don't win anything when you think that you're running fast. You actually lose more by changing existing products or testing coverage on stuff that has already been shipped. There is also a risk of no longer having people who may have been involved in the initial process around once the company has to revisit and fix these issues. This causes a lot of loss in terms of time, frustration and money. In the beginning, you might think that you're saving months, but you could be paying it threefold later on.

One of the roles of PMs is to make sure that developers have time to write code and that QA and developers have time for testing and increasing test coverage. This should be incorporated in the strategy.

There’s often the question: “When you have a bug in production, who’s at fault?”. Can you share some thoughts on the topic of ownership and bugs?

Who’s to blame? Either no one or everyone. There is no blame game — if there is a bug in production, the first step is to fix the bug. Of course, understanding the root cause is essential, but not in a "who wrote this code" manner.

Mistakes are just part of the journey. As long as a person or team can learn from their mistakes, it’s a good way for everyone to grow in experience and knowledge. In general, a Scrum team means that we are all responsible for successes and failures because we share our ups and downs. Usually, in terms of a whole company, Product represents the team and they’re responsible for eliminating bugs, fixing them in terms of “the face” of the team within the company. So they’re usually the ones to be blamed - but I don’t take it personally, just part of being a Product manager :)

About our guest

Maria is Senior Product Owner at a fintech startup in Berlin currently working in stealth mode. Previous to this, she was Senior Product Manager at finleap connect and at Nuri, Germany's second-largest challenger bank. Connect with Maria on LinkedIn.

‍