Are you skipping non-functional testing?

    Prashant Hegde

    Prashant Hegde specializes in building high-performing teams. He is an accomplished writer and an international speaker.

    Published on

    July 21, 2022
    Are you skipping non-functional testing?

    The first thing that comes to mind when we think about Software testing is Functional testing. Non-functional testing is just as vital as functional testing.  However, non-functional testing is many times an afterthought and occurs at the end of the SDLC. Due to tight schedules and deadlines, non-functional testing is often rushed or overlooked. Several testers refrain from performing non-functional testing because they feel it is outside their scope of testing.

    Software is considered reliable when it meets customers' expectations under various conditions. Non-functional testing plays a crucial role in meeting customer expectations. Non-functional testing overlooked not only leads to poor user experience but also can cause damage to the organization’s reputation.

    In this article, we will explore what non-functional testing means and how we can prevent missing out on testing the non-functional requirements.

    What is non-functional testing

    Testing the functionality of an application against a set of requirements is functional testing. Non-functional testing examines different factors that functional testing doesn’t cover. Non-functional testing involves evaluating non-functional quality attributes of software, such as Usability, Maintainability, Efficiency, Reliability, and Portability. Refer to the ISO standard (ISO/IEC 9126) to comprehend all the ‘-ilities’.  

    The purpose of functional tests is to determine if the software does what it's supposed to do, while non-functional tests measure how well it performs its intended tasks. Non-functional testing is a general term that consists of all testing types that focus on how well the software performs in the real world.

    Below are some of the noteworthy non-functional testing types -

    Diagram of noteworthy non-functional testing types covering reliability, usability, efficiency, maintanability, and p
    • Performance testing - How well does the application perform under a specific workload in terms of its responsiveness and stability.
    • Load testing - What happens when a large number of users access the application concurrently?
    • Reliability testing - How reliable is the application? How long can it run without failure under specified conditions?
    • Stress testing - What is the maximum amount of stress the application can handle? What happens beyond normal operational capacity?
    • Recovery testing - How well can an application recover from crashes or failures.
    • Security testing - How secure is the application? Is the software vulnerable to cyber-attacks?
    • Compatibility testing - How does the application function in different environments or OS?
    • Usability testing - How user-friendly is the application? How easily can a user accomplish a given task?
    • Accessibility testing - Does the design of software, devices, and services enable people with disabilities to use the application?
    • Compliance testing - Does the application complies with the requirements of a specific standard or regulation
    • Internationalization and Localization - Can the application adapt to different languages and regional differences without requiring significant engineering changes?
    • Documentation testing - How easy is it to understand the software documentation?
    • And more.

    Explore, discover, and define non-functional requirements

    All the different non-functional testing requirements showing documentation, internationalization, stress, accessibility testing, and so on.

    Customers communicate their requirements primarily through Functional requirements. The functional requirements define the tasks and feature users will interact with to achieve their goals using your software. Unlike functional requirements, non-functional requirements are often not defined in the requirement phase. There are certain qualities a system should have without having to specify them. In general, nonfunctional requirements are ambiguous and implicit. They can turn out to be tricky to define.

    • Just like your functional requirements, the nonfunctional requirements of your software needs should be defined and prioritized.
      • When gathering requirements, certain non-functional requirements can be identified by listening to customers and their feedback. Asking the right set of questions is the key to extracting non-functional requirements.
      • Some non-functional requirements might be hard to determine at the start of the requirement phase. They may evolve and get more apparent over time. Constantly evaluate and assess your product to discover non-functional requirements.
    • The non-functional requirements must be SMART (Specific, Measurable, Attainable, Realistic, Time-bound).
      • Non-functional requirements cannot be subjective. Customers may say things like, “The website should be fast." But you need measurable criteria to test. For instance, “Every page must load within 2 seconds”.
      • Understand the SLAs of your system/services that describe the non-functional requirements.
      • A Service Level Agreement (SLA) is a legal agreement between an organization and its customers. SLAs specify the quality standards to be adhered to by an organization's products, processes, and services. If there is a breach of the SLA, the organization has to give a remedy or a penalty as mentioned in the agreement. SLAs drive the design, architecture, and approach required to keep the systems reliable and the consumers satisfied.
        The SLAs are defined in a way that can be measured without ambiguity. Examples of SLA’s:
        • % of Uptime yearly for the Service should be 99.9% (acceptable downtime in a year is less than 8.76 hours) - Reliability
        • Not more than three major incidents per year - Reliability
        • The API response time should be lesser than 500ms for the service - Performance
        • The application must adhere to the Web Content Accessibility Guidelines WCAG 2.1- Accessibility
        • Database security must be compliant to HIPAA requirements- Security
      • As a tester, first, understand the SLA’s of your services/applications.
      • Understand how developers are measuring and monitoring the SLAs.
      • For new services, help developers to test if we are meeting the defined SLAs.
      • For existing services, conduct the required non-functional tests regularly.
      • Shift left and perform non-functional testing as early as possible. The late discovery of non-functional testing can turn out to be expensive.

    You don't have to be an expert to get started

    Let's be honest; not everyone has expertise in non-functional testing. Do not let inexperience hold you back. Because you do not feel like an expert, don’t be hesitant to start. Testers who succeed are the ones who constantly learn and acquire new skills. Learning new things has become easier than ever before.​​ Numerous online courses and resources are available to help you learn the new skills needed to start non-functional testing.

    While it's tempting to jump in and try out different tools, it's best to understand the fundamental concepts first. Below are some of my recommendations to get started-

    Non-functional Testing Recommended Reading for Beginners Open Source tools
    Performance testing Performance Testing Guidance
    for Web Applications
    Gatling, Locust, K6, Jmeter
    Security testing OWASP TOP 10 OWASP Zed Attack Proxy
    Burpsite community edition
    Accessibility testing WCAG 2.1 Guidelines WAVE ,axe, Pa11y
    Usability testing
    Usability testing 101

    There are a variety of non-functional testing types. Each one of them is essential in its own right. Start small; it might be overwhelming to start testing all the non-functional attributes simultaneously.

    Additionally, the cost of performing each type of non-functional test is high. Thus, according to the risks and constraints, you should choose non-functional tests wisely according to the risks and constraints. Depending on your customers and their preferences, specific non-functional testing may be prioritized over others.

    The wrap up

    Non-functional testing is necessary to give your product a competitive advantage in today’s market. A system could meet all functional requirements but still not meet user expectations.  To provide the best customer experience, don't forget the non-functional “ilities” that matter. When Non-functional requirements are neglected, they can impact the customer experience, and your customers look for alternatives.

    Non-functional testing should be at the top of testers' priorities. Building a high-quality system that meets your functional and non-functional requirements is relatively easier if you plan for it beforehand. Learn about, comprehend, and ask questions about the non-functional requirements. Incorporate non-functional testing into your testing strategy, moreover, in the development process. Devote some time every day in the quest to learn non-functional testing. Don't dismiss non-functional testing anymore.

    Data-rich bug reports loved by everyone

    Get visual proof, steps to reproduce and technical logs with one click

    Make bug reporting 50% faster and 100% less painful

    Rating LogosStars
    Category leader

    Liked the article? Spread the word

    Put your knowledge to practice

    Try Bird on your next bug - you’ll love it

    “Game changer”

    Julie, Head of QA


    Overall rating: 4.7/5

    Try Bird later, from your desktop