What we've learnt from 3000 bug reports

    Dan Makarov

    Dan Makarov is a co-founder at Bird Eats Bug.

    Published on

    March 3, 2020
    What we've learnt from 3000 bug reports

    A study of bug reporting patterns

    When do people report bugs? You probably never asked yourself that question, which was one of the reasons we wanted to answer it. The second reason was that we had the right type of data available, since our tool helps teams report and fix bugs. Long story short, we decided to learn more about the way software development teams approach the bug reporting process. For that exercise we've taken a random sample of 3000 bug recordings uploaded to our database in the last 2 months. Here's what we found.

    2pm is the golden hour for reporting bugs

    Graph of bugs reported by hour of the day

    Turns out that 2pm is the golden hour for reporting bugs. It is interesting to see that activity is growing in the morning up to 9am, and then again towards the afternoon, peaking at 2pm. There is also a local spike at 5. It's hard to know for sure why that is. A possible explanation is that some people do a round of testing before the end of their work day. That brings us to the next point - working hours.

    Is there even such a thing as a standard working day in software development circles? Turns out - not really. We compared how many recordings are uploaded within the regular 9-to-5 as opposed to other times.

    Pie chart depicting that bugs are reported 49% during working hours and 51% outside of working hours

    The split is almost equal, which was at first a bit surprising to us. When you look at the previous graph (bugs reported by hour), there seems to be more activity during the normal working day. You therefore don't expect an almost perfect 50/50 distribution. The explanation is quite straightforward though - 9-to-5 is only 1/3 of the day. So the remaining period is twice as long in terms of hours, but the rate of reporting bugs is two times lower. That calls for a question: are the individuals in product teams more flexible these days? Or are we just working longer and should reconsider our habits?

    A quarter of bug reports are filed on Tuesdays

    We expected people to file fewer bug reports on Fridays and weekends, than the rest of the week. This assumption proved to be correct. What we didn't expect is that the first 2 days would see higher activity. Our assumption was that there would be little to no difference between Monday, Tuesday, Wednesday and Thursday.

    Graph showing the difference in number of bug reported by day of week

    2/3 of bugs are reported on pre-production environments

    According to a 2009 study by Capers Jones, 85% of bugs are caught before the code gets to production. Is that the case today?

    Pie chart depicting 2/3 of bugs are reported on pre-production environments (local and staging)

    In our particular sample about 2/3 of bugs are reported on pre-production environments (local and staging). That number is lower than the one in the original research by Capers Jones.

    One of the plausible reasons can be a different user group. While Capers Jones seems to have focused primarily on software development teams, our product is used not only by Engineers, QA, PMs and Designers. We see an increasing amount of activity from customer support teams, for example. That means 2 things. First of all, business and operations roles in general don't use staging, so they can only report bugs on production. Secondly, certain things they might report as bugs, aren't actually bugs. Even a person, who knows the product inside-out, can't immediately tell if something's indeed a bug every single time. Some behaviours can be caused by hard-to-find settings, ad blockers or connection issues.

    We can't say for sure if this assumption correctly explains the 20% difference in the number of bugs reported on pre-production environments. Many factors are at play: company size, development methodology (agile/waterfall), quality assurance process, complexity of business logic and even how a particular company defines as a bug.

    "Bug" is the most common word in bug reports

    For that exercise we used the titles of bug reports to create a word cloud and see the most common themes.

    The result turned out to be rather predictable. Words like Bug, Error, Page, Test were the most common. To be frank, we did not expect "Collection" and "Date" to be that frequent - could be due to the sample, though.

    Image depicting that Words like Bug, Error, Page, Test were the most common in bug reports

    As the next step we removed the 7 highest ranking words to understand the rest of the group better. Now it's more apparent that adjectives and verbs are also on the list. Despite that, majority of the words are nouns.

    Image depicting the seven highest ranking words found in over 3000 bug reports we analyzed

    What we found the most interesting, and in a very positive way, is what words were NOT on top of the list. From personal experience we know that sometimes people describe bugs in very general terms, i.e. "Nothing works", "X is broken". To help companies avoid that, we have even created a bug report template to help especially non-technical people communicate bugs better. Yet, in our sample that behaviour was gladly uncommon.

    Most bug reports are made with macOS

    Graph showing the number of bugs that Bird Eats Bug users recorded in their own products, per operating system.
    Note: The graph shows the number of bugs that Bird Eats Bug users recorded in their own products. It does not in any way indicate bugs in the individual operating systems.

    Despite having only 17% of the general market, macOS is the most popular choice among bug reporters. More bug reports were made using this operating system, than Windows and Linux combined. Could it be because macOS makes you more productive?

    The majority of usage in the last 2 months came from the previous version of macOS (Mojave). Lately though we see more activity from Catalina, as you might expect.

    The data we have on Windows is a lot less granular. We can only say that almost all recordings were uploaded using Windows 10.

    Google keeps Chrome users on the latest software

    At the moment our product is officially supported only on Chrome, which is why we decided to exclude information about other browsers. We can however say that Chrome accounts for about 80% of our website traffic, followed by Firefox and Safari.

    The earliest bug reports in our dataset are 2 months old. At the same time, very few of them came from Chrome versions released over 4 months ago. It seems that Google is fairly good in making sure that users upgrade to newer versions.

    Graph showing the number of bugs that Bird users recorded in their own products using different Chrome versions
    Note: The graph shows the number of bugs that Bird Eats Bug users recorded in their own products. It does not in any way indicate bugs in various Chrome versions.

    As a company focused on bug reporting, we can only recommend this strategy as a best practice - it makes things much easier to debug. It also ensures a more homogeneous user base, which has positive implications not only on software development teams, but also technical support.

    Bug error. Test collection date issue.

    To conclude, we decided to take the data we had and create a bug report you are most likely to see. To do that, we combined the information from this article with the figures that didn't make it into the graphs above.

    And here it is - the most "average" bug report of this fall.

    • Title: Bug error. Test collection date issue.
    • Time: Tuesday, 2pm
    • Environment: Staging
    • Operating system: macOS 10.14.6
    • Device vendor: Apple
    • Browser: Chrome 77
    • Screen size: 1920x1080
    • Window size: 1920x798
    • Locale: en-US

    Data-rich bug reports loved by everyone

    Get visual proof, steps to reproduce and technical logs with one click

    Make bug reporting 50% faster and 100% less painful

    Rating LogosStars
    Category leader

    Liked the article? Spread the word

    Put your knowledge to practice

    Try Bird on your next bug - you’ll love it

    “Game changer”

    Julie, Head of QA


    Overall rating: 4.7/5

    Try Bird later, from your desktop