Published: September 03, 2023 • 2 min read
“How did this feature break? I thought we were writing tests.” –Frustrated Boss
We spent the time writing tests, and yet, a bug survived. Should we just stop writing tests?
No, but we should maybe write better tests, and think about them differently.
In this post, I’d like to explain a mental model I have that helps me write and maintain tests: thinking of bugs, and the tests that cover them, in classes.
Every test has limitations. Unit tests miss the big picture, end-to-end (E2E) tests miss the details. Rather than chasing the impossible, that the test suite catches all bugs, I’ve found it’s preferable to think about bugs and tests in classes, like the classifications of the animal kingdom.
Here are a few common classes of bugs:
undefined, “You have NaN new messages!”
cancelMeetingcalled instead of
These are classes of bugs, coverable only by the right classes of tests. Some examples:
Linting, formatting, static analysis, and type checking are great at catching dumb mistakes. Unit tests catch logical mistakes. Component tests catch integration mistakes, such as the wrong function being called when a button is clicked. And E2E tests catch, at a high cost, the fault lines between systems.
And none of these are good at catching anything else. Linting won’t catch a broken production API. E2E tests will pass when a string constructing function is replaced with hard-coded copy. You need the right class of test to catch the right the class of bug.
Thinking in classes has a few benefits and applications.
First, it’s empowering! We replace “everything is broken” with “our date function doesn’t handle null.” The latter prescribes its remedy– write a test where the input is null. Thinking of bugs in classes makes a big problem (software has bugs) smaller (untested inputs produce unpredictable behavior).
Even with these classes of tests, bugs still slip through. We tested what we thought mattered, missed things, learned, and added more cases. Thinking in classes gives you a set of tools to extend those cases with confidence.
What are your thoughts on classifying bugs? Let me know!
Join 100+ engineers who subscribe for advice, commentary, and technical deep-dives into the world of software.