If you have ever found yourself tasked with designing a continuous integration system for one or more teams, then you have likely encountered the "Test Type Debate".

Everyone seems to have their own opinions about what constitutes unit tests, integration tests, functional tests, end-to-end tests, acceptance tests, performance tests, smoke tests, and more. In my experience, the most debated test type has been the integration test. Your experience may be different.

It happened to me

I was tasked with designing a continuous integration system for a monorepo at one point in my career. I met with folks around the organization and we discussed the stages and the types of tests that would run at each stage.

I started down the typical path by defining stages for unit tests, integration tests, e2e tests, etc. This ultimately led to hours of debate between the folks involved and lots of revisions of the plan on my part.

Until one day, I had an idea.

Network Constraints

It occurred to me that tests could be divided into three groups:

1. Tests that do not require a network connection.

2. Tests that require a local network connection.

3. Tests that require an external network connection.

Tests that do not require a network connection

These tests can run during pull requests. They are meant to be self-contained, repeatable, and reliable. Tests that fall into this category often end up being unit tests, but they don't have to be. It also seems to be the case that these tests are the fastest.

Tests that require a local network connection

These tests can run during pull requests. Maybe you're testing something with a Postgres database that has mock data, or maybe you're testing a service that needs to interact with another service on a local network. You might call these integration tests or you might not.

These tests are generally slower than the tests that do not require a network connection, especially because there is the upfront cost of standing up the infrastructure before the test can run. They are also fairly reliable, assuming your local network connection is reliable.

Tests that require an external network connection

These tests usually cannot run during pull requests. Tests like these include smoke tests, e2e tests, and tests that use a pre-defined environment. These are great tests to run after something has been merged via an immediate trigger or a cron.

Takeaway

When designing continuous integration systems, consider segmenting tests with network constraints rather than their "test types". A continuous integration system should not care about opinionated nomenclature. It should only be concerned with test reliability and completion times. Segmenting tests into different network constraints allows you to easily classify them by reliability and completion time, while also forgoing hours of debate.