I have worked with both external an in-process component tests for ASP.NET APIs and they each have their advantages. This post points outs some differences to consider when making the decision.
Since you found this page, you will likely know the difference between External an In-Process tests. But to make sure we are talking about the same thing, here is my definition:
External tests
The service is deployed as for production, or started on the development machine. Some completely separate tests request information from the service.
In-Process Tests (a.k.a. In-Memory Tests)
The service does not need to be deployed, see image below.
- The tests start the service (xUnit for Asp.net APIs), in the same process. They adjust the configuration/DI as necessary.
- Tests start mocks for downstream APIs. All HTTP traffic will work as in ‘real life’, with a service listening to a port etc. The difference is that the mock service runs within the same process. Check out my post WireMock.Net for better Integration Tests to see how this works.
- The tests use DI to retrieve classes from the service as required. They can check data or manipulate objects in the API.
Summary / TLDR
In-Process tests (a.k.a In-Memory tests) are great when:
- You need the tests asap and there is no plan/budget for test automation.
Developers: This is almost as easy as writing unit tests, but can save you very embarrassing moments! - Service internals need to be tested – in that case developers are usually involved in testing
- Is data in the store/cache encrypted?
- Metrics, console logs
- Some dependencies are hard to mock externally. As an alternative, they can be mocked similar to unit tests.
- Different service configurations need to be tested
- The QA team is competent with C# and xUnit or willing to learn
Note: It is always an option to have both types of test.
Detailed Comparison
| Description | External Tests | In-Process Tests |
|---|---|---|
| Framework / Language | Any language or framework can be used. Most likely whatever the testing team prefers. I really like Mocha/Typescript for functional tests. | Using C# and the ASP.NET TestServer, the tests are generally implemented as xUnit tests. See the Microsoft documentation for details. |
| Setup in Pipelines | Can be ‘expensive’. An additional deployment is required, but with external dependencies mocked. It may also be necessary to deploy and configure mountebank or another mock service. | Really fast and easy. No deployment is required, they run similar to unit tests. The service is started in memory from within the tests, in the same process. |
| Running Speed | Fast. The service is deployed and started, so any response is usually really fast, except when downstream mocks have been configured with a delay. A sample could be that a mock downstream REST API simulates a timeout. | Can be slow. By default the in memory service is started for each test suite so that all tests start with a ‘clean slate’. This takes a bit of time. The new start is great for testing different configurations, and makes it easier to write tests without thinking about test data. However, to keep the test suite fast it may be a better option to ‘share’ an instance between many tests and implement the tests tolerant to existing data. |
| Test Data Management | Through existing endpoints or additional test helper endpoints. | As for external tests, but assuming an in-memory store is used, it is possible to retrieve this store from within the tests and manipulate/check this directly. |
| Testing Service Internals | Only possible by adding specific test helper endpoints. | Anything is possible. Output can be redirected, non REST dependencies can be mocked, any class can be accessed/checked directly by the tests. With great power comes great responsibility! The border between unit and component tests can become blurred. It is important to make sure the tests are valid and understood by the whole team. |
| Test Limits | Different service configurations require different deploys or changing configuration on the fly. This adds complexity to the tests. Anything internal. External dependencies without an in-memory implementation. This may not be a problem though for a lot of test cases – i.e. if metrics are not configured everything will still work. Likely logs/metrics | Deploy Configuration in pipelines Traffic across the network / DNS setup, … SSL termination, … |
| Problem Analysis | Logs, debug the service locally. Ideally it should be possible to find the problem by just reading the logs. | Locally it is so easy to debug that it is easy to forget to check the logs, which are the main error analysis tool in production |
Final Words
The decision between External vs. In-Process Component Tests is hard. I personally like Typescript Mocha tests for several reasons:
- They allow me to stay up to date with TS, even though I have mostly been working with C#
- The test names and test case generation is very nice and more flexible than with xUnit
- They are easier to sell to more people. Less abstract.
However, in real life there is barely ever enough time for pipeline setup. For this reason I am more likely to write In-Process tests with the Microsoft integration test framework, since they can be run in the CI pipelines so easily. The best test automation is worth much less if it does not run on every pull-request.
For information on component tests, improving mocks and best practices check my post Component Tests for Distributed Systems.
