Define: Functional, Unit and Integration Tests
I have recently read a blog post claiming that functional tests are not “true” tests. The author also claims that unit testing shows you where the problem is occurring, while functional testing simply identifies that a problem exists. This argument is deceptive and the conclusion dangerous. Different kinds of tests are not mutually exclusive. One is not superior to the other. They have different goals and can happily coexist. Let me explain the kinds of tests so that you could make enlightened decisions.
Functional Tests
Think of it as testing your application in the browser, but in an automated fashion. There are many ways to automate. Some tools can go as far as launch a specific browser and programmatically simulate user input, other tools can take a step back and send appropriate HTTP requests while inspecting the HTTP response.
In order to perform these, you do not need to know how the software is implemented. They are very high level tests. All you need to know is what the software is expected to do. For example, when you enter an invalid phone number and submit the form, the form should reload with an error message on top of the page. If the form saves at this point, you know that you have a problem, possibly in your validation or the validation may have been skipped. You know the general location of the problem, now you just have to dive into the code.
Functional tests will always prove useful and can do things that others cannot. For example, I use functional testing to simulate a sequence of user actions. Here is a simple scenario, written in natural language for your benefit:
- Authenticate a user that has only reading permissions.
- Navigate to an article’s draft.
- Assert that the status code is 404.
- Assert that the page contains a link to the most recently published article.
- Click the link and wait for the page to load.
- Assert that the status code is 200.
- Assert that the title of the article is correct.
There is almost no limit to what you can verify with functional tests. The more assertions you write, the easier it will be to pinpoint the location of a potential bug.
Unit Tests
Unit tests are low level tests. They are meant to test your code rather than the high-level features. You need to have an understanding of how the code works and how the functions are linked. For example, you could have a method called canOpenArticle which will take a user and an article argument. Based on the permissions of the user and the status of the article, the function would return true or false. You expect specific output based on the input. Examples:
- Call canOpenArticle with a reader and a draft article.
- The output should be false.
- Call
canOpenArticlewith an author and a draft article. - The output should be true.
User tests should operate on a single function or a small group of functions to be most effective. This will allow you to pinpoint the exact location of the issue. If you mix too many of your application functions in one test case, it will be much harder to tell which one of them is at fault when that test case fails. More on this in my next article: “The Missing Intro to Unit Tests”.
Integration Tests
There are many definitions to this floating around. To avoid overlap with other tests, I define this as the testing of self-contained modules that you expect to be working on their own. All you want to do is make sure that you use them correctly. For example, if you’re using the Google Maps API, you don’t expect it to have bugs. Yes, it’s possible that they have bugs, but for the sake of sanity and good practices, your responsibility should stop here. You want to make sure that you are using the API correctly, that’s all. When I write integration tests, I send parameters to the API and expect the output to be exactly as documented. If it doesn’t match, I first assume that I’m using it wrong. Did I forget a parameter? Did I send a date in the wrong format? Did I misspel something? Once all my integration tests pass, I expect the API to keep behaving. If the API changes without telling me, I’ll know pretty quickly with my tests and will fix my API calls accordingly.
Here are some more examples of integration tests: framework, plugins, libraries, payment gateway, etc. For some of them you may have access to the code, for others you can’t see the code. In any case, always treat these as black boxes and focus your integration tests on the communication between your software and these external components. Bonus: you can do something similar between your own modules for better decoupling, so that each module could be swapped as long as it uses the same communication protocol.
Now go and try all of these. Don’t favor one over the other, as they are all meant to work together like a seat belt and an airbag. You wouldn’t trust your safety to only one of these, right?
The title “User Tests” should be “Unit Tests”.
Thank you, fixed.
Good points Anna. Tests can serve two purposes: telling you if there is an error that will affect the application’s purpose, and telling you where the error is. Unit tests (according to the strict definition) will accomplish the second goal very well, but if all your unit tests pass and users still can’t log in then your tests have failed! Functional and integration tests can be very efficient for the first purpose because one simple test can allow you to detect the presence of many possible errors.
If you want to get effective testing without the costs of 100% coverage it makes sense to write functional tests first to detect the presence of errors. That can be enough for areas where changes and errors are very rare. Then you can add unit tests in areas where the errors are more common or more difficult to understand and fix. But we never write code with frequent errors right?
Having integration tests that incorporate external APIs can be very valuable to see easily if the API provider messed up. Sadly, most of them don’t have sufficient tests to avoid BC breaks. That does not only apply to web service APIs but also to some frameworks and libraries. However, having full integration test coverage for these things is hardly possible, too.
Finding the right degree for integration tests is a difficult task and the outcome highly varies from project to project. Writing integration tests only for self-contained modules is not a good rule of thumb, IMO.
I also like considering some sort system tests as good integration tests: Those that do not verify acceptance criteria as their main purpose. These can deal quite well as end-to-end integration tests.
From my experience, finding the optimal test mixture for different code areas and different perspectives of a project is one of the biggest challenges for people.
Garand
Although functional tests are defined as black-box testing, their output can still provide valuable information to a developer. They will help locate the error more efficiently than mere manual testing.
I agree that testing is not a race to 100% coverage. It is a desirable end result, but should not be the first objective. When testing existing code, I use pretty much the approach that you describe. I never settle for only one kind of tests, because they have different roles and are all necessary. They reinforce each other.
In a perfect world, all APIs are flawless. I have seen APIs completely change their protocol without warning. I stop working with APIs that expect users to just keep up. It’s a good thing that many use a different endpoint for each version.
Regarding frameworks and libraries, I tend to stick with the version I originally installed, because then no BC break can occur. I have had bad experiences in the past and tend to be a bit pessimistic.
The right mixture comes with experience, and that can be shared through case studies. Please do so on your blogs when you have a chance.
In regard of integration tests, we have to pay attention to the fact that for some external API (I’ve experienced that with a payment gateway), the service returns a random response in the sandbox environment, just for the sake of verifying if the URL is called correctly, with the correct parameters. This fact can lead to false negative and unexpected failing tests. I think that integration tests should run in a different test suite, and only after the unit tests are done. I also think that they should run in a smaller amount than units and that contínuous integration should not depend upon them.
Thanks for your article.
I just wanted to add that all the tests have their costs. Somtimes it’s natural to avoid unit testing (when an application is built on top of stable components), and use only functional or acceptance testing. Sometimes, otherwise, better to test with units. I.e when it’s hard to perform asserts in asynchronous calls to APIs. And sometimes manual testing is just enough. So the wise project leader should choose strategy according to project bottlenecks.
Well, I’d also like to share a testing framework I developed to simplify acceptance/functional/unit testing. http://codeception.com For any kind of tests can be executed in one engine.