1. To make sure we have appropriate integration quality, the first step is to “know your dependency or neighbors” – Dependency Analysis. This is probably the most important step as well to have appropriate integration coverage, in terms of timing and test case coverage. This in general needs very comprehensive and extensive analysis and understanding of underline software architecture and data flow of given software products. If it is not productive to do it yourself as a QA, it is always a good suggestion to work with your development team to do this to get accurate data. Here is a sample spreadsheet for reference to get this work started:
- Component type: The types of dependency products could be javascript library, jar library or other format of runtime library, restful web service API etc. It is important to understand the type of those dependency so that you can understand how your product is consuming its dependency and the risk area that could be impacted by dependency changes and the deployment requirement. For example, for jar library or other runtime library, as long as it is designed to consume the given version at any given release, you wont necessary worry about to integrate them in a daily-basis, that will make the integration risk very low.
- Dependency relationship: in general, we only care about the dependency we are consuming in the first level.
- Dependency versioning: most of runtime library are problem fixed version, meaning you can choose to update yourself. But javascript library, or some web service are mostly dynamic, meaning you automatic consume the latest version once they update it. CI makes most of sense for those conditions. So the dependency versioning will determine the integration risk and necessity of CI.
- Automatically deployment: we already introduce this in the integration readiness. The efficiency of CI depends on whether they have the automatic build and/or deployment script to consume this dependency in CI. Some of the team use virtual machine technology or even latest docker container to make this much easier for automatic deployment.
2. Once you have your dependency analysis and understand the state of your dependency, the next step is to understand in any given release, which dependency needs to be included in CI. The key element will determine that is feature change impact and release time line (compared with your own product release) of every dependency. With those data and analysis, you will determine which dependency you need to have the daily update of its stable build in your CI environment. For example, for any dependency who may release earlier than your next release, it is probably always a good idea to have their latest stable in your CI environment to make sure their changes won’t break your code.
3. Once you have your appropriate dependency in your CI, how to measure your regression test cases have covered all integration points? This is not a simple question to answer, a lot will depend on the domain knowledge of each QA on the integration points and data flow between its product and its dependency. But there is one useful tool that can provide a simplified measurement and at least help QA to identify some coverage omission – Code Coverage. For example, Cobertura, a code coverage open source tool for Java application. It provides line, function and branch coverage data by your test cases. It is extremely helpful to identify the omission of your test coverage. For example, when you know where your dependency latest feature code changes, it is very straight forward to find the missing coverage if it shows so in your code coverage report. Check Cobertura website for more details on the tool and code coverage concept.
With all the steps above, you identify what are the dependency, which critical dependency to pay attention in your CI, and how to detect the missing integration coverage, now it is your job as a QA to continue adding more test case coverage to mitigate integration risk. Simple enough.
