As in previous post, in CI, we have three environments needed for 3 stages: Dev environment for unit tests, QA integration testing environment (Dev/QA integration environment) and final full integration testing and prepared verification environment (staging). Unit test may or may not run in the same Dev/QA integration testing environment depending on their runtime requirement to the environment. It is controlled and dictated solely by Dev. I won’t cover that from QA perspective.
Dev/QA integration testing environment is where most CI happens. It is self-controlled by each product team, so it is easier to manipulate without broader impact to other team. Also since it is less complicate than a pre-prod environment that may contain much more strict requirement, it is much less expensive to maintain. This is also the environment where regression test suite runs most frequently. So it is a much better choice in terms of debugging and fixing integration issues. A well-executed Dev/QA integration testing environment should have the following characters:
- Include all your direct dependency components.
- Your dependency is regularly updated with latest stable version required for your release*
- Cover your 80-90% of integration points
* “Required”: Not all dependency need to get updated in any given release, it depends on the release schedule of given product and each of dependency (See integration readiness blog). “Stable”, general criteria will be nightly build run results, especially high priority test case results (again, see integration readiness blog).
The better we execute each Dev/QA integration testing environment, the better we have a clean staging environment.
The most argument or gray area is the purpose of staging environment. Compared with each team’s dev/QA integration testing environment, a Staging environment is much more complicate and expensive to maintain. It is also a shared environment, any issue brought to this environment will have much bigger impact. For example, any issue in staging environment could cause the delay of release of other products in the pipeline in terms of product promotion to next stage. Eventually any bug found in this stage is much more expensive. Thus, Staging should serve the following two purposes and these two purposes only:
- Verification before shipping to Prod.
- Integration testing that is too expensive to cover in each individual Dev/QA integration testing environment*.
* Again, this is where most gray area is. “Too expensive” depends on a conscious decisions of each Dev/QA team. For example, when you have automatic build and deployment from each of dependency products, the cost of deploy those dependency and test those integration in each team’s Dev/QA integration environment will be much cheaper, thus leave more integration testing into each nightly run instead leave to the staging (See more integration readiness blog).
Due to the lack of various readiness to some Dev/QA integration testing environment, one of the common instincts is to move the integration testing (or at least right coverage) into the staging environment. That causes the conflict of fast moving in the release pipeline for “verification” purpose because naturally some bugs will be found and debugging/fixing cycle will occur if they are well covered in each Dev/QA integration testing environment. In the end of the day, it will impact everybody one way or another. So for each QA, one of the hardest questions we have to ask ourselves is: do I have to test this feature or scenario or coverage in Staging, why cannot I cover it in our Dev/QA integration testing environment?
So the essential question for the right environment is sitting on how to make my own Dev/QA integration testing environment has the right elements for CI and most integration testing coverage. I will leave that to integrationreadiness blog.
