Continuous Integration (CI) is a key part of agile development engineering practice. It drives the success of implementation of almost any agile team. How to implement a successful CI? The core of CI could be summarized to the following:
Run the right test at the right time, in a right environment effectively.
So I am translating the above principle into the following topics to describe some of the best practices of implementation of CI:
- Continuous Integration Success Factors – Right Tests at Right Time
Address what are the right bucks of tests for different stage of code promotion process.
- Continuous Integration Success Factors – Effectiveness
Address how to improve the efficiency of test coverage throu
- Continuous Integration Success Factor – Right Environment
Address what is a seamless integration environment
Address what are the critical factors to determine whether your integration environment is ready for seamless integration.
- integration Quality Measurement
Address how to measure the integration quality for a CI implementation.
- CI Implementation Check-list
Some practice check-list for a QA/DevOps team when implementing a CI practice.
So for this blog, I will address the first key CI success factors – right tests at right time.
The core benefit of CI is to find the defects of software product as early as possible. I am sure everyone is familiar with (and mostly agree with) the following bug fix cost conceptual model:
So if we run the right tests at the right time of development life cycle, it will essentially help to find the bug or defect as early as possible without jeopardizing the speed software development and delivery. The following table is a summary of test buck design:
A couple of highlights:
1. QA Test bucks
Except for developer’s unit test, QA tests could fall into 3 bucks:
- BVT(build verification test) or acceptance test. This buck of test mainly covers the most critical path to verify the build. Its main function is to make sure every dev. code check in does not break the major piece of software functions. So it should have two attributes 1) Cover the most critical path, obviously 2) should be very selective so that its execution time should be short enough to validate each dev check in. Normally it is best to take less than 5-10 min.
- Regression test. This is fully function tests for the given software products. As it normally take very long and also it could include low priority tests, it won’t benefit much if it runs more frequently. Daily run is probably the most common case (or a fixed numbers of run every day). It provide a daily verification for all software features.
- Full Integration tests: Depending on the complexity of software integration environment, this buck of tests could be part of section 2 if integration is relative straight forward. But for some software products who may have very complex down-stream or up-stream dependency, this buck of test will focus on the system and functional integration of given software product with its down or up stream components.
Developer unit test owner is probably most straight forward. I will only want to highlight owner for integration test. For a lot of team when integration is straight enough, as stated above in section 1, most of team merge integration test and regression test together. So is the team. But for some cases, it could make sense to have a separate Integration testing team to handle integration test cases only:
- Integration environment are very complex, and have a lot of down/up stream dependency. Integration analysis is a dedicated job basically.
- It may require specific domain knowledge on integration scenarios.
- Or sometime it could involve specific technology for system integration. For example part of integration tests is to write a blueprint application for integration scenarios, this could require development and architecture skills that are beyond regular QA.
- Team constantly struggles with priority of functional regression tests and integration quality.
I am going to skip this section as it will be addressed in “Continuous Integration Success Factor – right environment”.
4. Failure triage
Continuous integration does not come without a price. The cost for test execution will be very low if it is automated. But failure triage could be a very high cost if test run very frequently. So the balance is obviously triage high priority test more frequently than low priority tests. That is why most of team need to instant attention for BVT failures while keeping the daily run failure triage less frequently (between daily or weekly). This principle indicates a good practice – always prioritize your tests.