DiffDoctor Validation Pipeline for DM.Auriga.Impression
See the detail design in Integrate DiffDoctor to test enviroment proposal.
Background
DiffDoctor validation is a critical development step for the data pipeline changes quality in DM.Auriga.Impression repository. As the complex manual steps in the legacy DiffDoctor validation process, we provide a more automatic validation process in Azure CD pipeline.
Overview
The new validation pipeline depends on the artifacts from DM.Auriga.Impression - Artifacts Publish and Validation. Devs can trigger the validation process in CI/CD pipelines base on the specific Build Pipeline Artifact. Validation pipeline will output the compare result between Baseline and Testline for the final validation.
- The
baselineis the master branch when you create the build pipeline. - The
testlineis the private branch which only contains your changes.
Architecture Design
Trigger the test environment automatically : Prepare the test environment (Baseline and Testline artifacts base on specific BuildVersion) on Cosmos.
Inject the values of parameteers in the indicator : Collect the latest parameters from the release pipeline and inject the parameters into a Indicator file.
Detail Design
Continuous Deployment Validation Pipeline
DiffDoctor Validation Pipeline - msai-datapipeline-testenv includes DiffDoctor Environment Validation and Pipeline Validation stages.
DiffDoctor Environment Validation
DiffDoctor Environment Validation is the dependency stage for other validation pipelines. It will confirm if the current buildversion artifact has been uploaded to the Cosmos completed. Detail see here
Validation Pipelines
Validation Pipelines stages are used for the different Data Cooking Pipeline validation. Currently, it has supported these following pipelines.
| Validation Pipeline Name | Used For Validaion Pipeline Name | Default Trigger Date |
|---|---|---|
| Logical Hourly Validation | CookingLogicalSearchImpressionHourly | @addhours(trigger().startTime,-30) |
| Logical Daily Validation | CookingLogicalSearchImpression | @adddays(trigger().startTime,-3) |
| SearchImpression Daily Validation | CookingSearchImpression | @adddays(trigger().startTime,-3) |
| SearchImpression Hourly Validation | CookingSearchImpressionHourly | @addhours(trigger().startTime,-30) |
| FastSession Validation | SearchSessionComputation_Fast | @adddays(trigger().startTime,-3) |
| CompleteFastSession Validation | SearchSessionComputation_Fast | @addhours(startofday(adddays(trigger().startTime,-3)),12) |
| SlowSession Validation | SearchSessionComputation_Slow | @adddays(trigger().startTime,-3) |
| Layer of code change | Entry points | Needs Validations |
|---|---|---|
| Logical Search Impression | teams.powerbar | Logical Daily Validation Lack of CompleteSlowSession Validation |
| Logical Search Impression | all except teams.powerbar | Logical Hourly Validation CompleteFastSession Validation |
| Search Impression | teams.powerbar | SearchImpression Daily Validation Lack of CompleteSlowSession Validation |
| Search Impression | all except teams.powerbar | Logical Hourly Validation CompleteFastSession Validation |
| Session | teams.powerbar | SlowSession Validation |
| Session | all except teams.powerbar | FastSession Validation |
Validation pipeline storage structure on Cosmos VC
These are the VC Paths used for the DiffDoctor Validation Pipeline. Binaries, TestOutput, DiffResult are distinguished by BuildNumber Version.
| Location VC Path | Usage for Valiation Pipeline |
|---|---|
| Binaries | Storage binaries with different build number |
| TestOutput | Storage test output with different build number |
| DiffResult | Storage validation result with different build number |
DiffDoctor Validation ADF Pipeline
See the AzureDataFactory pipeline detail.
CompleteFastSearchSession DiffDoctor Validation Pipeline
Here will introduct the Complete FastSearchSession DiffDoctor Validation Pipeline as it's a special validation pipeline compared with FastSession Validation.
The regular FastSession Validation is only used for the changes validation only in FastSearchSession pipeline.
However, Complete FastSearchSession DiffDoctor Validation Pipeline is a composite validation process including Impression and Session Pipelines. Both the baseline and testline will execute multiple hourly pipelines (Impression + LogicalImpression) as the upstream input for the final FastSearchSession pipeline seperately. This validation pipeline is used for the validation for the entire data cooking pipelines from Impression to Session. not only for input datasource but also dependencies.
Appendix
msai-datapipeline-testenv Pipeline
Check detail here.
DiffDoctor Environment Validation Stage
The DiffDoctor Environment Validation stage will be triggered automatically to check whether the Artifact has been uploaded to Cosmos completely. If the indicator exists, that proves the Artfact has been uploaded to Cosmos, otherwize the stage will upload the artifacts. The below shows the Artfiact of Baseline and Testline for Version 20220630.6 has been uploaded to Cosmos.
