MSAI Data Platform Release Pipeline Based Deployment

Last updated by Zhao Liu on 5/19/2023 9:07:46 AM

Background

For some historical reason, we had to fork a version to setup our deployment instance from the complex Phoenix of ODIN team, but we often encountered some big pain-points:

  • Phoenix is a very complex and large system, it's very hard to maintain and keep it in the healthy mode, and it always cost a lot for us to fix access or permission issue.
  • We encountered a deployment blocking issue due to the commit gaps on Phoenix: Incident 260243366 : MSAI Unified Data Platform Support: The code change in DM.Auriga.Impression repo can't be deployed due to Auriga Phoenix issue.
  • Phoenix is not a punctual system due to scheduling mechanism, DRI has to check the deployment status from time to time.
  • Phoenix UI is not very friendly and hard for the deployment history and version management, and very cost consuming even for any small improvement.
  • Lack of supervision due to no approval process, since anyone can do the deployment without any approval.
  • EUDB will be a big challenge for us to ensure this Phoenix instance extend its capability to deploy to Blueshift environment.
  • We just forked a branch from its master branch on June 19,2020, there are some big gaps between our branch and master branch, and as of now, there are 80+ commits checked in master branch.

Now We have a new ADF deployment to replace ODIN Phoenix. The new ADF deployment is based on a DevOps release pipeline. DRI can finish the deployment within 5 minutes by one click. The architecture of the new ADF deployment pipeline as blow:

ADF Deployment Architecture

Features:

  • A PPE environment for any commit into master branch to be auto deployed, to ensure each commit is automatically validated in PPE.
  • A Prod environment for DRI to manually pick up a healthy commit from PPE and deploy it with the approver signoff, to ensure all the deployment into prod are fully supervised.
  • Friendly deployment process, intuitive management of build version, deployment owner, history, health status, re-deployment.
  • The same release can be deployed to EU Blueshift Prod environment as well.

Permission Control

  • Who can edit the release: Contributor role to the project, such as [TEAM FOUNDATION]\MSAI Data Platform.
  • Who can do the deployment: Contributor role to the project.
  • Who can approve the deployment:Allowed List, such as MSAI Data Platform DRI.

Deployment Steps:

  1. Check ADF ppe job. Check report from Deployment Notices , if there is success, you can get BuildVersion. e.g. 20230518.1, then you can go to Step 2 to deploy release on 20230518.1, usually it is release on yesterday.

ADFDeployment ppe job report

ADF ppe pipeline at link, it run at 1:00 AM CST every day. If you want to mannul run it with lastest release, you can trigger it with parameter "Date" = today - 2 days. e.g. 2023-05-16T00:00:00Z

  1. Navigate to Azure DevOps -> Pipelines -> Releases -> msai-datapipeline at link

ADFDeployment Release Pipeline

You can see a list of release with differrent build version under "Releases" tab. Find a version you want to deploy or redeploy.

  1. Go into the release and click deploy on PROD stage.

ADF Deployment Deploy

There is a Pre-deployment approvals and there will be an email to the approvers, DRI can approve it himself/herself.

ADF Deployment Approve

  1. Check the deployment result.

You can view the deployment logs under "View logs" tab. The release pipeline use a "Azure PowerShell script" task to upload artifacts to Cosmos/Blueshift directly via ADLS.

ADF Deployment Logs

And you can see the deployment commit detail under "Commits" tab, use this information for "Deployment Notices".

ADF Deployment Logs

A deploy history show under "Deployments" tab.

ADF Deployment Deploy History

Check files update status at Cosmos and Blueshift.

ADF Deployment Cosmos Files

ADF Deployment Blueshift Files

A demo video show the deployment process:

ADF Deployment Demo Video

Relative Resources