Last updated by Na Li on 3/22/2024 8:49:36 AM

[[TOC]]

How To Test EuclidOIVIC Change On PPE Enviroment

This wiki page will guide you on how to test your changes in the PPE environment.

  1. Get the latest PPE version in Pipelines - Runs for EuclidOIVIC Official-Dogfood (yaml). Or you can create a new version with latest master branch via Run Pipeline, which may need 5 minutes to create corresponding release branch then you can go to step 2, not need to wait for it completed. getPPEversion

  2. Merge your branch into above latest PPE release branch​. And wait for the CI pipeline of your branch to be completed in Pipelines - Runs for EuclidOIVIC Official-Incremental (yaml), which may need 2 hours to complete. merge branch wait for CI completed

  3. After above CI pipeline completed, create a release by running EuclidOIVIC Official-Incremental with the above generated latest artifacts in Releases - Pipelines.

  4. Deploy above release pipeline, which needs a approve for deployment JIT. The deployment duration is about 30 minutes. deploy ppe release

  5. Apply the ADF debug permission with following cmd in Torus Client(PROD DMS - Datacenter Management Services), please keep monitor the status, if it's not auto-approved, ask team member help munally approve it(which may be a transient issue) - Request-AzureAdGroupRoleElevation -GroupName oivic-JIT-M365DataEngineerDebugger -Reason checkADF -Duration 6

  6. Go to PPE NAM ADF per the link in Asset List. Similar way to go other ADF. 6-1 And you check corresponding PPE version like - 6-4

  7. Trigger the job that you want to test in PPE NAM env. If you want to test BizChat/FVL/Oivic Curation job, you need to trigger sigs curation pipelines first(Backfill SignalType partitioned data pipeline can be used to backfill one day data with one trigger). When trigger BizChatCuration job, you can leverage batch rerun bizchat hourly pipeline to backfill one day data with one trigger, fill in JobDate and Category is enough. 7-0 7-1

    Note: You also can trigger a test job via rerun a previous succeed job, which will save the efforts of filling in parameters. 7-1-2 7-2 Note: if you need to validate there is no regression for your changes compared with baseline, you can update the used jar version to an older version and generate the baseline dataset for validation via debug run. update_jar_version

  8. Monitor the job is completed as expected.

  9. [Optional per needs] If you updated Heron Self-Serve SDK, please trigger a extraction test job via Heron PPE, which need to log-in with AAD account. Ask MSAI Data Platform team members help granting permission for you if you can't see MSAI Data Platform PPE project. And ensure corresponding self-server extract ADF pipeline completed as expected. 9-1 9-2 9-3 9-4

  10. [Optional per needs] If you want to have some statitics for new generated Dataset, you can leverage DataStatistics pipeline, input the DataSet(Copilot/OIVIC/FVL/OCLRawSIGS/RawSIGS) and InputPath(from above triggered test job) parameters per needs. After the pipeline completed, you can get the statitics from log via - 10-1 10-2 10-3