How to Test OIVIC Pipeline
Prerequisite
Please refer to here for OIVIC EWS required permissions.
Test OIVIC Pipeline Change in Dev Environment
For OIVIC code change or pure ADF pipeline change, you can test in Dev ADF workspace first.
If you make code changes in EuclidOIVIC (ex. PR), run "msbuild" to build the repo (reference) and get the updated jar package in your local machine:
Then upload the jar to scenarioartifacts - Microsoft Azure, you can rename the jar or use another path to avoid mixing with other version:
Generally you can use OIVIC MSIT NAM Coalesce POC pipeline to test, please create a new branch and update the jar path accordingly to your uploaded destination in OIVIC Curation Pipeline (just make sure to revert the path change after test, you can also clone a new pipeline and update there (ex. PR)):
If your change only stays in ADF pipeline, please ignore the steps above. After making your ADF changes, please temporarily delete the two activites below since they cannot run in dev environment:
Then you can click Debug button and trigger a debug run. Please change the parameter value of UpdatePayload to false to avoid copying SIGS data.
The test data (SIGS & ResponseActions) has already been uploaded to oivic - Microsoft Azure. You can also upload your own data there to a new Date folder and change the value of JobDate accordingly.
At last, you can check the debug run status in the Output tab below and check job output in oivic - Microsoft Azure after job completion.
If the spark activity fails, you can click the Output icon of the failed activity and find the YarnApplicationUri:
Then go to the link (need to login with admin account, please ask the passward from OLM team members), you can find the error message in Diagostics:
For more printed logs, please check the Logs page:
Note: The YarnApplicationUri is only accessible in Dev environment. For Prod job debugging, please refer to here.