ADF EU Pipelines
Background
According to the company wide EU commit, all EU data with EUPI will be processed and stored in Blueshift. For search pipelines, we have set up a set of EU pipelines in MSAI-DataPipeline, the same ADF resource where hosts Cosmos pipelines.
Please check ADFSourceControl for general ADF accessing and updating process.
Linked Services & Global Parameters for Blueshift
| Linked Service Name | Account | Description |
|---|---|---|
| substrate_adla_neu_prod | substrateadlaneuprod | ADLA account for Substrate (including MSAI) computation in EU Blueshift |
| substrate_adls_g2_neu_prod | substrateadlsg2neuprod | ADLS G2 account for Substrate (including MSAI) production data and files |
| substrate_adls_g2_neu_rawlog | substrateadlsg2neurawlog | ADLS G2 account for Substrate (including MSAI) to upload ODL raw log |
| Global Parameter Name | Value | Description |
|---|---|---|
| BlueshiftProdRoot | abfs://filesystem@substrateadlsg2neuprod.dfs.core.windows.net/local | Root folder of production Blueshift storage account |
Update EU Pipelines in ADF
For each search Cosmos pipeline in ADF, there is a corresponding Blueshift pipeline with prefix "EU_" in subfolder "EU Blueshift", which only cook EU data in Blueshift.For example, EU_CookingSearchImpressionHourly is the EU version of CookingSearchImpressionHourly:
The Blueshift pipeline shares the same script/binaries (which will be deployed to Blueshift along with Cosmos during deployment) with the corresponding Cosmos pipeline, so once there is a script change (ex. adding a new parameter) that need to update ADF Cosmos pipeline, please update the EU pipeline as well (example PR).
Generally, the parameter values in EU pipelines are extracted from json config files as well. Please note that the parameter values in json/dragonfly files are mainly for Cosmos pipelines, but some parameter values are different in Blueshift, such as input data path (should use absolute path in Blueshift rather than relative path). Please be careful when update parameter values in EU pipelines, and refer to the script code when you are not sure about the values.