Case: [CosmosJobSubmitSLA] Monthly_NGP_Export_Cortana_AppUsage_Packing

Last updated by Yana Wang on 3/14/2022 7:12:55 AM This content is over 788 days old.

Cooking logic in ADF job name:

Legacy Cortana Impression Daily

-> NGP Export Cortana AppUsage Cache Production Daily

-> NGP Export Cortana AppUsage Packing Monthly

Notes:

NGP Export Cortana AppUsage Daily job is based on NGP request, to show user app usage in privacy dashboard.

Steps

  1. Open [ADF monitor](msai-datapipeline - Azure Data Factory), check the job "NGP Export Cortana AppUsage Packing Monthly" error:

    missing Daily_Export_Cortana_AppUsage_Consumer_2021_11_03.ss in

COSMOS cosmos/exchange.storage.prod/local/projects/Cortana/NGP/Export/Cache/V1/2021/11/ (osdinfra.net) pic

  1. Check "NGP Export Cortana AppUsage Cache Production Daily" job. (Missing Daily_Export_Cortana_AppUsage_Consumer_2021_11_03.ss" is the output of "NGP Export Cortana AppUsage Cache Production Daily" job.)

  2. If "NGP Export Cortana AppUsage Cache Production Daily" job fails for "Directory adl://exchange-storage-prod-c14.azuredatalakestore.net/local/projects/CortanaData/Impression/2021/11/03/Output/ does not exist", then it means https://aad.cosmos14.osdinfra.net/cosmos/exchange.storage.prod//local/projects/CortanaData/Impression/2021/11/03/ has no Output directory. Then go to check "Legacy Cortana Impression Daily" job. (This data is the output of "Legacy Cortana Impression Daily" job.)

  3. Rerun "Legacy Cortana Impression Daily" job. If this job failed, go to step 7.

  4. After "Legacy Cortana Impression Daily" job complete, rerun "NGP Export Cortana AppUsage Cache Production Daily" job.

  5. After "NGP Export Cortana AppUsage Cache Production Daily" job complete, rerun ADF job name: NGP Export Cortana AppUsage Packing Monthly job.

  6. If "Legacy Cortana Impression Daily job failed with missing upstream file from bus.prod, then we should manually submit a scope job (similar with ADF NGP Export Cortana AppUsage Packing Monthly job) to cosmos using Visual Studio. The script should exclude the missing file. The job name should be the same in ADF.

    //Script GUID:4d4b6ab1-635d-4a6f-b8bf-a3dc5d937909
//Used for tracking history

#DECLARE ExportDailyCachePath_Consumer1 string = "/local/projects/Cortana/NGP/Export/Cache/V1/%Y/%m/Daily_Export_Cortana_AppUsage_Consumer_%Y_%m_%d.ss?date=2021-11-01...2021-11-02";
#DECLARE ExportDailyCachePath_Consumer2 string = "/local/projects/Cortana/NGP/Export/Cache/V1/%Y/%m/Daily_Export_Cortana_AppUsage_Consumer_%Y_%m_%d.ss?date=2021-11-04...2021-11-30";
#DECLARE ExportMonthlyCachePath_Consumer string = "/local/projects/Cortana/NGP/Export/Cache/V1/%Y/%m/Monthly_Agg_Export_Cortana_AppUsage_Consumer_%Y_%m_%d.ss?date=2021-11-01";
#DECLARE EXPIRYDays string = "540";

Raw =
    SELECT * FROM (SSTREAM @ExportDailyCachePath_Consumer1)
    UNION ALL
    SELECT * FROM (SSTREAM @ExportDailyCachePath_Consumer2);

OUTPUT
TO SSTREAM @ExportMonthlyCachePath_Consumer
CLUSTERED BY UserId, Timestamp
WITH STREAMEXPIRY @EXPIRYDays;

    pic