FVL Pipeline on ADF

Last updated by Yana Wang on 3/14/2022 7:12:55 AM This content is over 788 days old.

ADF Portal

Name Subscription Id Data Factory ADF Portal link
ODIN DEV 9ce40ff0-cb61-4fd0-8a84-63a847f44520 adfwqjdcihnbcbrm DEV
OIVIC PPE NAM 4622f018-2e49-490f-b462-1b990f549058 adfb4b03ae1nam PPE
OIVIC PROD NAM 4622f018-2e49-490f-b462-1b990f549058 adf6db9e51dnam PROD

FVL ADF Pipeline

fvl-adf-pipeline

Known Issues and TSG

1. Data Copy

[UserErrorOffice365DataLoaderError] Dataset Serving ADF pipeline failed to extract data

Operation on target Copy SIGS MSIT to Local Storage failed: ErrorCode=UserErrorOffice365DataLoaderError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Office365 data loading failed to execute. office365LoadErrorType: PermanentError. Dataset Serving ADF pipeline failed to extract data.,Source=Microsoft.DataTransfer.ClientLibrary,'

Sometimes the data copy pipeline is not stable enough, please try to rerun. If it still failed with the same error, please contact Data Catalog team to verify their linked service stability.

2. Approval Service

[Validation Status Failed] Error calling the endpoint 'https://valiadation_status_failed/'

Operation on target Validation Status Failed failed: Error calling the endpoint 'https://valiadation_status_failed/'. Response status code: 'ClientSideException : NameResolutionFailure'. More details:Exception message: 'An error occurred while sending the request.'.
Exception message: 'The remote name could not be resolved: 'valiadation_status_failed''.

It means that your approval request is denied. Please check if your approval service request input is correct (same as the approval template).

3. Payload Registration

[Payload Registration Web Activity Failed] Failed with HttpStatusCode - 'Forbidden'

Operation on target PayloadRegistrationIterator failed: Activity failed because an inner activity failed; Inner activity name: If RunPayload, Error: Activity failed because an inner activity failed

Invoking Web Activity failed with HttpStatusCode - 'Forbidden', message - ''

It indicates that the payload registration is forbidden, please do the following check:

  1. Payload Registration Url
Environment Subscription Id
DEV https://substrateintelligencetest.microsoft.com/DataCatalogService/Datasets//DatasetPayload
PPE https://substrateintelligenceppe.microsoft.com/DataCatalogService/Datasets//DatasetPayload
PROD https://nam.substrateintelligence.microsoft.com/DataCatalogService/Datasets//DatasetPayload
  1. Get Access Token Body

Access Token is for authorization when you raise a payload registration request.

client_id=@{pipeline().globalParameters.WorkspaceSpnId}&client_secret=@{activity('GetSecretFromKeyVault').output.value}&scope=@{variables('Scope')}&grant_type=@{variables('GrantType')}

a. Check if the WorkspaceSpnId is the same as the workspace id in the workspace json file like oivic_prod_v1.json. b. Check if the WorkspaceSpnId(App Id) has been granted API permission on Azure Portal.

app-api-permission-granted

  1. Payload Registration Request Body

The payload registration request body is a json which is the value of PayloadLookup output.

payloadlookup-value

Manual Payload Registration

Please refer to this OneNote Page How to use DC Payload Registration Tool .

After you raised a Lockbox Request, contact self serve team (self-serve@service.microsoft.com) for an approval.