Manage DiffDoctor Indicator Retention

Last updated by Yana Wang on 4/6/2022 4:38:13 AM This content is over 765 days old.

Background

We usually store some files in blob, which is a common data storage mode, but some historical files will not be used after being stored for a period of time. Keeping these files will occupy storage space, so we use life cycle management strategy to manage these files.

Check the DiffDoctor Indicator files on Azure Portal

Access Permission

  1. Request eligibility msaidataplat on OSP Portal . We use AAD authentication for external users, when the eligibility is granted, your debug account will be added into corresponding AAD group.

  2. Request Contributor on DMS-DatacenterManagement.

    Torus Command

    Request-AzureResourceRoleElevation -Role Contributor -SubscriptionId 8b8b2cf1-f7e3-4d86-8581-a15cd35463a4 -Reason "Publish ADF Pipeline" -Duration 4

DiffDoctor Azure Blob Storage Structure

Navigate to the MSAI Data Platform -> MSAI-DataPipeline -> diffdoctorvalidation storage account

DiffDoctor Azure Blob Storage Structure

Current Retention Rules

The last modification time of the file is more than 10 days from now. This file will be deleted.

Set lifecycle management policies

Step 1 :

Navigate to the diffdoctor Azure Storage Account resource and click "Lifecycle Management", and select "Add a rule" in lifecycle management.

Add Rule

Step 2 :

Enter the rule name, select the rule scope, select the Blob type (You can check the blob type base on your actual blobs), and select the subtypes of the Blob to manage.

Fill Rules

Step 3 :

Sets the number of days since the file was last modified before it was deleted.

Set Time Limit

Step 4 :

Sets the file to which this rule applies. A maximum of 10 filter criteria can be set.

Screening File

Step 5 :

Click the "Add" button and you will see the rule successfully set on this page. Click the rule name to change the Settings.

Succeeded Adding Rule

Attention

  1. Lifecycle policies only apply to block Blobs, not Page Blobs
  2. Azure reads the strategy once a day, and it may take 24 hours to take effect after the strategy is set up. The specific operation may also take a while to complete, so the new or updated strategy needs to wait for 48 hours before it can be successfully applied.

Reference

https://docs.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview