How To Resolve TSG for Cosmos Streamset Privacy Issues

Last updated by Yana Wang on 3/14/2022 7:12:55 AM This content is over 788 days old.

If your stream/file in Cosmos encounter privacy issue (check current issues here), first check whether the stream/file is for production or test purpose. If it's only for test, you can simply delete it. Otherwise please find the corresponding TSG below and follow to resolve the issue. If the issue is not listed here, please check the link in Reference section to find a solution.

Note: All sample data for search pipeline are currently stored in https://aad.cosmos14.osdinfra.net/cosmos/exchange.storage.prod/local/Resources/MSAI/SSS/SampleData/.

Fix delete readiness issues

For error like "The structured stream with random partition type is not supported", please follow the wiki below to solve it:

How to solve Random partition errors: https://ngpwiki.azurewebsites.net/StreamIssue

Fix tagging issues

If there is an action item created for your team to fix tagging violations and have one of the below tagging violations:

  1. Subject Identifier Missing
  2. Datatype Missing
  3. Data Class Missing

Please use this link https://ngpwiki.azurewebsites.net/Tagging  as a guide to resolve the tagging violations and be GDPR Compliant. For search pipeline stream tagging, you can also refer to this OneNote page.

Generally search pipeline streams can be classified as the following two types:

  1. Product and service usage data, such as impression/session related streams. They should has the tag of Privacy.DataType.ProductAndServiceUsage and either one between Privacy.Asset.IntermediateEngineering (retention > 30 days) and Privacy.Asset.LongTail (retention <= 30 days).

    pic

  2. Non-personal Data, such as aggregated business/DQ metrics. The tag of Privacy.Asset.NonPersonal would be enough.

    pic

For other tagging issues such as "Inferred tag will not be processed by NGP delete agent", please find the corresponding solution from https://ngpwiki.azurewebsites.net/TagIssue.

Reference