DataUsageMetadata
Overview
DataUsageMetadata dataset contains the streamset data usage metadata which is maintained by MSAI Data Platform. Data Usage metrics is used for indicating the consumer, consumed pipeline and jobs information and so on.
Basic Information
Data Latency: T-1Day
Freshness: Daily Refresh
Upstream Data Source: SOTEL Insights Data
Data StartTime: February 2022
Data Retention: No Expiration
Data Content: The snapshot streamset usage metadata.
Columns
| Column Name | Data Type | Description | Sample Value | IsNullable |
|---|---|---|---|---|
| ID | int | Auto-increment field | 1 | False |
| StoragePlatform | varchar(10) | Source data classification | Cosmos | True |
| ClusterName | varchar(200) | The cluster name of the streamset. | cosmos14-prod-cy2 | True |
| VCName | varchar(200) | The VC name of the streamset. | exchange.storage.prod | True |
| StreamSetNormalizedPath | varchar(500) | The normalized path of the streamset. | /local/Aggregated/Datasets/Public/Usage/Impression/LogicalSearchImpressionHourly/Mso/Search/%n/%n/%n/MsoLogicalSearchImpressions_%n-%n-%n.ss | True |
| StreamFolderPath | varchar(500) | The normalized path prefix of the streamset. | /local/Aggregated/Datasets/Public/Usage/Impression/LogicalSearchImpressionHourly/Mso/Search/ | True |
| JobClusterName | varchar(200) | The cluster name of the job, may be different from the StreamSetClusterName. | cosmos14-prod-cy2 | True |
| JobVCName | varchar(200) | The VC name of the job, may be different from the StreamSetVCName. | exchange.storage.prod | True |
| JobNormalizedName | varchar(200) | The normalized name of the job. | AnaExp_Others_Job%n_Run%n__Pri_%n.%n_lsrep_TrigFilt_OneOffJob_%n_%n_%nn%n_sssUsageQualityOfServiceMetric.SearchQualityOfService.AB.Standard_%n-%n-%nT%n_%n-%n-%nT%n | True |
| SubmittedBy | varchar(200) | The submittor of the job. | CO4AEAP000079D2 | True |
| JobCount | int | How many job instances this record contains. | 5 | True |
| LatestAccessTime | datetime | The date on which the latest job instance accessed the streamset. | 2022-02-01 13:46:13.000 | True |
| SnapshotTime | datetime | The snapshot time when capture this data usage metadata. | 2022-02-28 00:00:00.000 | True |
| ConsumerEMAlias | varchar(200) | The alias of engineer manager under which this jobset is registered | yupan | True |
| ConsumerJobContact | varchar(200) | The alias of contacts | 3sexpdri@microsoft.com | True |
| ConsumerServiceId | varchar(500) | The id of service tree node under which this jobset is registered | a3374a5c-dd8c-4da4-99de-bdaeedbcdbd2 | True |
| ConsumerServiceName | varchar(500) | The name of the service. | ExP Classic | True |
| DataCategory | varchar(200) | DataCategory by StreamSetNormalizedPath | Impression | True |
| DataSubCategory | varchar(200) | Data sub Category by StreamSetNormalizedPath | /LogicalSearchImpressionHourly | True |
Usage
Sample Scripts
Find specific data according to criteria of the DataUsageMetadata,as input or output
SELECT ID
,StoragePlatform
,ClusterName
,VCName
,StreamSetNormalizedPath
,StreamFolderPath
,JobClusterName
,JobVCName
,JobNormalizedName
,SubmittedBy
,JobCount
,LatestAccessTime
,SnapshotTime
,ConsumerEMAlias
,ConsumerJobContact
,ConsumerServiceId
,ConsumerServiceName
,DataCategory
,DataSubCategory
FROM [dbo].[DataUsageMetadata]
WHERE DataCategory = 'Session'
AND YEAR(LatestAccessTime) = 2022
AND MONTH(LatestAccessTime) = 02
AND DAY(LatestAccessTime)IN(01,02)
Sample Data
| ID | StoragePlatform | ClusterName | VCName | StreamSetNormalizedPath | StreamFolderPath | JobClusterName | JobVCName | JobNormalizedName | SubmittedBy | JobCount | LatestAccessTime | SnapshotTime | ConsumerEMAlias | ConsumerJobContact | ConsumerServiceId | ConsumerServiceName | DataCategory | DataSubCategory |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2 | Cosmos | cosmos14-prod-cy2 | exchange.storage.prod | /local/Aggregated/Datasets/Public/Usage/Session/SearchSessionComputationFast/%n/%n/SessionComputationFast_%n-%n-%n.ss | /local/Aggregated/Datasets/Public/Usage/Session | cosmos14-prod-cy2 | exchange.storage.prod | AnaExp_Others_Job%n_Run%n__Pri_%n.%n_lsrep_TrigFilt_OneOffJob_%n_%n_%nn%n_sssSuccessMetrics.MSOSearchSuccess.AB.Standard_%n-%n-%nT%n_%n-%n-%nT%n | CO4AEAP000079CE | 5 | 2022-02-01 13:52:36.000 | 2022-02-28 00:00:00.000 | yupan | 3sexpdri@microsoft.com | a3374a5c-dd8c-4da4-99de-bdaeedbcdbd2 | ExP Classic | Session | /SearchSessionComputationFast |
| 3 | Cosmos | cosmos14-prod-cy2 | exchange.storage.prod | /local/Aggregated/Datasets/Public/Usage/Session/SearchSessionComputationFast/%n/%n/SessionComputationFast_%n-%n-%n.ss | /local/Aggregated/Datasets/Public/Usage/Session/SearchSessionComputationFast/ | cosmos14-prod-cy2 | exchange.storage.prod | AnaExp_Others_Job%n_Run%n__Pri_%n.%n_lsrep_TrigFilt_OneOffJob_%n_%n_%nn%n_sssSuccessMetrics.MSOSearchSuccess.AB.Standard_%n-%n-%nT%n_%n-%n-%nT%n | CO4AEAP000079CE | 5 | 2022-02-01 13:52:36.000 | 2022-02-28 00:00:00.000 | yupan | 3sexpdri@microsoft.com | a3374a5c-dd8c-4da4-99de-bdaeedbcdbd2 | ExP Classic | Session | /SearchSessionComputationFast |
| 22 | Cosmos | cosmos14-prod-cy2 | exchange.storage.prod | /local/Aggregated/Datasets/Public/Usage/Session/SearchSessionComputationFast/%n/%n/SessionComputationFast_%n-%n-%n.ss | /local/Aggregated/Datasets/Public/Usage/Session | cosmos14-prod-cy2 | exchange.storage.prod | AnaExp_UX_Job%n_Run%n__Pri_%n.%n_lsrep_TrigFilt_OneOffJob_%n_%n_%nn%n_sssSuccessMetrics.MSBSearchSuccess.AB.Standard_%n-%n-%nT%n_%n-%n-%nT%n | CO4AEAP000079D4 | 2 | 2022-02-02 12:32:16.000 | 2022-02-28 00:00:00.000 | yupan | 3sexpdri@microsoft.com | a3374a5c-dd8c-4da4-99de-bdaeedbcdbd2 | ExP Classic | Session | /SearchSessionComputationFast |
| 23 | Cosmos | cosmos14-prod-cy2 | exchange.storage.prod | /local/Aggregated/Datasets/Public/Usage/Session/SearchSessionComputationFast/%n/%n/SessionComputationFast_%n-%n-%n.ss | /local/Aggregated/Datasets/Public/Usage/Session/SearchSessionComputationFast/ | cosmos14-prod-cy2 | exchange.storage.prod | AnaExp_UX_Job%n_Run%n__Pri_%n.%n_lsrep_TrigFilt_OneOffJob_%n_%n_%nn%n_sssSuccessMetrics.MSBSearchSuccess.AB.Standard_%n-%n-%nT%n_%n-%n-%nT%n | CO4AEAP000079D4 | 2 | 2022-02-02 12:32:16.000 | 2022-02-28 00:00:00.000 | yupan | 3sexpdri@microsoft.com | a3374a5c-dd8c-4da4-99de-bdaeedbcdbd2 | ExP Classic | Session | /SearchSessionComputationFast |
| 24 | Cosmos | cosmos14-prod-cy2 | exchange.storage.prod | /local/Aggregated/Datasets/Public/Usage/Session/SearchSessionComputationFast/%n/%n/SessionComputationFast_%n-%n-%n.ss | /local/Aggregated/Datasets/Public/Usage/Session | cosmos14-prod-cy2 | exchange.storage.prod | AnaExp_UX_Job%n_Run%n__Pri_%n.%n_lsrep_TrigFilt_OneOffJob_%n_%n_%nn%n_sssAnswersMetrics.DefaultAnswers.AB.Standard_%n-%n-%nT%n_%n-%n-%nT%n | CO4AEAP000079D1 | 2 | 2022-02-02 15:21:45.000 | 2022-02-28 00:00:00.000 | yupan | 3sexpdri@microsoft.com | a3374a5c-dd8c-4da4-99de-bdaeedbcdbd2 | ExP Classic | Session | /SearchSessionComputationFast |
| 25 | Cosmos | cosmos14-prod-cy2 | exchange.storage.prod | /local/Aggregated/Datasets/Public/Usage/Session/SearchSessionComputationFast/%n/%n/SessionComputationFast_%n-%n-%n.ss | /local/Aggregated/Datasets/Public/Usage/Session/SearchSessionComputationFast/ | cosmos14-prod-cy2 | exchange.storage.prod | AnaExp_UX_Job%n_Run%n__Pri_%n.%n_lsrep_TrigFilt_OneOffJob_%n_%n_%nn%n_sssAnswersMetrics.DefaultAnswers.AB.Standard_%n-%n-%nT%n_%n-%n-%nT%n | CO4AEAP000079D1 | 2 | 2022-02-02 15:21:45.000 | 2022-02-28 00:00:00.000 | yupan | 3sexpdri@microsoft.com | a3374a5c-dd8c-4da4-99de-bdaeedbcdbd2 | ExP Classic | Session | /SearchSessionComputationFast |