How to Configure DiagnosisInformation in AurigaSentry
Background
AurigaSentry provides different Diagnosis mechanisms to check the upstream or dependency job status to adjust the severity and fetch the debugging information for a better monitoring experience. This feature improves the accuracy of the monitors. It helps to reduce the noisy alerts and avoid the unnecessary invoke actions for the dependency/upstream issues.
Diagnosis Dependency Configuration
AurigaSentry has supported different kinds of Dependency configurations. They could detect different information sources.
BaseDependency Configuration
There are several common properties for the dependency.
| Property Name | Type | Description |
|---|---|---|
| Frequency | Frequency | The dependency frequecy e.g. Daily, Hourly etc. |
| LookbackWindowHours | List<int> | The lookback hour list. The number k from this list means the DateTime.Now.AddHours(-k) |
| IsInternalDependency | bool | This property means the current dependency ownership is same with the current monitor configuration ownership. Default value is False. |
StreamDependency Configuration
StreamDependency is used for checking the upstream input/dependency Cosmos stream/indicator/files existing.
| Property Name | Type | Description |
|---|---|---|
| AdlsPathPattern | string | The full ADLS path of the Dependency/Input file on Cosmos VCs. (e.g. "adl://exchange-storage-prod-c14.azuredatalakestore.net/local/Aggregated/Datasets/Public/Usage/Impression/SearchEventsLogsHourly/%Y/%m/%d/SearchEventsLogsHourly_%Y-%m-%d-%H.ss") |
The AdlsPathPattern of StreamDependency supports both Gen1 and Gen2 ADLS path pattern.
- ADLS Gen1 Path starts with adl://{adlsaccount}.azuredatalakestore.net/**
- ADLS Gen2 Path starts with abfs://filesystem@{adlsaccount}.dfs.core.windows.net/**
CosmosJobDependency Configuration
CosmosJobDependency is used for define the upstream Cosmos Job information to check its status. Sometimes, a data pipeline could be triggered until the upstream Cosmos Job completes Successfully. So, we may configure the CosmosJobDependency diagnosis config to debug the upstream job status so that reduce the current pipeline noisy alert.
| Property Name | Type | Description |
|---|---|---|
| VcName | string | The virtual cluster Name of the Cosmos (e.g. exchange.storage.prod) |
| Cluster | string | The cluster name of the Cosmos (e.g. Cosmos14) |
| JobNamePattern | string | The Cosmos Scope job name pattern with the time stamp pattern. (e.g. Cooking_Usage_SearchImpressionHourly_%Y-%m-%d-%H) |
AdlaJobDependency Configuration
Similar to the CosmosJobDependency config, AdlaJobDependency is used for define the upstream ADLA job information to check its status.
| Property Name | Type | Description |
|---|---|---|
| AdlaAccount | string | The ADLA account name. (e.g. substrateadlaneuprod) |
| JobNamePattern | string | The ADLA job name pattern with the timestamp pattern. (e.g. Cooking_Usage_SearchImpressionHourly_%Y-%m-%d-%H) |
SentryMonitorDependency Configuration
SentryMonitorDependency is used for define the parent sentry monitor dependency. As there may be upstream and downstream relationships among the different monitors. This dependency config could provide a mechanism if the current monitor is impacted by an existing active incidents triggered by other monitors.
| Property Name | Type | Description |
|---|---|---|
| MontorType | MonitorType | The dependency montor type. |
| MonitorId | int | The monitor identifier of the monitor. |
| DependencyName | string | The depedency name. |
Diagnosis Information Configuration
DiagnosisInformation is an optional column enabled in SLAMonitorConfigs table. It is a serialized Json string from DiagnosisInformation class. The diagnostors will detect and deserialize the configurations from this column. Then, AurigaSentry will diagnosis the dependency status for the specific monitor configuration.
Configuration instruction
| Dependency Name | Type | Usage Descrption |
|---|---|---|
| CosmosJobDependencies | List<CosmosJobDependency> | The dependencies list contains the upstream Cosmos Jobs information for the current job/pipeline. |
| CosmosStreamDependencies | List<StreamDependency> | The dependencies list contains the upstream Cosmos Streams information for the current job/pipeline. |
| SentryMonitorDependencies | List<SentryMonitorDependency> | The dependencies list contains the upstream Sentry Monitor information. The configuration is used for link the current monitor to the previous root cause monitors. |
| AdlaJobDependencies | List<AdlaJobDependency> | The dependencies list contains the upstrea Adla Jobs information for the current job/pipeline. |
| AdlsGen2StreamDependencies | List<StreamDependency> | The dependencies list contains the upstream Adls Gen2 Streams information for the current job/pipeline. |
Sample
{
"CosmosJobDependencies": [
{
"VcName": "exchange.storage.prod",
"Cluster": "cosmos14",
"JobNamePattern": "Cooking_Usage_SearchImpressionHourly_%Y-%m-%d-%H",
"Frequency": "Hourly",
"LookbackWindowHours": [0,-1,-2,-3,-4,-5,-6,-7,-8,-9,-10,-11,-12,-13,-14,-15,-16,-17,-18,-19,-20,-21,-22,-23]
},
{
"VcName": "exchange.storage.prod",
"Cluster": "cosmos14",
"JobNamePattern": "Cooking_Usage_LogicalSearchImpressionHourly_%Y-%m-%d-%H",
"Frequency": "Hourly",
"LookbackWindowHours": [0,-1,-2,-3,-4,-5,-6,-7,-8,-9,-10,-11,-12,-13,-14,-15,-16,-17,-18,-19,-20,-21,-22,-23]
}
],
"CosmosStreamDependencies": [
{
"AdlsPathPattern": "adl://exchange-storage-prod-c14.azuredatalakestore.net/local/Aggregated/Datasets/Public/Usage/Impression/Indicator24Hours/%Y/%m/LogicalSearchImpression24Hours_%Y_%m_%d.txt",
"Frequency": "Daily",
"LookbackWindowHours": [0],
"IsInternalDependency": true
}
],
"SentryMonitorDependencies": null,
"AdlaJobDependencies": null,
"AdlsGen2StreamDependencies": null
}