Self-serve metrics generation

Last updated by Yana Wang on 3/14/2022 8:56:29 AM This content is over 788 days old.

Top results (relevance) Coverage, PCR, CCR script for outlooks: https://microsoft-my.sharepoint.com/:u:/p/minzha/EeUC097GgllMihIxCb2gfZYBYjNI7MaZDk3lFhMZTtoRaQ?e=lG0uTd

Input:

SearchSession.view and specify the start, end dates and entypoints parameters.

Script Logics breakdown:

1. Line 14 till Line 57 are taken from the engagement metric set view ([link](https://msasg.visualstudio.com/Auriga/_git/DM.Auriga.Impression?path=%2Fsrc%2Fprivate%2FView%2FMetricSetViews%2FMetricSetViews%2FViews%2FMetricSet_MSAI_SubstrateSearch_Engagement.view)), which the scorecard directly consumes.
2. Line 75 till Line 86 are got from the scorecard mdl ([link](https://dev.azure.com/experimentation/ExpMetrics/_git/sssEngagementMetrics?path=%2FUserDefinedFiles%2FMDLSearchCoverageBySortOrderByGroup.xml&version=GBmaster&line=12&lineEnd=14&lineStartColumn=11&lineEndColumn=23&lineStyle=plain&_a=contents)), and could be revised to apply to other groups / entities (i.e. for specific group coverage: [link](https://dev.azure.com/experimentation/ExpMetrics/_git/sssEngagementMetrics?path=%2FUserDefinedFiles%2FMDLSearchCoverageOverallByGroup.xml&version=GBmaster&line=16&lineEnd=19&lineStartColumn=1&lineEndColumn=23&lineStyle=plain&_a=contents))
3. The script then mimics the logic in the scorecard to compute the Coverage / PCR / CCR, see an example of the logic breakdown blow.

Output:

The Coverage / PCR / CCR by different aggregation level.

Example output: https://microsoft-my.sharepoint.com/:x:/p/minzha/EZQY_EOeba1FhGS7bvcwl8kB3UfNbiG2y9kxIwxb9n828g?e=BFLvbG

Coverage/PCR/CCR logic breakdown

Coverage/PCR/CCR are computed at impression level and rolled up to upper level by weighted average. i.e. assume there are two conversations:

1. Conversation1 with 4 impressions:
	a. Impression1 with Top Email results shown and user clicked on it.
	b. Impression2 with Top Email results shown and user didn't click on it.
	c. Impression3 with Top Email results not shown.
	d. Impression4 with Top Email results not shown.
2. Conversation2 with 2 impressions:
	a. Impression1 with Top Email results shown and user clicked on it.
	b. Impression1 with Top Email results not shown.
	

So for impression level, 1. the coverage is 3/(2+4) = 0.5 2. the PCR is 2/(2+4) = 0.3333 3. the CCR is 2/(1+2) = 0.6667 For Conversation level, 1. the coverage is (2/4+1/2)/2 = 0.5 2. the PCR is (1/4+1/2)/2 = 0.375 3. the CCR is (1/2+1/1)/2 = 0.75

Similarly, the same logic applies for Session and User level.