Data Cooking Pipeline Code Repository Migration

Last updated by MerlinBot on 2/2/2024 5:22:16 AM

Background

We proposed to setup a new code repository for hosting the S+A data cooking pipeline codebase. As part of that, we need to migrate the Search data cooking pipeline codes including scripts, data schemas, functionality implementation and so on from legacy Odin repo to the new Auriga Repo.

For any questions, you could submit your question on the SO link.

Repo source link

Odin Repo: https://msasg.visualstudio.com/DefaultCollection/Shared%20Data/_git/DM.Odin.Impression
Auriga Repo: https://msasg.visualstudio.com/Auriga/_git/DM.Auriga.Impression

Proposal

MSAI S+A code repo proposal

Through the new data repository migiration, we propose to optimize the data cooking pipeline solution infrastructure. We need to figure out these points.

Goals:
1. Migrate the whole codebase to the new Auriga Repo.
2. Setup new Azure pipeline(YAML) for the buddy build.
3. Deprecate CoreXT environment so that support the VSTS Build on the test agent.
4. Convert the .csproj format to VS2017 format.
5. Setup the build pipelines (Buddy/Official etc.) on MSAI UDP-Pool.
6. Support to deploy from new code repo.
Non Goals:
1. Support the MSBuild 16.
2. Enable the code style rules from O364Exchange repo policies.
3. Convert code to be compatible with ADL SDK.
4. Test cases and pipelines optimization.

New Repo Projects Summary

We need to migrate the csproj projects off the CoreXT environment, convert the csproj format to VS2017 version.

DM.Odin.Impression repo have 35 projects (29 csproj projects + 6 scopeproj projects).

Scope script projects (Need to install ScopeStudio extension in VS)

Project name (.scopeproj)	Path
OfficeSearchImpressionLibrary	src/private/Pipeline/OfficeSearchImpression
OfficeSearchSessionLibrary	src/private/Pipeline/OfficeSearchSession
ImpressionViewLibrary	src/private/View/ImpressionView
LogicalImpressionViewLibrary	src/private/View/LogicalImpressionView
SearchSessionViewLibrary	src/private/View/SearchSessionView
AllViewTestsLibrary	src/private/View/AllViewsTests

.NET Framework library projects

Project name (.csproj)	Path	Switch from CoreXT to VSTSBuild	Convert to VS2017	Is Cosmos VC E2E Test
DataMining.Office.Impression.DataTypes	src/private/Common	Yes	Yes	No
DataMining.Office.Impression.Utility	src/private/Common	Yes	Yes	No
SearchImpressionPipeline	src/private/Pipeline/OfficeSearchImpression	Yes	Yes	No
OfficeSearchImpression.UnitTest	src/private/Pipeline/OfficeSearchImpression	Yes	Yes	No
OfficeSearchImpressionLibrary.Test	src/private/Pipeline/OfficeSearchImpression	Yes	Yes	Yes
OfficeSearchSession	src/private/Pipeline/OfficeSearchSession	Yes	Yes	No
OfficeSearchSession.UnitTest	src/private/Pipeline/OfficeSearchSession	Yes	Yes	No
OfficeSearchSessionLibrary.Test	src/private/Pipeline/OfficeSearchSession	Yes	Yes	Yes
AllViewTests	src/private/View/AllViewsTests	Yes	Yes	Yes
ImpressionView	src/private/View/ImpressionView	Yes	Yes	No
ImpressionView.UnitTest	src/private/View/ImpressionView	Yes	Yes	Yes
ImpressionViewLibrary.Test	src/private/View/ImpressionView	Yes	Yes	Yes
OdinImpressionViewResolver	src/private/View/ImpressionView	Yes	Yes	No
OdinImpressionViewResolverShared	src/private/View/ImpressionView	Yes	Yes	No
OdinImpressionViewResolver.UnitTest	src/private/View/ImpressionView	Yes	Yes	No
LogicalImpressionViewInternal	src/private/View/LogicalImpressionView	Yes	Yes	No
LogicalImpressionViewLibrary.Tests	src/private/View/LogicalImpressionView	Yes	Yes	Yes
OdinLogicalImpressionViewResolver	src/private/View/LogicalImpressionView	Yes	Yes	No
OdinLogicalImpressionViewResolver.UnitTest	src/private/View/LogicalImpressionView	Yes	Yes	No
MetricSetViews	src/private/View/MetricSetViews	Yes	Yes	No
MetricSetViews.Test	src/private/View/MetricSetViews	Yes	Yes	Yes
ScopeTestUtilities	src/private/View/MetricSetViews	Yes	Yes	No
ScopeTestUtilities.Test	src/private/View/MetricSetViews	Yes	Yes	Yes
ScorecardUtility	src/private/View/MetricSetViews	Yes	Yes	No
ScorecardUtility.UnitTest	src/private/View/MetricSetViews	Yes	Yes	No
SearchSessionView	src/private/View/SearchSessionView	Yes	Yes	No
SearchSessionView.UnitTest	src/private/View/SearchSessionView	Yes	Yes	Yes
SearchSessionViewInternal	src/private/View/SearchSessionView	Yes	Yes	No
SearchSessionViewLibrary.Test	src/private/View/SearchSessionView	Yes	Yes	Yes

Source Code Migration Steps

Sample Pull Requests
1. Pull Request 1811004: Migrate Data cooking pipeline to the VS2017 and deprecate CoreXT
2. Pull Request 1832289: Add E2E test step in build pipeline

1. Prerequisites

Visual Studio 2019 Download
.NET Framework >4.7.2 Runtime Download
ScopeStudio visx plugin for Visual Studio see //aka.ms/scopestudio .
CosmosVC access permissions. Check this OneNote Page.
A valid Agent Pool. Onboard one or several build agents into the agent pool.

2. Convert csproj format to VS2017 format

Keep the parity build configuration (AnyCPU)
Migrate the specific hint path reference from legacy csproj file.

3. Configure the Path Macro definitions

Define the global Path macro values in Directory.Build.props
Migrate the other path macro values from the legacy csproj file.

4. Migrate package references from CoreXT to 'PackageReference' Nuget

Migrate from packages.config to PackageReference
Configure the specific nuget reference

5. Port the robocopy task to the new project

You need to confirm if the new solution robocopy/binplace behaviour in parity with legacy projects. Especially, the library files such as .dll, .pdb, .xml and so on.

6. Build and test your new solution

Build and verify your new solution contains converted projects locally.
Check and pass the unit test and E2E test locally.
- For the Cosmos access E2E test cases, you need to migrate your implementations from ScopeSDK to ADLs SDK. See the changes in Pull Request 1832289: Add E2E test step in build pipeline.
- Limit the Scope.Run environment to the local environment.

7. Setup Azure Pipeline (Yaml)

Create a Azure Pipeline yaml file to setup the build pipeline for new Solutions.
You can kindly onboard your VM to our agent pool to run the build pipeline. Please follow Onboard New Test Agent.

7.1 Build and Unit Test yaml template

# Build and Run the Unit Tests for the SearchDataPipeline solutions

parameters:
  buildPlatform: 'Any CPU' # Build platform
  buildConfiguration: 'debug' # Build configuration
  solutions: 'src/private/**/*.sln' # Solution paths
  projects: 'src/private/**/*.csproj'
  runUnitTest: 'true'

steps:
  - task: NuGetToolInstaller@1
    displayName: 'Install NuGet 5.7.x'
    inputs:
      versionSpec: '5.7.x'

  - task: PowerShell@2
    displayName: 'Set Nuget magic environment variables'
    # Stuff like the NuGet restore step seem to sniff this out and this appears to be the only way to get it to restore the correct architecture
    inputs:
      pwsh: false
      failOnStderr: true
      targetType: 'inline'
      script: |
        Write-Host "##vso[task.setvariable variable=PLATFORM]${{parameters.buildPlatform}}"

  - task: PowerShell@2
    displayName: Clear NuGet cache
    continueOnError: true
    inputs:
      pwsh: false
      # This seems to randomly fail from time to time; don't want to fail the build on this, goal is just to make sure
      # the build machine doesn't run out of space when we download this build's packages. So run it as en exe and just
      # ignore any error messages.
      failOnStderr: false
      targetType: 'inline'
      script: nuget locals all -clear

  - task: NuGetAuthenticate@1
    displayName: 'Nuget Authenticate for Nuget feeds'
    inputs:
      nuGetServiceConnections: 'Auriga-BingPacman, Auriga-Cosmos, Auriga-O365, Auriga-CoreXT, Auriga-CoreXTNuget, Auriga-CloudBuild, Auriga-OSSMSFT, Auriga-OSS'
      forceReinstallCredentialProvider: true

  - task: NuGetCommand@2
    displayName: 'Nuget restore'
    inputs:
      command: 'restore'
      restoreSolution: '${{ parameters.projects }}'
      feedsToUse: 'config'
      nugetConfigPath: 'nuget.config'
      verbosityRestore: 'quiet'

  - task: VSBuild@1
    displayName: 'Build solutions'
    inputs:
      solution: '${{ parameters.solutions }}'
      platform: '${{ parameters.buildPlatform }}'
      configuration: '${{ parameters.buildConfiguration }}'

  - task: VSTest@2
    displayName: 'Run the unit tests without Cosmos E2E tests.'
    condition: and(succeeded(), eq('${{parameters.runUnitTest}}', 'true'))
    inputs:
      testSelector: 'testAssemblies'
      testAssemblyVer2: |
        src\**\*UnitTest*.dll
      searchFolder: '$(System.DefaultWorkingDirectory)'
      runSettingsFile: testsettings/unittests.runsettings
      platform: '${{ parameters.buildPlatform }}'
      configuration: '${{ parameters.buildConfiguration }}'

7.2 Build and E2E Test yaml

# .NET Desktop
# Build and run tests for .NET Desktop or Windows classic desktop solutions.
# Add steps that publish symbols, save build artifacts, and more:
# https://docs.microsoft.com/azure/devops/pipelines/apps/windows/dot-net

trigger:
  - master

jobs:
- job: 'BuildAndTest1'
  # Just keep the part 1 of the Cosmos E2E test cases in order to improve the build efficiency.
  # This job will execute the cases from:
  #    - OfficeSearchImpressionLibrary.Test
  displayName: 'Build and Test for Data Cooking pipelines Part-1' 
  timeoutInMinutes: 120
  pool:
    name: MSAI UDP-Pool
    demands:
    - msbuild
    - visualstudio
    - vstest  
  
  steps:
    - template: build/steps/build-and-unittests.yml

    - task: VSTest@2
      displayName: 'Run Cosmos E2E Tests'
      condition: and(succeeded(), ne(variables['Build.SourceBranch'], 'refs/heads/master'))
      inputs:
        testSelector: 'testAssemblies'
        testAssemblyVer2: |
            src\private\Pipeline\OfficeSearchImpression\OfficeSearchImpressionLibrary.Test\objd\x64\net472\OfficeSearchImpressionLibrary.Test.dll
        searchFolder: '$(System.DefaultWorkingDirectory)'
        runSettingsFile: testsettings/unittests.runsettings
        platform: '$(buildPlatform)'
        configuration: '$(buildConfiguration)'

- job: 'BuildAndTest2'
  # Just keep the part 2 of the Cosmos E2E test cases in order to improve the build efficiency.
  # This job will execute the cases from:
  #    - ImpressionViewLibrary.Test
  #    - LogicalImpressionViewLibrary.Tests
  #    - SearchSessionViewLibrary.Test
  #    - OfficeSearchSessionLibrary.Tests
  #    - MetricSetViews.Tests
  displayName: 'Build and Test for Data Cooking pipelines Part-2'
  timeoutInMinutes: 120
  pool:
    name: MSAI UDP-Pool
    demands:
    - msbuild
    - visualstudio
    - vstest  
  
  steps:
    - template: build/steps/build-and-unittests.yml

    - task: VSTest@2
      displayName: 'Run Cosmos E2E Tests'
      condition: and(succeeded(), ne(variables['Build.SourceBranch'], 'refs/heads/master'))
      inputs:
        testSelector: 'testAssemblies'
        testAssemblyVer2: |
            src\private\View\ImpressionView\ImpressionViewLibrary.Tests\objd\x64\net472\ImpressionViewLibrary.Tests.dll
            src\private\View\LogicalImpressionView\LogicalImpressionViewLibrary.Tests\objd\x64\net472\LogicalImpressionViewLibrary.Tests.dll
            src\private\View\MetricSetViews\MetricSetViews.Test\objd\x64\net472\MetricSetViews.Test.dll
            src\private\View\SearchSessionView\SearchSessionViewLibrary.Test\objd\x64\net472\SearchSessionViewLibrary.Test.dll
            src\private\Pipeline\OfficeSearchSession\OfficeSearchSessionLibrary.Tests\objd\x64\net472\OfficeSearchSessionLibrary.Tests.dll
        searchFolder: '$(System.DefaultWorkingDirectory)'
        runSettingsFile: testsettings/unittests.runsettings
        platform: '$(buildPlatform)'
        configuration: '$(buildConfiguration)'