How to run spark unit test code in local mode

Last updated by Na Li on 2/6/2024 6:00:01 AM

Goal

You can check out the code, compile, and run the unit tests in spark local mode.

Installation

1. Install IntelliJ Community edition

2. File -> Settings -> Plugins. Search for Scala and install.

scala plugin

3. Install Visual Studio 2022

The msbuild sdk version in global.json is configed for Visual Studio 2022. Visual Studio 2019 is another option that can be used as well, but you need to update the msbuild sdk version in global.json to be 3.1.426 when use Visual Studio 2019. Recommand to use Visual Studio 2022, which build faster.

4. Install JDK 1.8 (or 8)

IDEA bundled jdk is also a good option。

5. Install Maven

IDEA bundled maven is also a good option。

  • Download and unzip the package in C:\Program Files\ (e.g. C:\Program Files\apache-maven-3.6.3)

  • In an elevated cmd prompt set JAVA_HOME to the location of the JDK

    java home

    Note: You can check whether JAVA_HOME are set successfully via "echo %JAVA_HOME%" like below(if show "%JAVA_HOME%" means not not succeed). Then you can try set it via following 'edit the system environment variables' method as well.

    set java home succeed set java home via edit

  • From the Windows start menu, search for 'edit the system environment variables'

  • Edit the PATH variable by adding a new line pointing to the maven bin folder maven path

Repo and settings

1. Clone repo: EuclidOIVIC Branch: master.

2. File > Project structure. Choose the proper SDK:

project settings

3. In Global Libraries, install scala 2.11.8 SDK.

Build

1. Run "Developer Command Prompt for VS 2022" (or "Developer Command Prompt for VS 2019") as administrator

2. CD to the local folder of EuclidOIVIC, run "init.cmd"

3. Run "msbuild"

Note: If you are using "Developer Command Prompt for VS 2019", udapte the Microsoft.Build.Traversal in "EuclidOIVIC\global.json" to be 3.1.426 first, then run "msbuild" in cmd.

  • If "The term ‘msbuild’ is not recognized", append C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin to your PATH and restart the console

  • Make sure the sdk version in global.json matches the version of dotnet (C:\Program Files\dotnet\sdk, the folder name is the version)

  • If something always going wrong，e.g. %XPLATPROJROOT% variable can not be recognized by build.bat; null ls -F has no permission; can not find winutils.exe; try to delete the entire EuclidOIVIC folder and retry git clone， everything may be ok。

Open the project in IntelliJ and run unit test

Now click the green triangle beside the test function to run unit test.

scala unit test result can be different with mvn test，you need rely on mvn test result because it is in msbuild。

Note: If you can't find run icon next to the unit test, you can retry via only open the sub-project folder, like - EuclidOIVIC\sources\dev\Projects\xplat\oivic.curation. only open sub-project folder