This page summarizes the steps to install Spark 2.2.1 in your Windows environment.
Tools and Environment
- GIT Bash
- Command Prompt
- Windows 10
Download Binary Package
Download the latest binary from the following site:
In my case, I am saving the file to folder: F:\DataAnalytics.
UnZip binary package
Open Git Bash, and change directory (cd) to the folder where you save the binary package and then unzip:
$ cd F:\DataAnalytics
fahao@Raymond-Alienware MINGW64 /f/DataAnalytics $ tar -xvzf spark-2.2.1-bin-hadoop2.7.tgz
In my case, spark is extracted to: F:\DataAnalytics\spark-2.2.1-bin-hadoop2.7
Setup environment variables
JAVA\_HOME
Follow section ‘JAVA_HOME environment variable’ in the following page to setup JAVA_HOME
https://app.kontext.tech/docs/DataAndBusinessIntelligence/p/install-zeppelin-073-in-windows
SPARK\_HOME
Setup SPARK_HOME environment variable with value of your spark installation directory.
https://api.kontext.tech/resource/00bec7df-53c3-5b1c-91b7-7a820f20648a
PATH
Added ‘%SPARK_HOME%\bin’ to your path environment variable.
Verify the installation
Verify command
Run the following command in Command Prompt to verify the installation.
%SPARK_HOME%\bin\spark-shell
The screen should be similar to the following screenshot:
https://api.kontext.tech/resource/824dc488-5525-5ddf-90f2-1072c0ce0c5e
Run examples
Execute the following command in Command Prompt to run one example provided as part of Spark installation (class SparkPi with param 10).
https://spark.apache.org/docs/latest/
%SPARK_HOME%\bin\run-example.cmd SparkPi 10
The output looks like the following: https://api.kontext.tech/resource/1618e2ea-fcbb-5c61-b352-a644ce5222ae
Spark context UI
As printed out, Spark context Web UI available at http://172.24.144.1:4040.
The following is a screenshot of the UI:
https://api.kontext.tech/resource/9186acfa-18db-58fc-b089-c7d695e5413f
Spark developer tools
Refer to the following page if you are interested in any Spark developer tools.