관리 메뉴

꿈꾸는 사람.

우분투 20.04에서 Apache Spark 설치 본문

AI

우분투 20.04에서 Apache Spark 설치

현무랑 니니 2021. 7. 28. 13:14
반응형

 

Apache Spark 개요

Apache Spark는 대용량 데이터 처리를 위한 통합 분석 엔진으로 배치와 스트리밍 데이터를 고성능으로 처리하고 자바, 스칼라, 파이썬, R 등의 다양한 프로그래밍 언어를 지원한다.

Apache Spark의 프로그래밍 모델은 클러스터에서 분산 작동하는 RDD(Resilient Distributed Datasets)를 기반으로 한다.

최근 데이터 사이언티스트들이 가장 선호하는 도구들 중의 하나이다.

설치 개요

0. update & upgrade
sudo apt update
sudo apt-get upgrade

1. Java, scala, git 설치
sudo apt install default-jdk scala git

2. Apache Spark 다운로드
wget https://downloads.apache.org/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz

3. Spark를 /opt에 풀기
sudo mkdir /opt/spark
sudo tar -xf spark*.tgz -C /opt/spark --strip-component 1
sudo chmod -R 777 /opt/spark

4. 시스템 경로에 Spark 폴더 추가하기
echo "export SPARK_HOME=/opt/spark" >> ~/.bashrc
echo "export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin" >> ~/.bashrc
echo "export PYSPARK_PYTHON=/usr/bin/python3" >> ~/.bashrc

Reload shell:
source ~/.bashrc

5. 우분투에서 Apache Spark master 시작하기
start-master.sh

포트를 변경하고 싶을 경우 --port 옵션 사용
--port 서비스가 사용할 포트 (기본: 마스터는 7077이고 worker는 임의의 값)
--webui-port 웹 UI가 사용할 포트 (기본: 마스터는 8080, worker는 8081)

start-master.sh --port 7072 --webui-port 8082

6. 웹 인터페이스에서 Spark master 접속하기
브라우저 실행하고 http://127.0.0.1:8080에 접속하기

7. 워커 스크립트 실행
문법: start-worker.sh spark://hostname:port

명령: start-worker.sh spark://ubuntu:7077

메모리/램 사용량 변경해서 실행하려면
stop-worker.sh
start-worker.sh -m 212M spark://ubuntu:7077

8. Spark shell 사용
spark-shell

상세 절차

0. update & upgrade

ninipa@ninipa-VirtualBox:~$ sudo apt update
[sudo] ninipa의 암호: 
기존:1 https://download.docker.com/linux/ubuntu focal InRelease
기존:2 http://dl.google.com/linux/chrome/deb stable InRelease                  
기존:3 http://kr.archive.ubuntu.com/ubuntu focal InRelease                     
받기:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]    
받기:5 http://kr.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]   
받기:6 http://kr.archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]
받기:7 http://kr.archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [5,800 B]
받기:8 http://kr.archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [4,736 B]
받기:9 http://kr.archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [2,068 B]
내려받기 341 k바이트, 소요시간 3초 (122 k바이트/초)
패키지 목록을 읽는 중입니다... 완료
의존성 트리를 만드는 중입니다       
상태 정보를 읽는 중입니다... 완료
패키지 16이(가) 업그레이드되었습니다. 'apt list --upgradable'를 실행하여 확인해 보십시오.
ninipa@ninipa-VirtualBox:~$ sudo apt upgrade
패키지 목록을 읽는 중입니다... 완료
의존성 트리를 만드는 중입니다       
상태 정보를 읽는 중입니다... 완료
업그레이드를 계산하는 중입니다... 완료
다음 패키지가 자동으로 설치되었지만 더 이상 필요하지 않습니다:
  linux-headers-5.8.0-43-generic linux-hwe-5.8-headers-5.8.0-43
  linux-image-5.8.0-43-generic linux-modules-5.8.0-43-generic
  linux-modules-extra-5.8.0-43-generic
'sudo apt autoremove'를 이용하여 제거하십시오.
다음 새 패키지를 설치할 것입니다:
  distro-info
다음 패키지를 업그레이드할 것입니다:
  containerd.io gdm3 gir1.2-gdm-1.0 google-chrome-stable libgdm1 libglib2.0-0
  libglib2.0-bin libglib2.0-data libgupnp-1.2-0 linux-base networkd-dispatcher
  python3-distupgrade ubuntu-advantage-tools ubuntu-drivers-common
  ubuntu-release-upgrader-core ubuntu-release-upgrader-gtk
16개 업그레이드, 1개 새로 설치, 0개 제거 및 0개 업그레이드 안 함.
114 M바이트 아카이브를 받아야 합니다.
이 작업 후 9,369 k바이트의 디스크 공간이 비워집니다.
계속 하시겠습니까? [Y/n] y
받기:1 https://download.docker.com/linux/ubuntu focal/stable amd64 containerd.io amd64 1.4.8-1 [24.7 MB]
받기:2 http://dl.google.com/linux/chrome/deb stable/main amd64 google-chrome-stable amd64 92.0.4515.107-1 [86.2 MB]
받기:3 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 libglib2.0-data all 2.64.6-1~ubuntu20.04.4 [6,052 B]
받기:4 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 libglib2.0-bin amd64 2.64.6-1~ubuntu20.04.4 [72.6 kB]
받기:5 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 libglib2.0-0 amd64 2.64.6-1~ubuntu20.04.4 [1,287 kB]
받기:6 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 ubuntu-drivers-common amd64 1:0.9.0~0.20.04.1 [53.0 kB]
받기:7 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 networkd-dispatcher all 2.1-2~ubuntu20.04.1 [14.6 kB]
받기:8 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 distro-info amd64 0.23ubuntu1 [17.1 kB]
받기:9 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 ubuntu-advantage-tools amd64 27.2.1~20.04.1 [844 kB]
받기:10 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 ubuntu-release-upgrader-gtk all 1:20.04.35 [9,364 B]
받기:11 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 ubuntu-release-upgrader-core all 1:20.04.35 [23.9 kB]
받기:12 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-distupgrade all 1:20.04.35 [104 kB]
받기:13 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 gdm3 amd64 3.36.3-0ubuntu0.20.04.4 [261 kB]
받기:14 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 libgdm1 amd64 3.36.3-0ubuntu0.20.04.4 [63.6 kB]
받기:15 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 gir1.2-gdm-1.0 amd64 3.36.3-0ubuntu0.20.04.4 [9,584 B]
받기:16 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 libgupnp-1.2-0 amd64 1.2.4-0ubuntu1 [81.3 kB]
받기:17 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-base all 4.5ubuntu3.6 [17.8 kB]
내려받기 114 M바이트, 소요시간 31초 (3,692 k바이트/초)                         
패키지를 미리 설정하는 중입니다...
(데이터베이스 읽는중 ...현재 218865개의 파일과 디렉터리가 설치되어 있습니다.)
Preparing to unpack .../00-libglib2.0-data_2.64.6-1~ubuntu20.04.4_all.deb ...
Unpacking libglib2.0-data (2.64.6-1~ubuntu20.04.4) over (2.64.6-1~ubuntu20.04.3)
 ...
Preparing to unpack .../01-libglib2.0-bin_2.64.6-1~ubuntu20.04.4_amd64.deb ...
Unpacking libglib2.0-bin (2.64.6-1~ubuntu20.04.4) over (2.64.6-1~ubuntu20.04.3) 
...
Preparing to unpack .../02-libglib2.0-0_2.64.6-1~ubuntu20.04.4_amd64.deb ...
Unpacking libglib2.0-0:amd64 (2.64.6-1~ubuntu20.04.4) over (2.64.6-1~ubuntu20.04
.3) ...
Preparing to unpack .../03-google-chrome-stable_92.0.4515.107-1_amd64.deb ...
Unpacking google-chrome-stable (92.0.4515.107-1) over (91.0.4472.114-1) ...
Preparing to unpack .../04-ubuntu-drivers-common_1%3a0.9.0~0.20.04.1_amd64.deb .
..
Unpacking ubuntu-drivers-common (1:0.9.0~0.20.04.1) over (1:0.8.6.5~0.20.04.1) .
..
Preparing to unpack .../05-networkd-dispatcher_2.1-2~ubuntu20.04.1_all.deb ...
Unpacking networkd-dispatcher (2.1-2~ubuntu20.04.1) over (2.0.1-1) ...
Selecting previously unselected package distro-info.
Preparing to unpack .../06-distro-info_0.23ubuntu1_amd64.deb ...
Unpacking distro-info (0.23ubuntu1) ...
Preparing to unpack .../07-ubuntu-advantage-tools_27.2.1~20.04.1_amd64.deb ...
Unpacking ubuntu-advantage-tools (27.2.1~20.04.1) over (20.3) ...
Preparing to unpack .../08-ubuntu-release-upgrader-gtk_1%3a20.04.35_all.deb ...
Unpacking ubuntu-release-upgrader-gtk (1:20.04.35) over (1:20.04.33) ...
Preparing to unpack .../09-ubuntu-release-upgrader-core_1%3a20.04.35_all.deb ...
Unpacking ubuntu-release-upgrader-core (1:20.04.35) over (1:20.04.33) ...
Preparing to unpack .../10-python3-distupgrade_1%3a20.04.35_all.deb ...
Unpacking python3-distupgrade (1:20.04.35) over (1:20.04.33) ...
Preparing to unpack .../11-containerd.io_1.4.8-1_amd64.deb ...
Unpacking containerd.io (1.4.8-1) over (1.4.6-1) ...
Preparing to unpack .../12-gdm3_3.36.3-0ubuntu0.20.04.4_amd64.deb ...
Unpacking gdm3 (3.36.3-0ubuntu0.20.04.4) over (3.36.3-0ubuntu0.20.04.3) ...
Preparing to unpack .../13-libgdm1_3.36.3-0ubuntu0.20.04.4_amd64.deb ...
Unpacking libgdm1 (3.36.3-0ubuntu0.20.04.4) over (3.36.3-0ubuntu0.20.04.3) ...
Preparing to unpack .../14-gir1.2-gdm-1.0_3.36.3-0ubuntu0.20.04.4_amd64.deb ...
Unpacking gir1.2-gdm-1.0:amd64 (3.36.3-0ubuntu0.20.04.4) over (3.36.3-0ubuntu0.2
0.04.3) ...
Preparing to unpack .../15-libgupnp-1.2-0_1.2.4-0ubuntu1_amd64.deb ...
Unpacking libgupnp-1.2-0:amd64 (1.2.4-0ubuntu1) over (1.2.3-0ubuntu0.20.04.2) ..
.
Preparing to unpack .../16-linux-base_4.5ubuntu3.6_all.deb ...
Unpacking linux-base (4.5ubuntu3.6) over (4.5ubuntu3.5) ...
networkd-dispatcher (2.1-2~ubuntu20.04.1) 설정하는 중입니다 ...
distro-info (0.23ubuntu1) 설정하는 중입니다 ...
linux-base (4.5ubuntu3.6) 설정하는 중입니다 ...
ubuntu-drivers-common (1:0.9.0~0.20.04.1) 설정하는 중입니다 ...
libglib2.0-0:amd64 (2.64.6-1~ubuntu20.04.4) 설정하는 중입니다 ...
python3-distupgrade (1:20.04.35) 설정하는 중입니다 ...
libglib2.0-data (2.64.6-1~ubuntu20.04.4) 설정하는 중입니다 ...
containerd.io (1.4.8-1) 설정하는 중입니다 ...
ubuntu-release-upgrader-core (1:20.04.35) 설정하는 중입니다 ...
ubuntu-release-upgrader-gtk (1:20.04.35) 설정하는 중입니다 ...
ubuntu-advantage-tools (27.2.1~20.04.1) 설정하는 중입니다 ...
새 버전의 설정 파일 /etc/ubuntu-advantage/uaclient.conf 설치하는 중입니다 ...
Created symlink /etc/systemd/system/multi-user.target.wants/ua-reboot-cmds.servi
ce → /lib/systemd/system/ua-reboot-cmds.service.
Created symlink /etc/systemd/system/timers.target.wants/ua-messaging.timer → /li
b/systemd/system/ua-messaging.timer.
libgupnp-1.2-0:amd64 (1.2.4-0ubuntu1) 설정하는 중입니다 ...
google-chrome-stable (92.0.4515.107-1) 설정하는 중입니다 ...
libgdm1 (3.36.3-0ubuntu0.20.04.4) 설정하는 중입니다 ...
libglib2.0-bin (2.64.6-1~ubuntu20.04.4) 설정하는 중입니다 ...
gir1.2-gdm-1.0:amd64 (3.36.3-0ubuntu0.20.04.4) 설정하는 중입니다 ...
gdm3 (3.36.3-0ubuntu0.20.04.4) 설정하는 중입니다 ...
Processing triggers for mime-support (3.64ubuntu1) ...
Processing triggers for gnome-menus (3.36.0-1ubuntu1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
Processing triggers for systemd (245.4-4ubuntu3.11) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for dbus (1.12.16-2ubuntu2.1) ...
Processing triggers for desktop-file-utils (0.24-1ubuntu3) ...
ninipa@ninipa-VirtualBox:~$ 

1. java, scala, git 설치

Apache Spark는 필수로 java가 필요하고 scala는 프로그래밍을 위해 추가한다.

ninipa@ninipa-VirtualBox:~$ sudo apt install default-jdk scala
[sudo] ninipa의 암호: 
패키지 목록을 읽는 중입니다... 완료
의존성 트리를 만드는 중입니다       
상태 정보를 읽는 중입니다... 완료
다음 패키지가 자동으로 설치되었지만 더 이상 필요하지 않습니다:
  linux-headers-5.8.0-43-generic linux-hwe-5.8-headers-5.8.0-43
  linux-image-5.8.0-43-generic linux-modules-5.8.0-43-generic
  linux-modules-extra-5.8.0-43-generic
'sudo apt autoremove'를 이용하여 제거하십시오.
다음의 추가 패키지가 설치될 것입니다 :
  ca-certificates-java default-jdk-headless default-jre default-jre-headless
  fonts-dejavu-extra java-common libatk-wrapper-java libatk-wrapper-java-jni
  libhawtjni-runtime-java libice-dev libjansi-java libjansi-native-java
  libjline2-java libpthread-stubs0-dev libsm-dev libx11-dev libxau-dev
  libxcb1-dev libxdmcp-dev libxt-dev openjdk-11-jdk openjdk-11-jdk-headless
  openjdk-11-jre openjdk-11-jre-headless scala-library
  scala-parser-combinators scala-xml x11proto-core-dev x11proto-dev
  xorg-sgml-doctools xtrans-dev
제안하는 패키지:
  libice-doc libsm-doc libx11-doc libxcb-doc libxt-doc openjdk-11-demo
  openjdk-11-source visualvm fonts-ipafont-gothic fonts-ipafont-mincho
  fonts-wqy-microhei | fonts-wqy-zenhei scala-doc
다음 새 패키지를 설치할 것입니다:
  ca-certificates-java default-jdk default-jdk-headless default-jre
  default-jre-headless fonts-dejavu-extra java-common libatk-wrapper-java
  libatk-wrapper-java-jni libhawtjni-runtime-java libice-dev libjansi-java
  libjansi-native-java libjline2-java libpthread-stubs0-dev libsm-dev
  libx11-dev libxau-dev libxcb1-dev libxdmcp-dev libxt-dev openjdk-11-jdk
  openjdk-11-jdk-headless openjdk-11-jre openjdk-11-jre-headless scala
  scala-library scala-parser-combinators scala-xml x11proto-core-dev
  x11proto-dev xorg-sgml-doctools xtrans-dev
0개 업그레이드, 33개 새로 설치, 0개 제거 및 0개 업그레이드 안 함.
291 M바이트 아카이브를 받아야 합니다.
이 작업 후 449 M바이트의 디스크 공간을 더 사용하게 됩니다.
계속 하시겠습니까? [Y/n] y
받기:1 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 java-common all 0.72 [6,816 B]
받기:2 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 openjdk-11-jre-headless amd64 11.0.11+9-0ubuntu2~20.04 [37.2 MB]
받기:3 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 default-jre-headless amd64 2:1.11-72 [3,192 B]
받기:4 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 ca-certificates-java all 20190405ubuntu1 [12.2 kB]
받기:5 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 openjdk-11-jre amd64 11.0.11+9-0ubuntu2~20.04 [174 kB]
받기:6 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 default-jre amd64 2:1.11-72 [1,084 B]
받기:7 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 openjdk-11-jdk-headless amd64 11.0.11+9-0ubuntu2~20.04 [223 MB]
받기:8 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 default-jdk-headless amd64 2:1.11-72 [1,140 B]
받기:9 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 openjdk-11-jdk amd64 11.0.11+9-0ubuntu2~20.04 [1,442 kB]
받기:10 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 default-jdk amd64 2:1.11-72 [1,096 B]
받기:11 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 fonts-dejavu-extra all 2.37-1 [1,953 kB]
받기:12 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 libatk-wrapper-java all 0.37.1-1 [53.0 kB]
받기:13 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 libatk-wrapper-java-jni amd64 0.37.1-1 [45.1 kB]
받기:14 http://kr.archive.ubuntu.com/ubuntu focal/universe amd64 libhawtjni-runtime-java all 1.17-1 [28.8 kB]
받기:15 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 xorg-sgml-doctools all 1:1.11-1 [12.9 kB]
받기:16 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 x11proto-dev all 2019.2-1ubuntu1 [594 kB]
받기:17 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 x11proto-core-dev all 2019.2-1ubuntu1 [2,620 B]
받기:18 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 libice-dev amd64 2:1.0.10-0ubuntu1 [47.8 kB]
받기:19 http://kr.archive.ubuntu.com/ubuntu focal/universe amd64 libjansi-native-java all 1.8-1 [23.8 kB]
받기:20 http://kr.archive.ubuntu.com/ubuntu focal/universe amd64 libjansi-java all 1.18-1 [56.8 kB]
받기:21 http://kr.archive.ubuntu.com/ubuntu focal/universe amd64 libjline2-java all 2.14.6-3 [150 kB]
받기:22 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 libpthread-stubs0-dev amd64 0.4-1 [5,384 B]
받기:23 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 libsm-dev amd64 2:1.2.3-1 [17.0 kB]
받기:24 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 libxau-dev amd64 1:1.0.9-0ubuntu1 [9,552 B]
받기:25 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 libxdmcp-dev amd64 1:1.1.3-0ubuntu1 [25.3 kB]
받기:26 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 xtrans-dev all 1.4.0-1 [68.9 kB]
받기:27 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 libxcb1-dev amd64 1.14-2 [80.5 kB]
받기:28 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 libx11-dev amd64 2:1.6.9-2ubuntu1.2 [647 kB]
받기:29 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 libxt-dev amd64 1:1.1.5-1 [395 kB]
받기:30 http://kr.archive.ubuntu.com/ubuntu focal/universe amd64 scala-library all 2.11.12-4 [9,583 kB]
받기:31 http://kr.archive.ubuntu.com/ubuntu focal/universe amd64 scala-parser-combinators all 1.0.3-3 [355 kB]
받기:32 http://kr.archive.ubuntu.com/ubuntu focal/universe amd64 scala-xml all 1.0.3-3 [601 kB]
받기:33 http://kr.archive.ubuntu.com/ubuntu focal/universe amd64 scala all 2.11.12-4 [14.2 MB]
내려받기 291 M바이트, 소요시간 16분 1초 (303 k바이트/초)                       
패키지에서 템플릿을 추출하는 중: 100%
Selecting previously unselected package java-common.
(데이터베이스 읽는중 ...현재 218891개의 파일과 디렉터리가 설치되어 있습니다.)
Preparing to unpack .../00-java-common_0.72_all.deb ...
Unpacking java-common (0.72) ...
Selecting previously unselected package openjdk-11-jre-headless:amd64.
Preparing to unpack .../01-openjdk-11-jre-headless_11.0.11+9-0ubuntu2~20.04_amd6
4.deb ...
Unpacking openjdk-11-jre-headless:amd64 (11.0.11+9-0ubuntu2~20.04) ...
Selecting previously unselected package default-jre-headless.
Preparing to unpack .../02-default-jre-headless_2%3a1.11-72_amd64.deb ...
Unpacking default-jre-headless (2:1.11-72) ...
Selecting previously unselected package ca-certificates-java.
Preparing to unpack .../03-ca-certificates-java_20190405ubuntu1_all.deb ...
Unpacking ca-certificates-java (20190405ubuntu1) ...
Selecting previously unselected package openjdk-11-jre:amd64.
Preparing to unpack .../04-openjdk-11-jre_11.0.11+9-0ubuntu2~20.04_amd64.deb ...
Unpacking openjdk-11-jre:amd64 (11.0.11+9-0ubuntu2~20.04) ...
Selecting previously unselected package default-jre.
Preparing to unpack .../05-default-jre_2%3a1.11-72_amd64.deb ...
Unpacking default-jre (2:1.11-72) ...
Selecting previously unselected package openjdk-11-jdk-headless:amd64.
Preparing to unpack .../06-openjdk-11-jdk-headless_11.0.11+9-0ubuntu2~20.04_amd6
4.deb ...
Unpacking openjdk-11-jdk-headless:amd64 (11.0.11+9-0ubuntu2~20.04) ...
Selecting previously unselected package default-jdk-headless.
Preparing to unpack .../07-default-jdk-headless_2%3a1.11-72_amd64.deb ...
Unpacking default-jdk-headless (2:1.11-72) ...
Selecting previously unselected package openjdk-11-jdk:amd64.
Preparing to unpack .../08-openjdk-11-jdk_11.0.11+9-0ubuntu2~20.04_amd64.deb ...
Unpacking openjdk-11-jdk:amd64 (11.0.11+9-0ubuntu2~20.04) ...
Selecting previously unselected package default-jdk.
Preparing to unpack .../09-default-jdk_2%3a1.11-72_amd64.deb ...
Unpacking default-jdk (2:1.11-72) ...
Selecting previously unselected package fonts-dejavu-extra.
Preparing to unpack .../10-fonts-dejavu-extra_2.37-1_all.deb ...
Unpacking fonts-dejavu-extra (2.37-1) ...
Selecting previously unselected package libatk-wrapper-java.
Preparing to unpack .../11-libatk-wrapper-java_0.37.1-1_all.deb ...
Unpacking libatk-wrapper-java (0.37.1-1) ...
Selecting previously unselected package libatk-wrapper-java-jni:amd64.
Preparing to unpack .../12-libatk-wrapper-java-jni_0.37.1-1_amd64.deb ...
Unpacking libatk-wrapper-java-jni:amd64 (0.37.1-1) ...
Selecting previously unselected package libhawtjni-runtime-java.
Preparing to unpack .../13-libhawtjni-runtime-java_1.17-1_all.deb ...
Unpacking libhawtjni-runtime-java (1.17-1) ...
Selecting previously unselected package xorg-sgml-doctools.
Preparing to unpack .../14-xorg-sgml-doctools_1%3a1.11-1_all.deb ...
Unpacking xorg-sgml-doctools (1:1.11-1) ...
Selecting previously unselected package x11proto-dev.
Preparing to unpack .../15-x11proto-dev_2019.2-1ubuntu1_all.deb ...
Unpacking x11proto-dev (2019.2-1ubuntu1) ...
Selecting previously unselected package x11proto-core-dev.
Preparing to unpack .../16-x11proto-core-dev_2019.2-1ubuntu1_all.deb ...
Unpacking x11proto-core-dev (2019.2-1ubuntu1) ...
Selecting previously unselected package libice-dev:amd64.
Preparing to unpack .../17-libice-dev_2%3a1.0.10-0ubuntu1_amd64.deb ...
Unpacking libice-dev:amd64 (2:1.0.10-0ubuntu1) ...
Selecting previously unselected package libjansi-native-java.
Preparing to unpack .../18-libjansi-native-java_1.8-1_all.deb ...
Unpacking libjansi-native-java (1.8-1) ...
Selecting previously unselected package libjansi-java.
Preparing to unpack .../19-libjansi-java_1.18-1_all.deb ...
Unpacking libjansi-java (1.18-1) ...
Selecting previously unselected package libjline2-java.
Preparing to unpack .../20-libjline2-java_2.14.6-3_all.deb ...
Unpacking libjline2-java (2.14.6-3) ...
Selecting previously unselected package libpthread-stubs0-dev:amd64.
Preparing to unpack .../21-libpthread-stubs0-dev_0.4-1_amd64.deb ...
Unpacking libpthread-stubs0-dev:amd64 (0.4-1) ...
Selecting previously unselected package libsm-dev:amd64.
Preparing to unpack .../22-libsm-dev_2%3a1.2.3-1_amd64.deb ...
Unpacking libsm-dev:amd64 (2:1.2.3-1) ...
Selecting previously unselected package libxau-dev:amd64.
Preparing to unpack .../23-libxau-dev_1%3a1.0.9-0ubuntu1_amd64.deb ...
Unpacking libxau-dev:amd64 (1:1.0.9-0ubuntu1) ...
Selecting previously unselected package libxdmcp-dev:amd64.
Preparing to unpack .../24-libxdmcp-dev_1%3a1.1.3-0ubuntu1_amd64.deb ...
Unpacking libxdmcp-dev:amd64 (1:1.1.3-0ubuntu1) ...
Selecting previously unselected package xtrans-dev.
Preparing to unpack .../25-xtrans-dev_1.4.0-1_all.deb ...
Unpacking xtrans-dev (1.4.0-1) ...
Selecting previously unselected package libxcb1-dev:amd64.
Preparing to unpack .../26-libxcb1-dev_1.14-2_amd64.deb ...
Unpacking libxcb1-dev:amd64 (1.14-2) ...
Selecting previously unselected package libx11-dev:amd64.
Preparing to unpack .../27-libx11-dev_2%3a1.6.9-2ubuntu1.2_amd64.deb ...
Unpacking libx11-dev:amd64 (2:1.6.9-2ubuntu1.2) ...
Selecting previously unselected package libxt-dev:amd64.
Preparing to unpack .../28-libxt-dev_1%3a1.1.5-1_amd64.deb ...
Unpacking libxt-dev:amd64 (1:1.1.5-1) ...
Selecting previously unselected package scala-library.
Preparing to unpack .../29-scala-library_2.11.12-4_all.deb ...
Unpacking scala-library (2.11.12-4) ...
Selecting previously unselected package scala-parser-combinators.
Preparing to unpack .../30-scala-parser-combinators_1.0.3-3_all.deb ...
Unpacking scala-parser-combinators (1.0.3-3) ...
Selecting previously unselected package scala-xml.
Preparing to unpack .../31-scala-xml_1.0.3-3_all.deb ...
Unpacking scala-xml (1.0.3-3) ...
Selecting previously unselected package scala.
Preparing to unpack .../32-scala_2.11.12-4_all.deb ...
Unpacking scala (2.11.12-4) ...
java-common (0.72) 설정하는 중입니다 ...
scala-library (2.11.12-4) 설정하는 중입니다 ...
scala-xml (1.0.3-3) 설정하는 중입니다 ...
libpthread-stubs0-dev:amd64 (0.4-1) 설정하는 중입니다 ...
xtrans-dev (1.4.0-1) 설정하는 중입니다 ...
scala-parser-combinators (1.0.3-3) 설정하는 중입니다 ...
fonts-dejavu-extra (2.37-1) 설정하는 중입니다 ...
xorg-sgml-doctools (1:1.11-1) 설정하는 중입니다 ...
libatk-wrapper-java (0.37.1-1) 설정하는 중입니다 ...
libhawtjni-runtime-java (1.17-1) 설정하는 중입니다 ...
libatk-wrapper-java-jni:amd64 (0.37.1-1) 설정하는 중입니다 ...
libjansi-native-java (1.8-1) 설정하는 중입니다 ...
libjansi-java (1.18-1) 설정하는 중입니다 ...
libjline2-java (2.14.6-3) 설정하는 중입니다 ...
default-jre-headless (2:1.11-72) 설정하는 중입니다 ...
openjdk-11-jre-headless:amd64 (11.0.11+9-0ubuntu2~20.04) 설정하는 중입니다 ...
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/java to provid
e /usr/bin/java (java) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jjs to provide
 /usr/bin/jjs (jjs) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/keytool to pro
vide /usr/bin/keytool (keytool) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/rmid to provid
e /usr/bin/rmid (rmid) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/rmiregistry to
 provide /usr/bin/rmiregistry (rmiregistry) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/pack200 to pro
vide /usr/bin/pack200 (pack200) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/unpack200 to p
rovide /usr/bin/unpack200 (unpack200) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/lib/jexec to provi
de /usr/bin/jexec (jexec) in auto mode
openjdk-11-jre:amd64 (11.0.11+9-0ubuntu2~20.04) 설정하는 중입니다 ...
openjdk-11-jdk-headless:amd64 (11.0.11+9-0ubuntu2~20.04) 설정하는 중입니다 ...
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jar to provide
 /usr/bin/jar (jar) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jarsigner to p
rovide /usr/bin/jarsigner (jarsigner) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/javac to provi
de /usr/bin/javac (javac) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/javadoc to pro
vide /usr/bin/javadoc (javadoc) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/javap to provi
de /usr/bin/javap (javap) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jcmd to provid
e /usr/bin/jcmd (jcmd) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jdb to provide
 /usr/bin/jdb (jdb) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jdeprscan to p
rovide /usr/bin/jdeprscan (jdeprscan) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jdeps to provi
de /usr/bin/jdeps (jdeps) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jfr to provide
 /usr/bin/jfr (jfr) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jimage to prov
ide /usr/bin/jimage (jimage) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jinfo to provi
de /usr/bin/jinfo (jinfo) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jlink to provi
de /usr/bin/jlink (jlink) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jmap to provid
e /usr/bin/jmap (jmap) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jmod to provid
e /usr/bin/jmod (jmod) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jps to provide
 /usr/bin/jps (jps) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jrunscript to 
provide /usr/bin/jrunscript (jrunscript) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jshell to prov
ide /usr/bin/jshell (jshell) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jstack to prov
ide /usr/bin/jstack (jstack) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jstat to provi
de /usr/bin/jstat (jstat) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jstatd to prov
ide /usr/bin/jstatd (jstatd) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/rmic to provid
e /usr/bin/rmic (rmic) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/serialver to p
rovide /usr/bin/serialver (serialver) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jaotc to provi
de /usr/bin/jaotc (jaotc) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jhsdb to provi
de /usr/bin/jhsdb (jhsdb) in auto mode
default-jre (2:1.11-72) 설정하는 중입니다 ...
default-jdk-headless (2:1.11-72) 설정하는 중입니다 ...
openjdk-11-jdk:amd64 (11.0.11+9-0ubuntu2~20.04) 설정하는 중입니다 ...
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jconsole to pr
ovide /usr/bin/jconsole (jconsole) in auto mode
ca-certificates-java (20190405ubuntu1) 설정하는 중입니다 ...
head: 읽기를 위해 '/etc/ssl/certs/java/cacerts'을(를) 열 수 없음: 그런 파일이나 
디렉터리가 없습니다
Adding debian:Microsoft_RSA_Root_Certificate_Authority_2017.pem
Adding debian:Actalis_Authentication_Root_CA.pem
Adding debian:GlobalSign_ECC_Root_CA_-_R5.pem
Adding debian:DigiCert_Global_Root_CA.pem
Adding debian:SecureSign_RootCA11.pem
Adding debian:QuoVadis_Root_CA_2.pem
Adding debian:GlobalSign_ECC_Root_CA_-_R4.pem
Adding debian:Entrust_Root_Certification_Authority_-_G4.pem
Adding debian:COMODO_ECC_Certification_Authority.pem
Adding debian:Buypass_Class_3_Root_CA.pem
Adding debian:QuoVadis_Root_CA_1_G3.pem
Adding debian:VeriSign_Universal_Root_Certification_Authority.pem
Adding debian:Hongkong_Post_Root_CA_3.pem
Adding debian:GDCA_TrustAUTH_R5_ROOT.pem
Adding debian:GlobalSign_Root_CA.pem
Adding debian:certSIGN_Root_CA_G2.pem
Adding debian:Amazon_Root_CA_3.pem
Adding debian:AffirmTrust_Premium.pem
Adding debian:CFCA_EV_ROOT.pem
Adding debian:SSL.com_Root_Certification_Authority_RSA.pem
Adding debian:QuoVadis_Root_CA_3.pem
Adding debian:Amazon_Root_CA_1.pem
Adding debian:SZAFIR_ROOT_CA2.pem
Adding debian:Entrust_Root_Certification_Authority_-_G2.pem
Adding debian:SecureTrust_CA.pem
Adding debian:Trustwave_Global_ECC_P256_Certification_Authority.pem
Adding debian:Certum_Trusted_Network_CA.pem
Adding debian:Starfield_Services_Root_Certificate_Authority_-_G2.pem
Adding debian:Staat_der_Nederlanden_EV_Root_CA.pem
Adding debian:e-Szigno_Root_CA_2017.pem
Adding debian:Security_Communication_Root_CA.pem
Adding debian:GlobalSign_Root_CA_-_R3.pem
Adding debian:Certigna_Root_CA.pem
Adding debian:USERTrust_RSA_Certification_Authority.pem
Adding debian:DigiCert_High_Assurance_EV_Root_CA.pem
Adding debian:SSL.com_EV_Root_Certification_Authority_ECC.pem
Adding debian:Trustis_FPS_Root_CA.pem
Adding debian:Secure_Global_CA.pem
Adding debian:TWCA_Global_Root_CA.pem
Adding debian:OISTE_WISeKey_Global_Root_GC_CA.pem
Adding debian:Comodo_AAA_Services_root.pem
Adding debian:Buypass_Class_2_Root_CA.pem
Adding debian:Go_Daddy_Root_Certificate_Authority_-_G2.pem
Adding debian:DigiCert_Assured_ID_Root_CA.pem
Adding debian:DigiCert_Assured_ID_Root_G3.pem
Adding debian:QuoVadis_Root_CA_3_G3.pem
Adding debian:TrustCor_RootCert_CA-2.pem
Adding debian:IdenTrust_Commercial_Root_CA_1.pem
Adding debian:Staat_der_Nederlanden_Root_CA_-_G3.pem
Adding debian:TUBITAK_Kamu_SM_SSL_Kok_Sertifikasi_-_Surum_1.pem
Adding debian:D-TRUST_Root_Class_3_CA_2_EV_2009.pem
Adding debian:AffirmTrust_Networking.pem
Adding debian:UCA_Extended_Validation_Root.pem
Adding debian:emSign_ECC_Root_CA_-_G3.pem
Adding debian:GTS_Root_R3.pem
Adding debian:AC_RAIZ_FNMT-RCM.pem
Adding debian:Amazon_Root_CA_4.pem
Adding debian:E-Tugra_Certification_Authority.pem
Adding debian:Chambers_of_Commerce_Root_-_2008.pem
Adding debian:Microsoft_ECC_Root_Certificate_Authority_2017.pem
Adding debian:GeoTrust_Primary_Certification_Authority_-_G2.pem
Adding debian:Network_Solutions_Certificate_Authority.pem
Adding debian:TrustCor_RootCert_CA-1.pem
Adding debian:Cybertrust_Global_Root.pem
Adding debian:Certigna.pem
Adding debian:QuoVadis_Root_CA_2_G3.pem
Adding debian:GTS_Root_R4.pem
Adding debian:ACCVRAIZ1.pem
Adding debian:Starfield_Class_2_CA.pem
Adding debian:T-TeleSec_GlobalRoot_Class_2.pem
Adding debian:Go_Daddy_Class_2_CA.pem
Adding debian:DigiCert_Trusted_Root_G4.pem
Adding debian:Sonera_Class_2_Root_CA.pem
Adding debian:NetLock_Arany_=Class_Gold=_Főtanúsítvány.pem
Adding debian:T-TeleSec_GlobalRoot_Class_3.pem
Adding debian:OISTE_WISeKey_Global_Root_GB_CA.pem
Adding debian:Entrust_Root_Certification_Authority_-_EC1.pem
Adding debian:emSign_Root_CA_-_G1.pem
Adding debian:Atos_TrustedRoot_2011.pem
Adding debian:DigiCert_Global_Root_G2.pem
Adding debian:Hellenic_Academic_and_Research_Institutions_ECC_RootCA_2015.pem
Adding debian:emSign_Root_CA_-_C1.pem
Adding debian:GTS_Root_R1.pem
Adding debian:Baltimore_CyberTrust_Root.pem
Adding debian:AffirmTrust_Commercial.pem
Adding debian:Trustwave_Global_ECC_P384_Certification_Authority.pem
Adding debian:Hellenic_Academic_and_Research_Institutions_RootCA_2011.pem
Adding debian:SSL.com_Root_Certification_Authority_ECC.pem
Adding debian:Microsec_e-Szigno_Root_CA_2009.pem
Adding debian:USERTrust_ECC_Certification_Authority.pem
Adding debian:AffirmTrust_Premium_ECC.pem
Adding debian:COMODO_Certification_Authority.pem
Adding debian:SwissSign_Silver_CA_-_G2.pem
Adding debian:QuoVadis_Root_CA.pem
Adding debian:GlobalSign_Root_CA_-_R6.pem
Adding debian:Hellenic_Academic_and_Research_Institutions_RootCA_2015.pem
Adding debian:Entrust.net_Premium_2048_Secure_Server_CA.pem
Adding debian:ePKI_Root_Certification_Authority.pem
Adding debian:Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.pem
Adding debian:ISRG_Root_X1.pem
Adding debian:emSign_ECC_Root_CA_-_C3.pem
Adding debian:Global_Chambersign_Root_-_2008.pem
Adding debian:NAVER_Global_Root_Certification_Authority.pem
Adding debian:XRamp_Global_CA_Root.pem
Adding debian:GTS_Root_R2.pem
Adding debian:DigiCert_Assured_ID_Root_G2.pem
Adding debian:GlobalSign_Root_CA_-_R2.pem
Adding debian:Trustwave_Global_Certification_Authority.pem
Adding debian:Security_Communication_RootCA2.pem
Adding debian:Hongkong_Post_Root_CA_1.pem
Adding debian:Certum_Trusted_Network_CA_2.pem
Adding debian:Izenpe.com.pem
Adding debian:Amazon_Root_CA_2.pem
Adding debian:D-TRUST_Root_Class_3_CA_2_2009.pem
Adding debian:IdenTrust_Public_Sector_Root_CA_1.pem
Adding debian:Entrust_Root_Certification_Authority.pem
Adding debian:certSIGN_ROOT_CA.pem
Adding debian:TrustCor_ECA-1.pem
Adding debian:EC-ACC.pem
Adding debian:CA_Disig_Root_R2.pem
Adding debian:COMODO_RSA_Certification_Authority.pem
Adding debian:DigiCert_Global_Root_G3.pem
Adding debian:DST_Root_CA_X3.pem
Adding debian:TWCA_Root_Certification_Authority.pem
Adding debian:Starfield_Root_Certificate_Authority_-_G2.pem
Adding debian:SSL.com_EV_Root_Certification_Authority_RSA_R2.pem
Adding debian:ssl-cert-snakeoil.pem
Adding debian:TeliaSonera_Root_CA_v1.pem
Adding debian:UCA_Global_G2_Root.pem
Adding debian:SwissSign_Gold_CA_-_G2.pem
done.
scala (2.11.12-4) 설정하는 중입니다 ...
update-alternatives: using /usr/share/scala-2.11/bin/scala to provide /usr/bin/s
cala (scala) in auto mode
default-jdk (2:1.11-72) 설정하는 중입니다 ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for ca-certificates (20210119~20.04.1) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

done.
done.
Processing triggers for sgml-base (1.29.1) ...
x11proto-dev (2019.2-1ubuntu1) 설정하는 중입니다 ...
Processing triggers for fontconfig (2.13.1-2ubuntu3) ...
Processing triggers for desktop-file-utils (0.24-1ubuntu3) ...
Processing triggers for mime-support (3.64ubuntu1) ...
libxau-dev:amd64 (1:1.0.9-0ubuntu1) 설정하는 중입니다 ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
libice-dev:amd64 (2:1.0.10-0ubuntu1) 설정하는 중입니다 ...
Processing triggers for gnome-menus (3.36.0-1ubuntu1) ...
libsm-dev:amd64 (2:1.2.3-1) 설정하는 중입니다 ...
libxdmcp-dev:amd64 (1:1.1.3-0ubuntu1) 설정하는 중입니다 ...
x11proto-core-dev (2019.2-1ubuntu1) 설정하는 중입니다 ...
libxcb1-dev:amd64 (1.14-2) 설정하는 중입니다 ...
libx11-dev:amd64 (2:1.6.9-2ubuntu1.2) 설정하는 중입니다 ...
libxt-dev:amd64 (1:1.1.5-1) 설정하는 중입니다 ...
ninipa@ninipa-VirtualBox:~$ 
ninipa@ninipa-VirtualBox:~$ java --version
openjdk 11.0.11 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing)
ninipa@ninipa-VirtualBox:~$ scala -version
Scala code runner version 2.11.12 -- Copyright 2002-2017, LAMP/EPFL
ninipa@ninipa-VirtualBox:~$ 

2. Apache Spark 다운로드

ninipa@ninipa-VirtualBox:~$ wget https://downloads.apache.org/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz
--2021-07-27 20:05:04--  https://downloads.apache.org/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz
downloads.apache.org (downloads.apache.org)을(를) 해석하는 중... 135.181.209.10, 88.99.95.219, 135.181.214.104, ...
접속 downloads.apache.org (downloads.apache.org)|135.181.209.10|:443... 접속됨.
HTTP 요청을 전송했습니다. 응답을 기다리는 중입니다... 200 OK
길이: 228834641 (218M) [application/x-gzip]
다음 위치에 저장: `spark-3.1.2-bin-hadoop3.2.tgz'

spark-3.1.2-bin-had   3%[                    ]   7.54M  2.40MB/s    남은 시간 88park-3.1.2-bin-hado   4%[                    ]   8.74M  2.42MB/s    남은 시간 88ark-3.1.2-bin-hadoo   4%[                    ]   9.40M  2.45MB/s    남은 시간 88rk-3.1.2-bin-hadoop   4%[                    ]  10.34M  2.53MB/s    남은 시간 88k-3.1.2-bin-hadoop3   5%[>                   ]  12.38M  2.89MB/s    남은 시간 71-3.1.2-bin-hadoop3.   5%[>                   ]  13.04M  2.87MB/s    남은 시간 713.1.2-bin-hadoop3.2   6%[>                   ]  13.73M  2.76MB/s    남은 시간 71.1.2-bin-hadoop3.2.   6%[>                   ]  14.10M  2.73MB/s    남은 시간 711.2-bin-hadoop3.2.t   6%[>                   ]  14.59M  2.88MB/s    남은 시간 75.2-bin-hadoop3.2.tg   6%[>                   ]  15.12M  3.22MB/s    남은 시간 752-bin-hadoop3.2.tgz   7%[>                   ]  16.65M  3.61MB/s    남은 시간 75-bin-hadoop3.2.tgz    7%[>                   ]  16.91M  3.72MB/s    남은 시간 75bin-hadoop3.2.tgz     8%[>                   ]  19.15M  4.46MB/s    남은 시간 75in-hadoop3.2.tgz      8%[>                   ]  19.18M  4.08MB/s    남은 시간 69n-hadoop3.2.tgz       9%[>                   ]  20.59M  4.02MB/s    남은 시간 69-hadoop3.2.tgz       10%[=>                  ]  22.18M  3.68MB/s    남은 시간 69hadoop3.2.tgz        10%[=>                  ]  22.57M  3.80MB/s    남은 시간 69adoop3.2.tgz         10%[=>                  ]  23.63M  4.00MB/s    남은 시간 69doop3.2.tgz          10%[=>                  ]  23.91M  3.51MB/s    남은 시간 65oop3.2.tgz           11%[=>                  ]  24.41M  3.46MB/s    남은 시간 65op3.2.tgz            12%[=>                  ]  26.59M  3.54MB/s    남은 시간 65p3.2.tgz             12%[=>                  ]  27.65M  3.62MB/s    남은 시간 653.2.tgz              12%[=>                  ]  28.13M  3.46MB/s    남은 시간 62.2.tgz               13%[=>                  ]  29.74M  3.71MB/s    남은 시간 622.tgz                14%[=>                  ]  31.15M  3.95MB/s    남은 시간 62.tgz                 14%[=>                  ]  31.85M  3.99MB/s    남은 시간 62tgz                  14%[=>                  ]  32.63M  3.81MB/s    남은 시간 62gz                   15%[==>                 ]  33.30M  3.85MB/s    남은 시간 57z                    15%[==>                 ]  34.09M  3.56MB/s    남은 시간 57                     15%[==>                 ]  34.84M  3.95MB/s    남은 시간 57                  s  16%[==>                 ]  35.55M  3.76MB/s    남은 시간 57                 sp  16%[==>                 ]  36.26M  3.57MB/s    남은 시간 57                spa  16%[==>                 ]  37.01M  3.64MB/s    남은 시간 55               spar  17%[==>                 ]  37.76M  3.56MB/s    남은 시간 55              spark  17%[==>                 ]  38.48M  4.06MB/s    남은 시간 55             spark-  17%[==>                 ]  39.18M  3.65MB/s    남은 시간 55            spark-3  18%[==>                 ]  39.88M  3.17MB/s    남은 시간 56           spark-3.  19%[==>                 ]  42.05M  3.98MB/s    남은 시간 56          spark-3.1  19%[==>                 ]  42.46M  3.66MB/s    남은 시간 56         spark-3.1.  19%[==>                 ]  43.40M  3.51MB/s    남은 시간 56        spark-3.1.2  20%[===>                ]  44.24M  3.58MB/s    남은 시간 56       spark-3.1.2-  20%[===>                ]  44.63M  3.45MB/s    남은 시간 52      spark-3.1.2-b  20%[===>                ]  45.62M  3.52MB/s    남은 시간 52     spark-3.1.2-bi  21%[===>                ]  46.43M  3.55MB/s    남은 시간 52    spark-3.1.2-bin  21%[===>                ]  46.90M  3.47MB/s    남은 시간 52   spark-3.1.2-bin-  21%[===>                ]  47.91M  3.55MB/s    남은 시간 52  spark-3.1.2-bin-h  22%[===>                ]  48.57M  3.54MB/s    남은 시간 51 spark-3.1.2-bin-ha  22%[===>                ]  49.24M  3.52MB/s    남은 시간 51spark-3.1.2-bin-had  22%[===>                ]  50.13M  3.56MB/s    남은 시간 51park-3.1.2-bin-hado  23%[===>                ]  50.84M  3.55MB/s    남은 시간 51ark-3.1.2-bin-hadoo  23%[===>                ]  51.59M  3.56MB/s    남은 시간 51rk-3.1.2-bin-hadoop  23%[===>                ]  52.30M  3.56MB/s    남은 시간 49k-3.1.2-bin-hadoop3  24%[===>                ]  53.05M  3.93MB/s    남은 시간 49-3.1.2-bin-hadoop3.  24%[===>                ]  53.76M  3.60MB/s    남은 시간 493.1.2-bin-hadoop3.2  24%[===>                ]  54.48M  3.70MB/s    남은 시간 49.1.2-bin-hadoop3.2.  25%[====>               ]  55.26M  3.73MB/s    남은 시간 491.2-bin-hadoop3.2.t  25%[====>               ]  55.98M  3.61MB/s    남은 시간 48.2-bin-hadoop3.2.tg  25%[====>               ]  56.68M  3.71MB/s    남은 시간 482-bin-hadoop3.2.tgz  26%[====>               ]  57.40M  3.69MB/s    남은 시간 48-bin-hadoop3.2.tgz   26%[====>               ]  58.18M  3.59MB/s    남은 시간 48bin-hadoop3.2.tgz    26%[====>               ]  58.88M  3.68MB/s    남은 시간 48in-hadoop3.2.tgz     27%[====>               ]  59.60M  3.60MB/s    남은 시간 47n-hadoop3.2.tgz      27%[====>               ]  60.30M  3.55MB/s    남은 시간 47-hadoop3.2.tgz       27%[====>               ]  61.05M  3.55MB/s    남은 시간 47hadoop3.2.tgz        28%[====>               ]  61.80M  3.57MB/s    남은 시간 47adoop3.2.tgz         28%[====>               ]  62.52M  3.56MB/s    남은 시간 47doop3.2.tgz          28%[====>               ]  63.23M  3.54MB/s    남은 시간 46oop3.2.tgz           29%[====>               ]  63.98M  3.55MB/s    남은 시간 46op3.2.tgz            29%[====>               ]  64.73M  3.57MB/s    남은 시간 46p3.2.tgz             29%[====>               ]  65.45M  3.56MB/s    남은 시간 463.2.tgz              30%[=====>              ]  66.15M  3.55MB/s    남은 시간 46.2.tgz               30%[=====>              ]  66.90M  3.56MB/s    남은 시간 442.tgz                30%[=====>              ]  67.65M  3.56MB/s    남은 시간 44.tgz                 31%[=====>              ]  68.35M  3.56MB/s    남은 시간 44tgz                  31%[=====>              ]  69.07M  3.56MB/s    남은 시간 44gz                   31%[=====>              ]  69.82M  3.57MB/s    남은 시간 44z                    32%[=====>              ]  70.52M  3.56MB/s    남은 시간 43                     32%[=====>              ]  71.27M  3.56MB/s    남은 시간 43                  s  32%[=====>              ]  71.99M  3.56MB/s    남은 시간 43                 sp  33%[=====>              ]  72.74M  3.55MB/s    남은 시간 43                spa  33%[=====>              ]  73.45M  3.55MB/s    남은 시간 43               spar  33%[=====>              ]  74.20M  3.56MB/s    남은 시간 42              spark  34%[=====>              ]  74.91M  3.57MB/s    남은 시간 42             spark-  34%[=====>              ]  75.65M  3.58MB/s    남은 시간 42            spark-3  34%[=====>              ]  76.37M  3.55MB/s    남은 시간 42           spark-3.  35%[======>             ]  77.07M  3.55MB/s    남은 시간 42          spark-3.1  35%[======>             ]  77.82M  3.56MB/s    남은 시간 41         spark-3.1.  36%[======>             ]  78.57M  3.56MB/s    남은 시간 41        spark-3.1.2  36%[======>             ]  79.27M  3.55MB/s    남은 시간 41       spark-3.1.2-  36%[======>             ]  79.99M  3.55MB/s    남은 시간 41      spark-3.1.2-b  36%[======>             ]  80.74M  3.56MB/s    남은 시간 41     spark-3.1.2-bi  37%[======>             ]  81.45M  3.56MB/s    남은 시간 40    spark-3.1.2-bin  37%[======>             ]  82.18M  3.55MB/s    남은 시간 40   spark-3.1.2-bin-  37%[======>             ]  82.90M  3.54MB/s    남은 시간 40  spark-3.1.2-bin-h  38%[======>             ]  83.63M  3.55MB/s    남은 시간 40 spark-3.1.2-bin-ha  38%[======>             ]  84.38M  3.56MB/s    남은 시간 40spark-3.1.2-bin-had  38%[======>             ]  85.10M  3.55MB/s    남은 시간 39park-3.1.2-bin-hado  39%[======>             ]  85.80M  3.54MB/s    남은 시간 39ark-3.1.2-bin-hadoo  39%[======>             ]  86.52M  3.54MB/s    남은 시간 39rk-3.1.2-bin-hadoop  40%[=======>            ]  87.30M  3.53MB/s    남은 시간 39k-3.1.2-bin-hadoop3  40%[=======>            ]  88.09M  3.55MB/s    남은 시간 39-3.1.2-bin-hadoop3.  40%[=======>            ]  88.84M  3.55MB/s    남은 시간 383.1.2-bin-hadoop3.2  41%[=======>            ]  89.54M  3.55MB/s    남은 시간 38.1.2-bin-hadoop3.2.  41%[=======>            ]  90.27M  3.55MB/s    남은 시간 381.2-bin-hadoop3.2.t  41%[=======>            ]  90.99M  3.55MB/s    남은 시간 38.2-bin-hadoop3.2.tg  42%[=======>            ]  91.71M  3.55MB/s    남은 시간 382-bin-hadoop3.2.tgz  42%[=======>            ]  92.41M  3.54MB/s    남은 시간 36-bin-hadoop3.2.tgz   42%[=======>            ]  93.16M  3.56MB/s    남은 시간 36bin-hadoop3.2.tgz    43%[=======>            ]  93.88M  3.56MB/s    남은 시간 36in-hadoop3.2.tgz     43%[=======>            ]  94.59M  3.55MB/s    남은 시간 36n-hadoop3.2.tgz      43%[=======>            ]  95.30M  3.55MB/s    남은 시간 36-hadoop3.2.tgz       43%[=======>            ]  95.54M  3.12MB/s    남은 시간 36hadoop3.2.tgz        44%[=======>            ]  97.71M  3.55MB/s    남은 시간 36adoop3.2.tgz         44%[=======>            ]  98.15M  3.47MB/s    남은 시간 36doop3.2.tgz          45%[========>           ]  99.09M  3.53MB/s    남은 시간 36oop3.2.tgz           45%[========>           ]  99.77M  3.52MB/s    남은 시간 36op3.2.tgz            45%[========>           ] 100.37M  3.47MB/s    남은 시간 34p3.2.tgz             46%[========>           ] 101.38M  3.56MB/s    남은 시간 343.2.tgz              46%[========>           ] 101.96M  3.51MB/s    남은 시간 34.2.tgz               47%[========>           ] 102.60M  3.48MB/s    남은 시간 342.tgz                47%[========>           ] 103.55M  3.55MB/s    남은 시간 34.tgz                 47%[========>           ] 104.30M  3.57MB/s    남은 시간 33tgz                  48%[========>           ] 105.01M  3.56MB/s    남은 시간 33gz                   48%[========>           ] 105.73M  3.56MB/s    남은 시간 33z                    48%[========>           ] 106.43M  3.55MB/s    남은 시간 33                     49%[========>           ] 107.18M  3.56MB/s    남은 시간 33                  s  49%[========>           ] 107.90M  3.57MB/s    남은 시간 32                 sp  49%[========>           ] 108.60M  3.94MB/s    남은 시간 32                spa  50%[=========>          ] 109.32M  3.59MB/s    남은 시간 32               spar  50%[=========>          ] 110.07M  3.70MB/s    남은 시간 32              spark  50%[=========>          ] 110.77M  3.70MB/s    남은 시간 32             spark-  51%[=========>          ] 111.49M  3.57MB/s    남은 시간 31            spark-3  51%[=========>          ] 112.20M  3.67MB/s    남은 시간 31           spark-3.  51%[=========>          ] 112.91M  3.64MB/s    남은 시간 31          spark-3.1  52%[=========>          ] 113.70M  3.56MB/s    남은 시간 31         spark-3.1.  52%[=========>          ] 114.40M  3.60MB/s    남은 시간 31        spark-3.1.2  52%[=========>          ] 115.12M  3.56MB/s    남은 시간 30       spark-3.1.2-  53%[=========>          ] 115.82M  3.55MB/s    남은 시간 30      spark-3.1.2-b  53%[=========>          ] 116.57M  3.56MB/s    남은 시간 30     spark-3.1.2-bi  53%[=========>          ] 117.29M  3.57MB/s    남은 시간 30    spark-3.1.2-bin  54%[=========>          ] 117.99M  3.56MB/s    남은 시간 30   spark-3.1.2-bin-  54%[=========>          ] 118.02M  3.13MB/s    남은 시간 29  spark-3.1.2-bin-h  54%[=========>          ] 119.10M  3.24MB/s    남은 시간 29 spark-3.1.2-bin-ha  55%[==========>         ] 120.59M  3.45MB/s    남은 시간 29spark-3.1.2-bin-had  55%[==========>         ] 121.29M  3.45MB/s    남은 시간 29park-3.1.2-bin-hado  55%[==========>         ] 121.77M  3.36MB/s    남은 시간 29ark-3.1.2-bin-hadoo  56%[==========>         ] 122.74M  3.43MB/s    남은 시간 28rk-3.1.2-bin-hadoop  56%[==========>         ] 123.59M  3.47MB/s    남은 시간 28k-3.1.2-bin-hadoop3  56%[==========>         ] 123.98M  3.38MB/s    남은 시간 28-3.1.2-bin-hadoop3.  57%[==========>         ] 124.98M  3.45MB/s    남은 시간 283.1.2-bin-hadoop3.2  57%[==========>         ] 125.60M  3.41MB/s    남은 시간 28.1.2-bin-hadoop3.2.  57%[==========>         ] 126.26M  3.40MB/s    남은 시간 271.2-bin-hadoop3.2.t  58%[==========>         ] 127.29M  3.49MB/s    남은 시간 27.2-bin-hadoop3.2.tg  58%[==========>         ] 127.93M  3.47MB/s    남은 시간 272-bin-hadoop3.2.tgz  58%[==========>         ] 128.60M  3.45MB/s    남은 시간 27-bin-hadoop3.2.tgz   59%[==========>         ] 129.65M  3.55MB/s    남은 시간 27bin-hadoop3.2.tgz    59%[==========>         ] 130.09M  3.92MB/s    남은 시간 25in-hadoop3.2.tgz     60%[===========>        ] 131.01M  3.90MB/s    남은 시간 25n-hadoop3.2.tgz      60%[===========>        ] 131.82M  3.59MB/s    남은 시간 25-hadoop3.2.tgz       60%[===========>        ] 132.52M  3.60MB/s    남은 시간 25hadoop3.2.tgz        61%[===========>        ] 133.24M  3.72MB/s    남은 시간 25adoop3.2.tgz         61%[===========>        ] 134.01M  3.69MB/s    남은 시간 24doop3.2.tgz          61%[===========>        ] 134.73M  3.69MB/s    남은 시간 24oop3.2.tgz           62%[===========>        ] 135.45M  3.72MB/s    남은 시간 24op3.2.tgz            62%[===========>        ] 136.15M  3.71MB/s    남은 시간 24p3.2.tgz             62%[===========>        ] 136.90M  3.71MB/s    남은 시간 243.2.tgz              63%[===========>        ] 137.62M  3.69MB/s    남은 시간 23.2.tgz               63%[===========>        ] 138.32M  3.69MB/s    남은 시간 232.tgz                63%[===========>        ] 139.04M  3.66MB/s    남은 시간 23.tgz                 64%[===========>        ] 139.79M  3.64MB/s    남은 시간 23tgz                  64%[===========>        ] 140.49M  3.67MB/s    남은 시간 23gz                   64%[===========>        ] 141.21M  3.60MB/s    남은 시간 22z                    65%[============>       ] 141.93M  3.56MB/s    남은 시간 22                     65%[============>       ] 142.63M  3.60MB/s    남은 시간 22                  s  65%[============>       ] 143.38M  3.56MB/s    남은 시간 22                 sp  66%[============>       ] 144.09M  3.56MB/s    남은 시간 22                spa  66%[============>       ] 144.80M  3.55MB/s    남은 시간 21               spar  66%[============>       ] 145.52M  3.56MB/s    남은 시간 21              spark  67%[============>       ] 146.26M  3.57MB/s    남은 시간 21             spark-  67%[============>       ] 146.98M  3.60MB/s    남은 시간 21            spark-3  67%[============>       ] 147.68M  3.56MB/s    남은 시간 21           spark-3.  67%[============>       ] 148.40M  3.55MB/s    남은 시간 20          spark-3.1  68%[============>       ] 149.15M  3.57MB/s    남은 시간 20         spark-3.1.  68%[============>       ] 149.85M  3.56MB/s    남은 시간 20        spark-3.1.2  68%[============>       ] 150.57M  3.55MB/s    남은 시간 20       spark-3.1.2-  69%[============>       ] 151.29M  3.56MB/s    남은 시간 20      spark-3.1.2-b  69%[============>       ] 151.99M  3.55MB/s    남은 시간 19     spark-3.1.2-bi  69%[============>       ] 152.74M  3.56MB/s    남은 시간 19    spark-3.1.2-bin  70%[=============>      ] 153.45M  3.56MB/s    남은 시간 19   spark-3.1.2-bin-  70%[=============>      ] 154.16M  3.55MB/s    남은 시간 19  spark-3.1.2-bin-h  70%[=============>      ] 154.88M  3.55MB/s    남은 시간 19 spark-3.1.2-bin-ha  71%[=============>      ] 155.62M  3.56MB/s    남은 시간 18spark-3.1.2-bin-had  71%[=============>      ] 156.37M  3.56MB/s    남은 시간 18park-3.1.2-bin-hado  71%[=============>      ] 157.09M  3.55MB/s    남은 시간 18ark-3.1.2-bin-hadoo  72%[=============>      ] 157.79M  3.55MB/s    남은 시간 18rk-3.1.2-bin-hadoop  72%[=============>      ] 158.54M  3.56MB/s    남은 시간 18k-3.1.2-bin-hadoop3  72%[=============>      ] 159.26M  3.56MB/s    남은 시간 17-3.1.2-bin-hadoop3.  73%[=============>      ] 160.01M  3.56MB/s    남은 시간 173.1.2-bin-hadoop3.2  73%[=============>      ] 160.71M  3.55MB/s    남은 시간 17.1.2-bin-hadoop3.2.  73%[=============>      ] 161.43M  3.55MB/s    남은 시간 171.2-bin-hadoop3.2.t  74%[=============>      ] 162.21M  3.56MB/s    남은 시간 17.2-bin-hadoop3.2.tg  74%[=============>      ] 162.91M  3.56MB/s    남은 시간 162-bin-hadoop3.2.tgz  74%[=============>      ] 163.63M  3.56MB/s    남은 시간 16-bin-hadoop3.2.tgz   75%[==============>     ] 164.35M  3.55MB/s    남은 시간 16bin-hadoop3.2.tgz    75%[==============>     ] 165.13M  3.57MB/s    남은 시간 16in-hadoop3.2.tgz     75%[==============>     ] 165.84M  3.56MB/s    남은 시간 16n-hadoop3.2.tgz      76%[==============>     ] 166.54M  3.55MB/s    남은 시간 15-hadoop3.2.tgz       76%[==============>     ] 167.26M  3.55MB/s    남은 시간 15hadoop3.2.tgz        76%[==============>     ] 168.04M  3.57MB/s    남은 시간 15adoop3.2.tgz         77%[==============>     ] 168.74M  3.56MB/s    남은 시간 15doop3.2.tgz          77%[==============>     ] 169.46M  3.56MB/s    남은 시간 15oop3.2.tgz           77%[==============>     ] 170.16M  3.55MB/s    남은 시간 14op3.2.tgz            78%[==============>     ] 170.96M  3.56MB/s    남은 시간 14p3.2.tgz             78%[==============>     ] 171.66M  3.56MB/s    남은 시간 143.2.tgz              78%[==============>     ] 172.38M  3.56MB/s    남은 시간 14.2.tgz               79%[==============>     ] 173.09M  3.55MB/s    남은 시간 142.tgz                79%[==============>     ] 173.84M  3.55MB/s    남은 시간 13.tgz                 79%[==============>     ] 174.59M  3.56MB/s    남은 시간 13tgz                  80%[===============>    ] 175.29M  3.56MB/s    남은 시간 13gz                   80%[===============>    ] 175.46M  3.13MB/s    남은 시간 13z                    80%[===============>    ] 176.59M  3.25MB/s    남은 시간 12                     81%[===============>    ] 178.16M  3.49MB/s    남은 시간 12                  s  81%[===============>    ] 178.84M  3.48MB/s    남은 시간 12                 sp  82%[===============>    ] 179.46M  3.46MB/s    남은 시간 12                spa  82%[===============>    ] 180.41M  3.51MB/s    남은 시간 12               spar  82%[===============>    ] 180.88M  3.45MB/s    남은 시간 11              spark  83%[===============>    ] 181.80M  3.50MB/s    남은 시간 11             spark-  83%[===============>    ] 182.70M  3.55MB/s    남은 시간 11            spark-3  83%[===============>    ] 183.27M  3.51MB/s    남은 시간 11           spark-3.  84%[===============>    ] 184.15M  3.56MB/s    남은 시간 11          spark-3.1  84%[===============>    ] 184.90M  3.56MB/s    남은 시간 10         spark-3.1.  85%[================>   ] 185.62M  3.56MB/s    남은 시간 10        spark-3.1.2  85%[================>   ] 186.32M  3.56MB/s    남은 시간 10       spark-3.1.2-  85%[================>   ] 187.07M  3.56MB/s    남은 시간 10      spark-3.1.2-b  86%[================>   ] 187.79M  3.56MB/s    남은 시간 10spark-3.1.2-bin-had 100%[===================>] 218.23M  3.53MB/s    / 62s      s

2021-07-27 20:06:07 (3.51 MB/s) - `spark-3.1.2-bin-hadoop3.2.tgz' 저장됨 [228834641/228834641]

ninipa@ninipa-VirtualBox:~$ 

3. Spark를 /opt에 풀기

ninipa@ninipa-VirtualBox:~$ sudo mkdir /opt/spark
[sudo] ninipa의 암호: 
ninipa@ninipa-VirtualBox:~$ sudo tar -xf spark*.tgz -C /opt/spark --strip-component 1
ninipa@ninipa-VirtualBox:~$ ls -al /opt/spark/
합계 164
drwxr-xr-x 13 root   root    4096  7월 27 20:10 .
drwxr-xr-x  6 root   root    4096  7월 27 20:09 ..
-rw-r--r--  1 ninipa ninipa 23235  5월 24 13:45 LICENSE
-rw-r--r--  1 ninipa ninipa 57677  5월 24 13:45 NOTICE
drwxr-xr-x  3 ninipa ninipa  4096  5월 24 13:45 R
-rw-r--r--  1 ninipa ninipa  4488  5월 24 13:45 README.md
-rw-r--r--  1 ninipa ninipa   183  5월 24 13:45 RELEASE
drwxr-xr-x  2 ninipa ninipa  4096  5월 24 13:45 bin
drwxr-xr-x  2 ninipa ninipa  4096  5월 24 13:45 conf
drwxr-xr-x  5 ninipa ninipa  4096  5월 24 13:45 data
drwxr-xr-x  4 ninipa ninipa  4096  5월 24 13:45 examples
drwxr-xr-x  2 ninipa ninipa 20480  5월 24 13:45 jars
drwxr-xr-x  4 ninipa ninipa  4096  5월 24 13:45 kubernetes
drwxr-xr-x  2 ninipa ninipa  4096  5월 24 13:45 licenses
drwxr-xr-x  9 ninipa ninipa  4096  5월 24 13:45 python
drwxr-xr-x  2 ninipa ninipa  4096  5월 24 13:45 sbin
drwxr-xr-x  2 ninipa ninipa  4096  5월 24 13:45 yarn
ninipa@ninipa-VirtualBox:~$ sudo chmod -R 777 /opt/spark
ninipa@ninipa-VirtualBox:~$ ls -al /opt/spark/
합계 164
drwxrwxrwx 13 root   root    4096  7월 27 20:10 .
drwxr-xr-x  6 root   root    4096  7월 27 20:09 ..
-rwxrwxrwx  1 ninipa ninipa 23235  5월 24 13:45 LICENSE
-rwxrwxrwx  1 ninipa ninipa 57677  5월 24 13:45 NOTICE
drwxrwxrwx  3 ninipa ninipa  4096  5월 24 13:45 R
-rwxrwxrwx  1 ninipa ninipa  4488  5월 24 13:45 README.md
-rwxrwxrwx  1 ninipa ninipa   183  5월 24 13:45 RELEASE
drwxrwxrwx  2 ninipa ninipa  4096  5월 24 13:45 bin
drwxrwxrwx  2 ninipa ninipa  4096  5월 24 13:45 conf
drwxrwxrwx  5 ninipa ninipa  4096  5월 24 13:45 data
drwxrwxrwx  4 ninipa ninipa  4096  5월 24 13:45 examples
drwxrwxrwx  2 ninipa ninipa 20480  5월 24 13:45 jars
drwxrwxrwx  4 ninipa ninipa  4096  5월 24 13:45 kubernetes
drwxrwxrwx  2 ninipa ninipa  4096  5월 24 13:45 licenses
drwxrwxrwx  9 ninipa ninipa  4096  5월 24 13:45 python
drwxrwxrwx  2 ninipa ninipa  4096  5월 24 13:45 sbin
drwxrwxrwx  2 ninipa ninipa  4096  5월 24 13:45 yarn
ninipa@ninipa-VirtualBox:~$ 

4. 시스템 경로에 Spark 폴더 추가하기

ninipa@ninipa-VirtualBox:~$ nano ~/.bashrc

아래 3 줄 추가
export SPARK_HOME=/opt/spark
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin
export PYSPARK_PYTHON=/usr/bin/python3 

 추가 후 쉘 스크립트를 실행하여 변경사항을 적용한다.

ninipa@ninipa-VirtualBox:~$ source ~/.bashrc

5. 우분투에서 Apache Spark master 시작하기

ninipa@ninipa-VirtualBox:~$ start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /opt/spark/logs/spark-ninipa-org.apache.spark.deploy.master.Master-1-ninipa-VirtualBox.out
ninipa@ninipa-VirtualBox:~$ 

기본으로 master는 7077번 포트를 worker는 8080 포트를 사용한다.

6. 웹 인터페이스에서 Spark master 접속하기

브라우저 실행하고 http://127.0.0.1:8080에 접속하기

이후 /opt/spark/log 경로에 생긴 master 로그 파일과 그 내용은 아래와 같다.

ninipa@ninipa-VirtualBox:/opt/spark/logs$ ls -al
합계 12
drwxrwxr-x  2 ninipa ninipa 4096  7월 28 10:00 .
drwxrwxrwx 14 root   root   4096  7월 28 10:00 ..
-rw-rw-r--  1 ninipa ninipa 2555  7월 28 10:00 spark-ninipa-org.apache.spark.deploy.master.Master-1-ninipa-VirtualBox.out
ninipa@ninipa-VirtualBox:/opt/spark/logs$ cat spark-ninipa-org.apache.spark.deploy.master.Master-1-ninipa-VirtualBox.out 
Spark Command: /usr/lib/jvm/java-11-openjdk-amd64/bin/java -cp /opt/spark/conf/:/opt/spark/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host ninipa-VirtualBox --port 7077 --webui-port 8080
========================================
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/07/28 10:00:54 INFO Master: Started daemon with process name: 38607@ninipa-VirtualBox
21/07/28 10:00:54 INFO SignalUtils: Registering signal handler for TERM
21/07/28 10:00:54 INFO SignalUtils: Registering signal handler for HUP
21/07/28 10:00:54 INFO SignalUtils: Registering signal handler for INT
21/07/28 10:00:54 WARN Utils: Your hostname, ninipa-VirtualBox resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface enp0s3)
21/07/28 10:00:54 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.1.2.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
21/07/28 10:00:55 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/07/28 10:00:55 INFO SecurityManager: Changing view acls to: ninipa
21/07/28 10:00:55 INFO SecurityManager: Changing modify acls to: ninipa
21/07/28 10:00:55 INFO SecurityManager: Changing view acls groups to: 
21/07/28 10:00:55 INFO SecurityManager: Changing modify acls groups to: 
21/07/28 10:00:55 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(ninipa); groups with view permissions: Set(); users  with modify permissions: Set(ninipa); groups with modify permissions: Set()
21/07/28 10:00:55 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
21/07/28 10:00:55 INFO Master: Starting Spark master at spark://ninipa-VirtualBox:7077
21/07/28 10:00:55 INFO Master: Running Spark version 3.1.2
21/07/28 10:00:56 INFO Utils: Successfully started service 'MasterUI' on port 8080.
21/07/28 10:00:56 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://10.0.2.15:8080
21/07/28 10:00:56 INFO Master: I have been elected leader! New state: ALIVE
ninipa@ninipa-VirtualBox:/opt/spark/logs$ 

7. worker 스크립트 실행

문법: start-worker.sh spark://hostname:port

명령: start-worker.sh spark://ninipa-VirtualBox:7077
ninipa@ninipa-VirtualBox:~$ start-worker.sh spark://ninipa-VirtualBox:7077
starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark/logs/spark-ninipa-org.apache.spark.deploy.worker.Worker-1-ninipa-VirtualBox.out
ninipa@ninipa-VirtualBox:~$ 

이후 /opt/spark/log 경로에 생긴 worker로그 파일과 그 내용은 아래와 같다.

ninipa@ninipa-VirtualBox:/opt/spark/logs$ ls -al
합계 16
drwxrwxr-x  2 ninipa ninipa 4096  7월 28 10:17 .
drwxrwxrwx 15 root   root   4096  7월 28 10:18 ..
-rw-rw-r--  1 ninipa ninipa 2647  7월 28 10:18 spark-ninipa-org.apache.spark.deploy.master.Master-1-ninipa-VirtualBox.out
-rw-rw-r--  1 ninipa ninipa 3253  7월 28 10:18 spark-ninipa-org.apache.spark.deploy.worker.Worker-1-ninipa-VirtualBox.out
ninipa@ninipa-VirtualBox:/opt/spark/logs$ cat spark-ninipa-org.apache.spark.deploy.worker.Worker-1-ninipa-VirtualBox.out 
Spark Command: /usr/lib/jvm/java-11-openjdk-amd64/bin/java -cp /opt/spark/conf/:/opt/spark/jars/* -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://ninipa-VirtualBox:7077
========================================
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/07/28 10:18:00 INFO Worker: Started daemon with process name: 38979@ninipa-VirtualBox
21/07/28 10:18:00 INFO SignalUtils: Registering signal handler for TERM
21/07/28 10:18:00 INFO SignalUtils: Registering signal handler for HUP
21/07/28 10:18:00 INFO SignalUtils: Registering signal handler for INT
21/07/28 10:18:00 WARN Utils: Your hostname, ninipa-VirtualBox resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface enp0s3)
21/07/28 10:18:00 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.1.2.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
21/07/28 10:18:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/07/28 10:18:01 INFO SecurityManager: Changing view acls to: ninipa
21/07/28 10:18:01 INFO SecurityManager: Changing modify acls to: ninipa
21/07/28 10:18:01 INFO SecurityManager: Changing view acls groups to: 
21/07/28 10:18:01 INFO SecurityManager: Changing modify acls groups to: 
21/07/28 10:18:01 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(ninipa); groups with view permissions: Set(); users  with modify permissions: Set(ninipa); groups with modify permissions: Set()
21/07/28 10:18:01 INFO Utils: Successfully started service 'sparkWorker' on port 33489.
21/07/28 10:18:01 INFO Worker: Worker decommissioning not enabled, SIGPWR will result in exiting.
21/07/28 10:18:01 INFO Worker: Starting Spark worker 10.0.2.15:33489 with 1 cores, 6.8 GiB RAM
21/07/28 10:18:01 INFO Worker: Running Spark version 3.1.2
21/07/28 10:18:01 INFO Worker: Spark home: /opt/spark
21/07/28 10:18:01 INFO ResourceUtils: ==============================================================
21/07/28 10:18:01 INFO ResourceUtils: No custom resources configured for spark.worker.
21/07/28 10:18:01 INFO ResourceUtils: ==============================================================
21/07/28 10:18:02 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
21/07/28 10:18:02 INFO WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://10.0.2.15:8081
21/07/28 10:18:02 INFO Worker: Connecting to master ninipa-VirtualBox:7077...
21/07/28 10:18:02 INFO TransportClientFactory: Successfully created connection to ninipa-VirtualBox/127.0.1.1:7077 after 48 ms (0 ms spent in bootstraps)
21/07/28 10:18:02 INFO Worker: Successfully registered with master spark://ninipa-VirtualBox:7077
ninipa@ninipa-VirtualBox:/opt/spark/logs$ 

8. Spark shell 사용

다음 명령을 입력하여 Spark 셸을 사용한 프로그래밍을 할 수 있다.

ninipa@ninipa-VirtualBox:~$ spark-shell
21/07/28 10:26:03 WARN Utils: Your hostname, ninipa-VirtualBox resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface enp0s3)
21/07/28 10:26:03 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.1.2.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
21/07/28 10:26:03 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://10.0.2.15:4040
Spark context available as 'sc' (master = local[*], app id = local-1627435573621).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.1.2
      /_/
         
Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 11.0.11)
Type in expressions to have them evaluated.
Type :help for more information.

scala> 

Spark 실행 후 브라우저에서 master와 worker의  Web UI 실행한 내용은 아래와 같다.

Worker ID와 그 상태 및 사용 리소스에 대한 정보가 표시된다.

 

반응형
0 Comments
댓글쓰기 폼