우분투 20.04에서 Apache Spark 설치

 

Apache Spark 개요

Apache Spark는 대용량 데이터 처리를 위한 통합 분석 엔진으로 배치와 스트리밍 데이터를 고성능으로 처리하고 자바, 스칼라, 파이썬, R 등의 다양한 프로그래밍 언어를 지원한다.

Apache Spark의 프로그래밍 모델은 클러스터에서 분산 작동하는 RDD(Resilient Distributed Datasets)를 기반으로 한다.

최근 데이터 사이언티스트들이 가장 선호하는 도구들 중의 하나이다.

설치 개요

0. update & upgrade sudo apt update sudo apt-get upgrade 1. Java, scala, git 설치 sudo apt install default-jdk scala git 2. Apache Spark 다운로드 wget https://downloads.apache.org/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz 3. Spark를 /opt에 풀기 sudo mkdir /opt/spark sudo tar -xf spark*.tgz -C /opt/spark --strip-component 1 sudo chmod -R 777 /opt/spark 4. 시스템 경로에 Spark 폴더 추가하기 echo "export SPARK_HOME=/opt/spark" >> ~/.bashrc echo "export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin" >> ~/.bashrc echo "export PYSPARK_PYTHON=/usr/bin/python3" >> ~/.bashrc Reload shell: source ~/.bashrc 5. 우분투에서 Apache Spark master 시작하기 start-master.sh 포트를 변경하고 싶을 경우 --port 옵션 사용 --port 서비스가 사용할 포트 (기본: 마스터는 7077이고 worker는 임의의 값) --webui-port 웹 UI가 사용할 포트 (기본: 마스터는 8080, worker는 8081) start-master.sh --port 7072 --webui-port 8082 6. 웹 인터페이스에서 Spark master 접속하기 브라우저 실행하고 http://127.0.0.1:8080에 접속하기 7. 워커 스크립트 실행 문법: start-worker.sh spark://hostname:port 명령: start-worker.sh spark://ubuntu:7077 메모리/램 사용량 변경해서 실행하려면 stop-worker.sh start-worker.sh -m 212M spark://ubuntu:7077 8. Spark shell 사용 spark-shell

상세 절차

0. update & upgrade

ninipa@ninipa-VirtualBox:~$ sudo apt update [sudo] ninipa의 암호:  기존:1 https://download.docker.com/linux/ubuntu focal InRelease 기존:2 http://dl.google.com/linux/chrome/deb stable InRelease                   기존:3 http://kr.archive.ubuntu.com/ubuntu focal InRelease                      받기:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]     받기:5 http://kr.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]    받기:6 http://kr.archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB] 받기:7 http://kr.archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [5,800 B] 받기:8 http://kr.archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [4,736 B] 받기:9 http://kr.archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [2,068 B] 내려받기 341 k바이트, 소요시간 3초 (122 k바이트/초) 패키지 목록을 읽는 중입니다... 완료 의존성 트리를 만드는 중입니다        상태 정보를 읽는 중입니다... 완료 패키지 16이(가) 업그레이드되었습니다. 'apt list --upgradable'를 실행하여 확인해 보십시오. ninipa@ninipa-VirtualBox:~$ sudo apt upgrade 패키지 목록을 읽는 중입니다... 완료 의존성 트리를 만드는 중입니다        상태 정보를 읽는 중입니다... 완료 업그레이드를 계산하는 중입니다... 완료 다음 패키지가 자동으로 설치되었지만 더 이상 필요하지 않습니다:   linux-headers-5.8.0-43-generic linux-hwe-5.8-headers-5.8.0-43   linux-image-5.8.0-43-generic linux-modules-5.8.0-43-generic   linux-modules-extra-5.8.0-43-generic 'sudo apt autoremove'를 이용하여 제거하십시오. 다음 새 패키지를 설치할 것입니다:   distro-info 다음 패키지를 업그레이드할 것입니다:   containerd.io gdm3 gir1.2-gdm-1.0 google-chrome-stable libgdm1 libglib2.0-0   libglib2.0-bin libglib2.0-data libgupnp-1.2-0 linux-base networkd-dispatcher   python3-distupgrade ubuntu-advantage-tools ubuntu-drivers-common   ubuntu-release-upgrader-core ubuntu-release-upgrader-gtk 16개 업그레이드, 1개 새로 설치, 0개 제거 및 0개 업그레이드 안 함. 114 M바이트 아카이브를 받아야 합니다. 이 작업 후 9,369 k바이트의 디스크 공간이 비워집니다. 계속 하시겠습니까? [Y/n] y 받기:1 https://download.docker.com/linux/ubuntu focal/stable amd64 containerd.io amd64 1.4.8-1 [24.7 MB] 받기:2 http://dl.google.com/linux/chrome/deb stable/main amd64 google-chrome-stable amd64 92.0.4515.107-1 [86.2 MB] 받기:3 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 libglib2.0-data all 2.64.6-1~ubuntu20.04.4 [6,052 B] 받기:4 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 libglib2.0-bin amd64 2.64.6-1~ubuntu20.04.4 [72.6 kB] 받기:5 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 libglib2.0-0 amd64 2.64.6-1~ubuntu20.04.4 [1,287 kB] 받기:6 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 ubuntu-drivers-common amd64 1:0.9.0~0.20.04.1 [53.0 kB] 받기:7 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 networkd-dispatcher all 2.1-2~ubuntu20.04.1 [14.6 kB] 받기:8 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 distro-info amd64 0.23ubuntu1 [17.1 kB] 받기:9 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 ubuntu-advantage-tools amd64 27.2.1~20.04.1 [844 kB] 받기:10 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 ubuntu-release-upgrader-gtk all 1:20.04.35 [9,364 B] 받기:11 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 ubuntu-release-upgrader-core all 1:20.04.35 [23.9 kB] 받기:12 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-distupgrade all 1:20.04.35 [104 kB] 받기:13 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 gdm3 amd64 3.36.3-0ubuntu0.20.04.4 [261 kB] 받기:14 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 libgdm1 amd64 3.36.3-0ubuntu0.20.04.4 [63.6 kB] 받기:15 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 gir1.2-gdm-1.0 amd64 3.36.3-0ubuntu0.20.04.4 [9,584 B] 받기:16 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 libgupnp-1.2-0 amd64 1.2.4-0ubuntu1 [81.3 kB] 받기:17 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-base all 4.5ubuntu3.6 [17.8 kB] 내려받기 114 M바이트, 소요시간 31초 (3,692 k바이트/초)                          패키지를 미리 설정하는 중입니다... (데이터베이스 읽는중 ...현재 218865개의 파일과 디렉터리가 설치되어 있습니다.) Preparing to unpack .../00-libglib2.0-data_2.64.6-1~ubuntu20.04.4_all.deb ... Unpacking libglib2.0-data (2.64.6-1~ubuntu20.04.4) over (2.64.6-1~ubuntu20.04.3)  ... Preparing to unpack .../01-libglib2.0-bin_2.64.6-1~ubuntu20.04.4_amd64.deb ... Unpacking libglib2.0-bin (2.64.6-1~ubuntu20.04.4) over (2.64.6-1~ubuntu20.04.3)  ... Preparing to unpack .../02-libglib2.0-0_2.64.6-1~ubuntu20.04.4_amd64.deb ... Unpacking libglib2.0-0:amd64 (2.64.6-1~ubuntu20.04.4) over (2.64.6-1~ubuntu20.04 .3) ... Preparing to unpack .../03-google-chrome-stable_92.0.4515.107-1_amd64.deb ... Unpacking google-chrome-stable (92.0.4515.107-1) over (91.0.4472.114-1) ... Preparing to unpack .../04-ubuntu-drivers-common_1%3a0.9.0~0.20.04.1_amd64.deb . .. Unpacking ubuntu-drivers-common (1:0.9.0~0.20.04.1) over (1:0.8.6.5~0.20.04.1) . .. Preparing to unpack .../05-networkd-dispatcher_2.1-2~ubuntu20.04.1_all.deb ... Unpacking networkd-dispatcher (2.1-2~ubuntu20.04.1) over (2.0.1-1) ... Selecting previously unselected package distro-info. Preparing to unpack .../06-distro-info_0.23ubuntu1_amd64.deb ... Unpacking distro-info (0.23ubuntu1) ... Preparing to unpack .../07-ubuntu-advantage-tools_27.2.1~20.04.1_amd64.deb ... Unpacking ubuntu-advantage-tools (27.2.1~20.04.1) over (20.3) ... Preparing to unpack .../08-ubuntu-release-upgrader-gtk_1%3a20.04.35_all.deb ... Unpacking ubuntu-release-upgrader-gtk (1:20.04.35) over (1:20.04.33) ... Preparing to unpack .../09-ubuntu-release-upgrader-core_1%3a20.04.35_all.deb ... Unpacking ubuntu-release-upgrader-core (1:20.04.35) over (1:20.04.33) ... Preparing to unpack .../10-python3-distupgrade_1%3a20.04.35_all.deb ... Unpacking python3-distupgrade (1:20.04.35) over (1:20.04.33) ... Preparing to unpack .../11-containerd.io_1.4.8-1_amd64.deb ... Unpacking containerd.io (1.4.8-1) over (1.4.6-1) ... Preparing to unpack .../12-gdm3_3.36.3-0ubuntu0.20.04.4_amd64.deb ... Unpacking gdm3 (3.36.3-0ubuntu0.20.04.4) over (3.36.3-0ubuntu0.20.04.3) ... Preparing to unpack .../13-libgdm1_3.36.3-0ubuntu0.20.04.4_amd64.deb ... Unpacking libgdm1 (3.36.3-0ubuntu0.20.04.4) over (3.36.3-0ubuntu0.20.04.3) ... Preparing to unpack .../14-gir1.2-gdm-1.0_3.36.3-0ubuntu0.20.04.4_amd64.deb ... Unpacking gir1.2-gdm-1.0:amd64 (3.36.3-0ubuntu0.20.04.4) over (3.36.3-0ubuntu0.2 0.04.3) ... Preparing to unpack .../15-libgupnp-1.2-0_1.2.4-0ubuntu1_amd64.deb ... Unpacking libgupnp-1.2-0:amd64 (1.2.4-0ubuntu1) over (1.2.3-0ubuntu0.20.04.2) .. . Preparing to unpack .../16-linux-base_4.5ubuntu3.6_all.deb ... Unpacking linux-base (4.5ubuntu3.6) over (4.5ubuntu3.5) ... networkd-dispatcher (2.1-2~ubuntu20.04.1) 설정하는 중입니다 ... distro-info (0.23ubuntu1) 설정하는 중입니다 ... linux-base (4.5ubuntu3.6) 설정하는 중입니다 ... ubuntu-drivers-common (1:0.9.0~0.20.04.1) 설정하는 중입니다 ... libglib2.0-0:amd64 (2.64.6-1~ubuntu20.04.4) 설정하는 중입니다 ... python3-distupgrade (1:20.04.35) 설정하는 중입니다 ... libglib2.0-data (2.64.6-1~ubuntu20.04.4) 설정하는 중입니다 ... containerd.io (1.4.8-1) 설정하는 중입니다 ... ubuntu-release-upgrader-core (1:20.04.35) 설정하는 중입니다 ... ubuntu-release-upgrader-gtk (1:20.04.35) 설정하는 중입니다 ... ubuntu-advantage-tools (27.2.1~20.04.1) 설정하는 중입니다 ... 새 버전의 설정 파일 /etc/ubuntu-advantage/uaclient.conf 설치하는 중입니다 ... Created symlink /etc/systemd/system/multi-user.target.wants/ua-reboot-cmds.servi ce → /lib/systemd/system/ua-reboot-cmds.service. Created symlink /etc/systemd/system/timers.target.wants/ua-messaging.timer → /li b/systemd/system/ua-messaging.timer. libgupnp-1.2-0:amd64 (1.2.4-0ubuntu1) 설정하는 중입니다 ... google-chrome-stable (92.0.4515.107-1) 설정하는 중입니다 ... libgdm1 (3.36.3-0ubuntu0.20.04.4) 설정하는 중입니다 ... libglib2.0-bin (2.64.6-1~ubuntu20.04.4) 설정하는 중입니다 ... gir1.2-gdm-1.0:amd64 (3.36.3-0ubuntu0.20.04.4) 설정하는 중입니다 ... gdm3 (3.36.3-0ubuntu0.20.04.4) 설정하는 중입니다 ... Processing triggers for mime-support (3.64ubuntu1) ... Processing triggers for gnome-menus (3.36.0-1ubuntu1) ... Processing triggers for libc-bin (2.31-0ubuntu9.2) ... Processing triggers for systemd (245.4-4ubuntu3.11) ... Processing triggers for man-db (2.9.1-1) ... Processing triggers for dbus (1.12.16-2ubuntu2.1) ... Processing triggers for desktop-file-utils (0.24-1ubuntu3) ... ninipa@ninipa-VirtualBox:~$

1. java, scala, git 설치

Apache Spark는 필수로 java가 필요하고 scala는 프로그래밍을 위해 추가한다.

ninipa@ninipa-VirtualBox:~$ sudo apt install default-jdk scala [sudo] ninipa의 암호:  패키지 목록을 읽는 중입니다... 완료 의존성 트리를 만드는 중입니다        상태 정보를 읽는 중입니다... 완료 다음 패키지가 자동으로 설치되었지만 더 이상 필요하지 않습니다:   linux-headers-5.8.0-43-generic linux-hwe-5.8-headers-5.8.0-43   linux-image-5.8.0-43-generic linux-modules-5.8.0-43-generic   linux-modules-extra-5.8.0-43-generic 'sudo apt autoremove'를 이용하여 제거하십시오. 다음의 추가 패키지가 설치될 것입니다 :   ca-certificates-java default-jdk-headless default-jre default-jre-headless   fonts-dejavu-extra java-common libatk-wrapper-java libatk-wrapper-java-jni   libhawtjni-runtime-java libice-dev libjansi-java libjansi-native-java   libjline2-java libpthread-stubs0-dev libsm-dev libx11-dev libxau-dev   libxcb1-dev libxdmcp-dev libxt-dev openjdk-11-jdk openjdk-11-jdk-headless   openjdk-11-jre openjdk-11-jre-headless scala-library   scala-parser-combinators scala-xml x11proto-core-dev x11proto-dev   xorg-sgml-doctools xtrans-dev 제안하는 패키지:   libice-doc libsm-doc libx11-doc libxcb-doc libxt-doc openjdk-11-demo   openjdk-11-source visualvm fonts-ipafont-gothic fonts-ipafont-mincho   fonts-wqy-microhei | fonts-wqy-zenhei scala-doc 다음 새 패키지를 설치할 것입니다:   ca-certificates-java default-jdk default-jdk-headless default-jre   default-jre-headless fonts-dejavu-extra java-common libatk-wrapper-java   libatk-wrapper-java-jni libhawtjni-runtime-java libice-dev libjansi-java   libjansi-native-java libjline2-java libpthread-stubs0-dev libsm-dev   libx11-dev libxau-dev libxcb1-dev libxdmcp-dev libxt-dev openjdk-11-jdk   openjdk-11-jdk-headless openjdk-11-jre openjdk-11-jre-headless scala   scala-library scala-parser-combinators scala-xml x11proto-core-dev   x11proto-dev xorg-sgml-doctools xtrans-dev 0개 업그레이드, 33개 새로 설치, 0개 제거 및 0개 업그레이드 안 함. 291 M바이트 아카이브를 받아야 합니다. 이 작업 후 449 M바이트의 디스크 공간을 더 사용하게 됩니다. 계속 하시겠습니까? [Y/n] y 받기:1 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 java-common all 0.72 [6,816 B] 받기:2 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 openjdk-11-jre-headless amd64 11.0.11+9-0ubuntu2~20.04 [37.2 MB] 받기:3 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 default-jre-headless amd64 2:1.11-72 [3,192 B] 받기:4 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 ca-certificates-java all 20190405ubuntu1 [12.2 kB] 받기:5 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 openjdk-11-jre amd64 11.0.11+9-0ubuntu2~20.04 [174 kB] 받기:6 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 default-jre amd64 2:1.11-72 [1,084 B] 받기:7 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 openjdk-11-jdk-headless amd64 11.0.11+9-0ubuntu2~20.04 [223 MB] 받기:8 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 default-jdk-headless amd64 2:1.11-72 [1,140 B] 받기:9 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 openjdk-11-jdk amd64 11.0.11+9-0ubuntu2~20.04 [1,442 kB] 받기:10 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 default-jdk amd64 2:1.11-72 [1,096 B] 받기:11 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 fonts-dejavu-extra all 2.37-1 [1,953 kB] 받기:12 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 libatk-wrapper-java all 0.37.1-1 [53.0 kB] 받기:13 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 libatk-wrapper-java-jni amd64 0.37.1-1 [45.1 kB] 받기:14 http://kr.archive.ubuntu.com/ubuntu focal/universe amd64 libhawtjni-runtime-java all 1.17-1 [28.8 kB] 받기:15 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 xorg-sgml-doctools all 1:1.11-1 [12.9 kB] 받기:16 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 x11proto-dev all 2019.2-1ubuntu1 [594 kB] 받기:17 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 x11proto-core-dev all 2019.2-1ubuntu1 [2,620 B] 받기:18 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 libice-dev amd64 2:1.0.10-0ubuntu1 [47.8 kB] 받기:19 http://kr.archive.ubuntu.com/ubuntu focal/universe amd64 libjansi-native-java all 1.8-1 [23.8 kB] 받기:20 http://kr.archive.ubuntu.com/ubuntu focal/universe amd64 libjansi-java all 1.18-1 [56.8 kB] 받기:21 http://kr.archive.ubuntu.com/ubuntu focal/universe amd64 libjline2-java all 2.14.6-3 [150 kB] 받기:22 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 libpthread-stubs0-dev amd64 0.4-1 [5,384 B] 받기:23 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 libsm-dev amd64 2:1.2.3-1 [17.0 kB] 받기:24 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 libxau-dev amd64 1:1.0.9-0ubuntu1 [9,552 B] 받기:25 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 libxdmcp-dev amd64 1:1.1.3-0ubuntu1 [25.3 kB] 받기:26 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 xtrans-dev all 1.4.0-1 [68.9 kB] 받기:27 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 libxcb1-dev amd64 1.14-2 [80.5 kB] 받기:28 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 libx11-dev amd64 2:1.6.9-2ubuntu1.2 [647 kB] 받기:29 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 libxt-dev amd64 1:1.1.5-1 [395 kB] 받기:30 http://kr.archive.ubuntu.com/ubuntu focal/universe amd64 scala-library all 2.11.12-4 [9,583 kB] 받기:31 http://kr.archive.ubuntu.com/ubuntu focal/universe amd64 scala-parser-combinators all 1.0.3-3 [355 kB] 받기:32 http://kr.archive.ubuntu.com/ubuntu focal/universe amd64 scala-xml all 1.0.3-3 [601 kB] 받기:33 http://kr.archive.ubuntu.com/ubuntu focal/universe amd64 scala all 2.11.12-4 [14.2 MB] 내려받기 291 M바이트, 소요시간 16분 1초 (303 k바이트/초)                        패키지에서 템플릿을 추출하는 중: 100% Selecting previously unselected package java-common. (데이터베이스 읽는중 ...현재 218891개의 파일과 디렉터리가 설치되어 있습니다.) Preparing to unpack .../00-java-common_0.72_all.deb ... Unpacking java-common (0.72) ... Selecting previously unselected package openjdk-11-jre-headless:amd64. Preparing to unpack .../01-openjdk-11-jre-headless_11.0.11+9-0ubuntu2~20.04_amd6 4.deb ... Unpacking openjdk-11-jre-headless:amd64 (11.0.11+9-0ubuntu2~20.04) ... Selecting previously unselected package default-jre-headless. Preparing to unpack .../02-default-jre-headless_2%3a1.11-72_amd64.deb ... Unpacking default-jre-headless (2:1.11-72) ... Selecting previously unselected package ca-certificates-java. Preparing to unpack .../03-ca-certificates-java_20190405ubuntu1_all.deb ... Unpacking ca-certificates-java (20190405ubuntu1) ... Selecting previously unselected package openjdk-11-jre:amd64. Preparing to unpack .../04-openjdk-11-jre_11.0.11+9-0ubuntu2~20.04_amd64.deb ... Unpacking openjdk-11-jre:amd64 (11.0.11+9-0ubuntu2~20.04) ... Selecting previously unselected package default-jre. Preparing to unpack .../05-default-jre_2%3a1.11-72_amd64.deb ... Unpacking default-jre (2:1.11-72) ... Selecting previously unselected package openjdk-11-jdk-headless:amd64. Preparing to unpack .../06-openjdk-11-jdk-headless_11.0.11+9-0ubuntu2~20.04_amd6 4.deb ... Unpacking openjdk-11-jdk-headless:amd64 (11.0.11+9-0ubuntu2~20.04) ... Selecting previously unselected package default-jdk-headless. Preparing to unpack .../07-default-jdk-headless_2%3a1.11-72_amd64.deb ... Unpacking default-jdk-headless (2:1.11-72) ... Selecting previously unselected package openjdk-11-jdk:amd64. Preparing to unpack .../08-openjdk-11-jdk_11.0.11+9-0ubuntu2~20.04_amd64.deb ... Unpacking openjdk-11-jdk:amd64 (11.0.11+9-0ubuntu2~20.04) ... Selecting previously unselected package default-jdk. Preparing to unpack .../09-default-jdk_2%3a1.11-72_amd64.deb ... Unpacking default-jdk (2:1.11-72) ... Selecting previously unselected package fonts-dejavu-extra. Preparing to unpack .../10-fonts-dejavu-extra_2.37-1_all.deb ... Unpacking fonts-dejavu-extra (2.37-1) ... Selecting previously unselected package libatk-wrapper-java. Preparing to unpack .../11-libatk-wrapper-java_0.37.1-1_all.deb ... Unpacking libatk-wrapper-java (0.37.1-1) ... Selecting previously unselected package libatk-wrapper-java-jni:amd64. Preparing to unpack .../12-libatk-wrapper-java-jni_0.37.1-1_amd64.deb ... Unpacking libatk-wrapper-java-jni:amd64 (0.37.1-1) ... Selecting previously unselected package libhawtjni-runtime-java. Preparing to unpack .../13-libhawtjni-runtime-java_1.17-1_all.deb ... Unpacking libhawtjni-runtime-java (1.17-1) ... Selecting previously unselected package xorg-sgml-doctools. Preparing to unpack .../14-xorg-sgml-doctools_1%3a1.11-1_all.deb ... Unpacking xorg-sgml-doctools (1:1.11-1) ... Selecting previously unselected package x11proto-dev. Preparing to unpack .../15-x11proto-dev_2019.2-1ubuntu1_all.deb ... Unpacking x11proto-dev (2019.2-1ubuntu1) ... Selecting previously unselected package x11proto-core-dev. Preparing to unpack .../16-x11proto-core-dev_2019.2-1ubuntu1_all.deb ... Unpacking x11proto-core-dev (2019.2-1ubuntu1) ... Selecting previously unselected package libice-dev:amd64. Preparing to unpack .../17-libice-dev_2%3a1.0.10-0ubuntu1_amd64.deb ... Unpacking libice-dev:amd64 (2:1.0.10-0ubuntu1) ... Selecting previously unselected package libjansi-native-java. Preparing to unpack .../18-libjansi-native-java_1.8-1_all.deb ... Unpacking libjansi-native-java (1.8-1) ... Selecting previously unselected package libjansi-java. Preparing to unpack .../19-libjansi-java_1.18-1_all.deb ... Unpacking libjansi-java (1.18-1) ... Selecting previously unselected package libjline2-java. Preparing to unpack .../20-libjline2-java_2.14.6-3_all.deb ... Unpacking libjline2-java (2.14.6-3) ... Selecting previously unselected package libpthread-stubs0-dev:amd64. Preparing to unpack .../21-libpthread-stubs0-dev_0.4-1_amd64.deb ... Unpacking libpthread-stubs0-dev:amd64 (0.4-1) ... Selecting previously unselected package libsm-dev:amd64. Preparing to unpack .../22-libsm-dev_2%3a1.2.3-1_amd64.deb ... Unpacking libsm-dev:amd64 (2:1.2.3-1) ... Selecting previously unselected package libxau-dev:amd64. Preparing to unpack .../23-libxau-dev_1%3a1.0.9-0ubuntu1_amd64.deb ... Unpacking libxau-dev:amd64 (1:1.0.9-0ubuntu1) ... Selecting previously unselected package libxdmcp-dev:amd64. Preparing to unpack .../24-libxdmcp-dev_1%3a1.1.3-0ubuntu1_amd64.deb ... Unpacking libxdmcp-dev:amd64 (1:1.1.3-0ubuntu1) ... Selecting previously unselected package xtrans-dev. Preparing to unpack .../25-xtrans-dev_1.4.0-1_all.deb ... Unpacking xtrans-dev (1.4.0-1) ... Selecting previously unselected package libxcb1-dev:amd64. Preparing to unpack .../26-libxcb1-dev_1.14-2_amd64.deb ... Unpacking libxcb1-dev:amd64 (1.14-2) ... Selecting previously unselected package libx11-dev:amd64. Preparing to unpack .../27-libx11-dev_2%3a1.6.9-2ubuntu1.2_amd64.deb ... Unpacking libx11-dev:amd64 (2:1.6.9-2ubuntu1.2) ... Selecting previously unselected package libxt-dev:amd64. Preparing to unpack .../28-libxt-dev_1%3a1.1.5-1_amd64.deb ... Unpacking libxt-dev:amd64 (1:1.1.5-1) ... Selecting previously unselected package scala-library. Preparing to unpack .../29-scala-library_2.11.12-4_all.deb ... Unpacking scala-library (2.11.12-4) ... Selecting previously unselected package scala-parser-combinators. Preparing to unpack .../30-scala-parser-combinators_1.0.3-3_all.deb ... Unpacking scala-parser-combinators (1.0.3-3) ... Selecting previously unselected package scala-xml. Preparing to unpack .../31-scala-xml_1.0.3-3_all.deb ... Unpacking scala-xml (1.0.3-3) ... Selecting previously unselected package scala. Preparing to unpack .../32-scala_2.11.12-4_all.deb ... Unpacking scala (2.11.12-4) ... java-common (0.72) 설정하는 중입니다 ... scala-library (2.11.12-4) 설정하는 중입니다 ... scala-xml (1.0.3-3) 설정하는 중입니다 ... libpthread-stubs0-dev:amd64 (0.4-1) 설정하는 중입니다 ... xtrans-dev (1.4.0-1) 설정하는 중입니다 ... scala-parser-combinators (1.0.3-3) 설정하는 중입니다 ... fonts-dejavu-extra (2.37-1) 설정하는 중입니다 ... xorg-sgml-doctools (1:1.11-1) 설정하는 중입니다 ... libatk-wrapper-java (0.37.1-1) 설정하는 중입니다 ... libhawtjni-runtime-java (1.17-1) 설정하는 중입니다 ... libatk-wrapper-java-jni:amd64 (0.37.1-1) 설정하는 중입니다 ... libjansi-native-java (1.8-1) 설정하는 중입니다 ... libjansi-java (1.18-1) 설정하는 중입니다 ... libjline2-java (2.14.6-3) 설정하는 중입니다 ... default-jre-headless (2:1.11-72) 설정하는 중입니다 ... openjdk-11-jre-headless:amd64 (11.0.11+9-0ubuntu2~20.04) 설정하는 중입니다 ... update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/java to provid e /usr/bin/java (java) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jjs to provide  /usr/bin/jjs (jjs) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/keytool to pro vide /usr/bin/keytool (keytool) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/rmid to provid e /usr/bin/rmid (rmid) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/rmiregistry to  provide /usr/bin/rmiregistry (rmiregistry) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/pack200 to pro vide /usr/bin/pack200 (pack200) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/unpack200 to p rovide /usr/bin/unpack200 (unpack200) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/lib/jexec to provi de /usr/bin/jexec (jexec) in auto mode openjdk-11-jre:amd64 (11.0.11+9-0ubuntu2~20.04) 설정하는 중입니다 ... openjdk-11-jdk-headless:amd64 (11.0.11+9-0ubuntu2~20.04) 설정하는 중입니다 ... update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jar to provide  /usr/bin/jar (jar) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jarsigner to p rovide /usr/bin/jarsigner (jarsigner) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/javac to provi de /usr/bin/javac (javac) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/javadoc to pro vide /usr/bin/javadoc (javadoc) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/javap to provi de /usr/bin/javap (javap) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jcmd to provid e /usr/bin/jcmd (jcmd) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jdb to provide  /usr/bin/jdb (jdb) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jdeprscan to p rovide /usr/bin/jdeprscan (jdeprscan) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jdeps to provi de /usr/bin/jdeps (jdeps) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jfr to provide  /usr/bin/jfr (jfr) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jimage to prov ide /usr/bin/jimage (jimage) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jinfo to provi de /usr/bin/jinfo (jinfo) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jlink to provi de /usr/bin/jlink (jlink) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jmap to provid e /usr/bin/jmap (jmap) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jmod to provid e /usr/bin/jmod (jmod) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jps to provide  /usr/bin/jps (jps) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jrunscript to  provide /usr/bin/jrunscript (jrunscript) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jshell to prov ide /usr/bin/jshell (jshell) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jstack to prov ide /usr/bin/jstack (jstack) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jstat to provi de /usr/bin/jstat (jstat) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jstatd to prov ide /usr/bin/jstatd (jstatd) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/rmic to provid e /usr/bin/rmic (rmic) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/serialver to p rovide /usr/bin/serialver (serialver) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jaotc to provi de /usr/bin/jaotc (jaotc) in auto mode update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jhsdb to provi de /usr/bin/jhsdb (jhsdb) in auto mode default-jre (2:1.11-72) 설정하는 중입니다 ... default-jdk-headless (2:1.11-72) 설정하는 중입니다 ... openjdk-11-jdk:amd64 (11.0.11+9-0ubuntu2~20.04) 설정하는 중입니다 ... update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jconsole to pr ovide /usr/bin/jconsole (jconsole) in auto mode ca-certificates-java (20190405ubuntu1) 설정하는 중입니다 ... head: 읽기를 위해 '/etc/ssl/certs/java/cacerts'을(를) 열 수 없음: 그런 파일이나  디렉터리가 없습니다 Adding debian:Microsoft_RSA_Root_Certificate_Authority_2017.pem Adding debian:Actalis_Authentication_Root_CA.pem Adding debian:GlobalSign_ECC_Root_CA_-_R5.pem Adding debian:DigiCert_Global_Root_CA.pem Adding debian:SecureSign_RootCA11.pem Adding debian:QuoVadis_Root_CA_2.pem Adding debian:GlobalSign_ECC_Root_CA_-_R4.pem Adding debian:Entrust_Root_Certification_Authority_-_G4.pem Adding debian:COMODO_ECC_Certification_Authority.pem Adding debian:Buypass_Class_3_Root_CA.pem Adding debian:QuoVadis_Root_CA_1_G3.pem Adding debian:VeriSign_Universal_Root_Certification_Authority.pem Adding debian:Hongkong_Post_Root_CA_3.pem Adding debian:GDCA_TrustAUTH_R5_ROOT.pem Adding debian:GlobalSign_Root_CA.pem Adding debian:certSIGN_Root_CA_G2.pem Adding debian:Amazon_Root_CA_3.pem Adding debian:AffirmTrust_Premium.pem Adding debian:CFCA_EV_ROOT.pem Adding debian:SSL.com_Root_Certification_Authority_RSA.pem Adding debian:QuoVadis_Root_CA_3.pem Adding debian:Amazon_Root_CA_1.pem Adding debian:SZAFIR_ROOT_CA2.pem Adding debian:Entrust_Root_Certification_Authority_-_G2.pem Adding debian:SecureTrust_CA.pem Adding debian:Trustwave_Global_ECC_P256_Certification_Authority.pem Adding debian:Certum_Trusted_Network_CA.pem Adding debian:Starfield_Services_Root_Certificate_Authority_-_G2.pem Adding debian:Staat_der_Nederlanden_EV_Root_CA.pem Adding debian:e-Szigno_Root_CA_2017.pem Adding debian:Security_Communication_Root_CA.pem Adding debian:GlobalSign_Root_CA_-_R3.pem Adding debian:Certigna_Root_CA.pem Adding debian:USERTrust_RSA_Certification_Authority.pem Adding debian:DigiCert_High_Assurance_EV_Root_CA.pem Adding debian:SSL.com_EV_Root_Certification_Authority_ECC.pem Adding debian:Trustis_FPS_Root_CA.pem Adding debian:Secure_Global_CA.pem Adding debian:TWCA_Global_Root_CA.pem Adding debian:OISTE_WISeKey_Global_Root_GC_CA.pem Adding debian:Comodo_AAA_Services_root.pem Adding debian:Buypass_Class_2_Root_CA.pem Adding debian:Go_Daddy_Root_Certificate_Authority_-_G2.pem Adding debian:DigiCert_Assured_ID_Root_CA.pem Adding debian:DigiCert_Assured_ID_Root_G3.pem Adding debian:QuoVadis_Root_CA_3_G3.pem Adding debian:TrustCor_RootCert_CA-2.pem Adding debian:IdenTrust_Commercial_Root_CA_1.pem Adding debian:Staat_der_Nederlanden_Root_CA_-_G3.pem Adding debian:TUBITAK_Kamu_SM_SSL_Kok_Sertifikasi_-_Surum_1.pem Adding debian:D-TRUST_Root_Class_3_CA_2_EV_2009.pem Adding debian:AffirmTrust_Networking.pem Adding debian:UCA_Extended_Validation_Root.pem Adding debian:emSign_ECC_Root_CA_-_G3.pem Adding debian:GTS_Root_R3.pem Adding debian:AC_RAIZ_FNMT-RCM.pem Adding debian:Amazon_Root_CA_4.pem Adding debian:E-Tugra_Certification_Authority.pem Adding debian:Chambers_of_Commerce_Root_-_2008.pem Adding debian:Microsoft_ECC_Root_Certificate_Authority_2017.pem Adding debian:GeoTrust_Primary_Certification_Authority_-_G2.pem Adding debian:Network_Solutions_Certificate_Authority.pem Adding debian:TrustCor_RootCert_CA-1.pem Adding debian:Cybertrust_Global_Root.pem Adding debian:Certigna.pem Adding debian:QuoVadis_Root_CA_2_G3.pem Adding debian:GTS_Root_R4.pem Adding debian:ACCVRAIZ1.pem Adding debian:Starfield_Class_2_CA.pem Adding debian:T-TeleSec_GlobalRoot_Class_2.pem Adding debian:Go_Daddy_Class_2_CA.pem Adding debian:DigiCert_Trusted_Root_G4.pem Adding debian:Sonera_Class_2_Root_CA.pem Adding debian:NetLock_Arany_=Class_Gold=_Főtanúsítvány.pem Adding debian:T-TeleSec_GlobalRoot_Class_3.pem Adding debian:OISTE_WISeKey_Global_Root_GB_CA.pem Adding debian:Entrust_Root_Certification_Authority_-_EC1.pem Adding debian:emSign_Root_CA_-_G1.pem Adding debian:Atos_TrustedRoot_2011.pem Adding debian:DigiCert_Global_Root_G2.pem Adding debian:Hellenic_Academic_and_Research_Institutions_ECC_RootCA_2015.pem Adding debian:emSign_Root_CA_-_C1.pem Adding debian:GTS_Root_R1.pem Adding debian:Baltimore_CyberTrust_Root.pem Adding debian:AffirmTrust_Commercial.pem Adding debian:Trustwave_Global_ECC_P384_Certification_Authority.pem Adding debian:Hellenic_Academic_and_Research_Institutions_RootCA_2011.pem Adding debian:SSL.com_Root_Certification_Authority_ECC.pem Adding debian:Microsec_e-Szigno_Root_CA_2009.pem Adding debian:USERTrust_ECC_Certification_Authority.pem Adding debian:AffirmTrust_Premium_ECC.pem Adding debian:COMODO_Certification_Authority.pem Adding debian:SwissSign_Silver_CA_-_G2.pem Adding debian:QuoVadis_Root_CA.pem Adding debian:GlobalSign_Root_CA_-_R6.pem Adding debian:Hellenic_Academic_and_Research_Institutions_RootCA_2015.pem Adding debian:Entrust.net_Premium_2048_Secure_Server_CA.pem Adding debian:ePKI_Root_Certification_Authority.pem Adding debian:Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.pem Adding debian:ISRG_Root_X1.pem Adding debian:emSign_ECC_Root_CA_-_C3.pem Adding debian:Global_Chambersign_Root_-_2008.pem Adding debian:NAVER_Global_Root_Certification_Authority.pem Adding debian:XRamp_Global_CA_Root.pem Adding debian:GTS_Root_R2.pem Adding debian:DigiCert_Assured_ID_Root_G2.pem Adding debian:GlobalSign_Root_CA_-_R2.pem Adding debian:Trustwave_Global_Certification_Authority.pem Adding debian:Security_Communication_RootCA2.pem Adding debian:Hongkong_Post_Root_CA_1.pem Adding debian:Certum_Trusted_Network_CA_2.pem Adding debian:Izenpe.com.pem Adding debian:Amazon_Root_CA_2.pem Adding debian:D-TRUST_Root_Class_3_CA_2_2009.pem Adding debian:IdenTrust_Public_Sector_Root_CA_1.pem Adding debian:Entrust_Root_Certification_Authority.pem Adding debian:certSIGN_ROOT_CA.pem Adding debian:TrustCor_ECA-1.pem Adding debian:EC-ACC.pem Adding debian:CA_Disig_Root_R2.pem Adding debian:COMODO_RSA_Certification_Authority.pem Adding debian:DigiCert_Global_Root_G3.pem Adding debian:DST_Root_CA_X3.pem Adding debian:TWCA_Root_Certification_Authority.pem Adding debian:Starfield_Root_Certificate_Authority_-_G2.pem Adding debian:SSL.com_EV_Root_Certification_Authority_RSA_R2.pem Adding debian:ssl-cert-snakeoil.pem Adding debian:TeliaSonera_Root_CA_v1.pem Adding debian:UCA_Global_G2_Root.pem Adding debian:SwissSign_Gold_CA_-_G2.pem done. scala (2.11.12-4) 설정하는 중입니다 ... update-alternatives: using /usr/share/scala-2.11/bin/scala to provide /usr/bin/s cala (scala) in auto mode default-jdk (2:1.11-72) 설정하는 중입니다 ... Processing triggers for man-db (2.9.1-1) ... Processing triggers for ca-certificates (20210119~20.04.1) ... Updating certificates in /etc/ssl/certs... 0 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d... done. done. Processing triggers for sgml-base (1.29.1) ... x11proto-dev (2019.2-1ubuntu1) 설정하는 중입니다 ... Processing triggers for fontconfig (2.13.1-2ubuntu3) ... Processing triggers for desktop-file-utils (0.24-1ubuntu3) ... Processing triggers for mime-support (3.64ubuntu1) ... libxau-dev:amd64 (1:1.0.9-0ubuntu1) 설정하는 중입니다 ... Processing triggers for hicolor-icon-theme (0.17-2) ... libice-dev:amd64 (2:1.0.10-0ubuntu1) 설정하는 중입니다 ... Processing triggers for gnome-menus (3.36.0-1ubuntu1) ... libsm-dev:amd64 (2:1.2.3-1) 설정하는 중입니다 ... libxdmcp-dev:amd64 (1:1.1.3-0ubuntu1) 설정하는 중입니다 ... x11proto-core-dev (2019.2-1ubuntu1) 설정하는 중입니다 ... libxcb1-dev:amd64 (1.14-2) 설정하는 중입니다 ... libx11-dev:amd64 (2:1.6.9-2ubuntu1.2) 설정하는 중입니다 ... libxt-dev:amd64 (1:1.1.5-1) 설정하는 중입니다 ... ninipa@ninipa-VirtualBox:~$  ninipa@ninipa-VirtualBox:~$ java --version openjdk 11.0.11 2021-04-20 OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04) OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing) ninipa@ninipa-VirtualBox:~$ scala -version Scala code runner version 2.11.12 -- Copyright 2002-2017, LAMP/EPFL ninipa@ninipa-VirtualBox:~$

2. Apache Spark 다운로드

ninipa@ninipa-VirtualBox:~$ wget https://downloads.apache.org/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz --2021-07-27 20:05:04--  https://downloads.apache.org/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz downloads.apache.org (downloads.apache.org)을(를) 해석하는 중... 135.181.209.10, 88.99.95.219, 135.181.214.104, ... 접속 downloads.apache.org (downloads.apache.org)|135.181.209.10|:443... 접속됨. HTTP 요청을 전송했습니다. 응답을 기다리는 중입니다... 200 OK 길이: 228834641 (218M) [application/x-gzip] 다음 위치에 저장: `spark-3.1.2-bin-hadoop3.2.tgz' spark-3.1.2-bin-had   3%[                    ]   7.54M  2.40MB/s    남은 시간 88park-3.1.2-bin-hado   4%[                    ]   8.74M  2.42MB/s    남은 시간 88ark-3.1.2-bin-hadoo   4%[                    ]   9.40M  2.45MB/s    남은 시간 88rk-3.1.2-bin-hadoop   4%[                    ]  10.34M  2.53MB/s    남은 시간 88k-3.1.2-bin-hadoop3   5%[>                   ]  12.38M  2.89MB/s    남은 시간 71-3.1.2-bin-hadoop3.   5%[>                   ]  13.04M  2.87MB/s    남은 시간 713.1.2-bin-hadoop3.2   6%[>                   ]  13.73M  2.76MB/s    남은 시간 71.1.2-bin-hadoop3.2.   6%[>                   ]  14.10M  2.73MB/s    남은 시간 711.2-bin-hadoop3.2.t   6%[>                   ]  14.59M  2.88MB/s    남은 시간 75.2-bin-hadoop3.2.tg   6%[>                   ]  15.12M  3.22MB/s    남은 시간 752-bin-hadoop3.2.tgz   7%[>                   ]  16.65M  3.61MB/s    남은 시간 75-bin-hadoop3.2.tgz    7%[>                   ]  16.91M  3.72MB/s    남은 시간 75bin-hadoop3.2.tgz     8%[>                   ]  19.15M  4.46MB/s    남은 시간 75in-hadoop3.2.tgz      8%[>                   ]  19.18M  4.08MB/s    남은 시간 69n-hadoop3.2.tgz       9%[>                   ]  20.59M  4.02MB/s    남은 시간 69-hadoop3.2.tgz       10%[=>                  ]  22.18M  3.68MB/s    남은 시간 69hadoop3.2.tgz        10%[=>                  ]  22.57M  3.80MB/s    남은 시간 69adoop3.2.tgz         10%[=>                  ]  23.63M  4.00MB/s    남은 시간 69doop3.2.tgz          10%[=>                  ]  23.91M  3.51MB/s    남은 시간 65oop3.2.tgz           11%[=>                  ]  24.41M  3.46MB/s    남은 시간 65op3.2.tgz            12%[=>                  ]  26.59M  3.54MB/s    남은 시간 65p3.2.tgz             12%[=>                  ]  27.65M  3.62MB/s    남은 시간 653.2.tgz              12%[=>                  ]  28.13M  3.46MB/s    남은 시간 62.2.tgz               13%[=>                  ]  29.74M  3.71MB/s    남은 시간 622.tgz                14%[=>                  ]  31.15M  3.95MB/s    남은 시간 62.tgz                 14%[=>                  ]  31.85M  3.99MB/s    남은 시간 62tgz                  14%[=>                  ]  32.63M  3.81MB/s    남은 시간 62gz                   15%[==>                 ]  33.30M  3.85MB/s    남은 시간 57z                    15%[==>                 ]  34.09M  3.56MB/s    남은 시간 57                     15%[==>                 ]  34.84M  3.95MB/s    남은 시간 57                  s  16%[==>                 ]  35.55M  3.76MB/s    남은 시간 57                 sp  16%[==>                 ]  36.26M  3.57MB/s    남은 시간 57                spa  16%[==>                 ]  37.01M  3.64MB/s    남은 시간 55               spar  17%[==>                 ]  37.76M  3.56MB/s    남은 시간 55              spark  17%[==>                 ]  38.48M  4.06MB/s    남은 시간 55             spark-  17%[==>                 ]  39.18M  3.65MB/s    남은 시간 55            spark-3  18%[==>                 ]  39.88M  3.17MB/s    남은 시간 56           spark-3.  19%[==>                 ]  42.05M  3.98MB/s    남은 시간 56          spark-3.1  19%[==>                 ]  42.46M  3.66MB/s    남은 시간 56         spark-3.1.  19%[==>                 ]  43.40M  3.51MB/s    남은 시간 56        spark-3.1.2  20%[===>                ]  44.24M  3.58MB/s    남은 시간 56       spark-3.1.2-  20%[===>                ]  44.63M  3.45MB/s    남은 시간 52      spark-3.1.2-b  20%[===>                ]  45.62M  3.52MB/s    남은 시간 52     spark-3.1.2-bi  21%[===>                ]  46.43M  3.55MB/s    남은 시간 52    spark-3.1.2-bin  21%[===>                ]  46.90M  3.47MB/s    남은 시간 52   spark-3.1.2-bin-  21%[===>                ]  47.91M  3.55MB/s    남은 시간 52  spark-3.1.2-bin-h  22%[===>                ]  48.57M  3.54MB/s    남은 시간 51 spark-3.1.2-bin-ha  22%[===>                ]  49.24M  3.52MB/s    남은 시간 51spark-3.1.2-bin-had  22%[===>                ]  50.13M  3.56MB/s    남은 시간 51park-3.1.2-bin-hado  23%[===>                ]  50.84M  3.55MB/s    남은 시간 51ark-3.1.2-bin-hadoo  23%[===>                ]  51.59M  3.56MB/s    남은 시간 51rk-3.1.2-bin-hadoop  23%[===>                ]  52.30M  3.56MB/s    남은 시간 49k-3.1.2-bin-hadoop3  24%[===>                ]  53.05M  3.93MB/s    남은 시간 49-3.1.2-bin-hadoop3.  24%[===>                ]  53.76M  3.60MB/s    남은 시간 493.1.2-bin-hadoop3.2  24%[===>                ]  54.48M  3.70MB/s    남은 시간 49.1.2-bin-hadoop3.2.  25%[====>               ]  55.26M  3.73MB/s    남은 시간 491.2-bin-hadoop3.2.t  25%[====>               ]  55.98M  3.61MB/s    남은 시간 48.2-bin-hadoop3.2.tg  25%[====>               ]  56.68M  3.71MB/s    남은 시간 482-bin-hadoop3.2.tgz  26%[====>               ]  57.40M  3.69MB/s    남은 시간 48-bin-hadoop3.2.tgz   26%[====>               ]  58.18M  3.59MB/s    남은 시간 48bin-hadoop3.2.tgz    26%[====>               ]  58.88M  3.68MB/s    남은 시간 48in-hadoop3.2.tgz     27%[====>               ]  59.60M  3.60MB/s    남은 시간 47n-hadoop3.2.tgz      27%[====>               ]  60.30M  3.55MB/s    남은 시간 47-hadoop3.2.tgz       27%[====>               ]  61.05M  3.55MB/s    남은 시간 47hadoop3.2.tgz        28%[====>               ]  61.80M  3.57MB/s    남은 시간 47adoop3.2.tgz         28%[====>               ]  62.52M  3.56MB/s    남은 시간 47doop3.2.tgz          28%[====>               ]  63.23M  3.54MB/s    남은 시간 46oop3.2.tgz           29%[====>               ]  63.98M  3.55MB/s    남은 시간 46op3.2.tgz            29%[====>               ]  64.73M  3.57MB/s    남은 시간 46p3.2.tgz             29%[====>               ]  65.45M  3.56MB/s    남은 시간 463.2.tgz              30%[=====>              ]  66.15M  3.55MB/s    남은 시간 46.2.tgz               30%[=====>              ]  66.90M  3.56MB/s    남은 시간 442.tgz                30%[=====>              ]  67.65M  3.56MB/s    남은 시간 44.tgz                 31%[=====>              ]  68.35M  3.56MB/s    남은 시간 44tgz                  31%[=====>              ]  69.07M  3.56MB/s    남은 시간 44gz                   31%[=====>              ]  69.82M  3.57MB/s    남은 시간 44z                    32%[=====>              ]  70.52M  3.56MB/s    남은 시간 43                     32%[=====>              ]  71.27M  3.56MB/s    남은 시간 43                  s  32%[=====>              ]  71.99M  3.56MB/s    남은 시간 43                 sp  33%[=====>              ]  72.74M  3.55MB/s    남은 시간 43                spa  33%[=====>              ]  73.45M  3.55MB/s    남은 시간 43               spar  33%[=====>              ]  74.20M  3.56MB/s    남은 시간 42              spark  34%[=====>              ]  74.91M  3.57MB/s    남은 시간 42             spark-  34%[=====>              ]  75.65M  3.58MB/s    남은 시간 42            spark-3  34%[=====>              ]  76.37M  3.55MB/s    남은 시간 42           spark-3.  35%[======>             ]  77.07M  3.55MB/s    남은 시간 42          spark-3.1  35%[======>             ]  77.82M  3.56MB/s    남은 시간 41         spark-3.1.  36%[======>             ]  78.57M  3.56MB/s    남은 시간 41        spark-3.1.2  36%[======>             ]  79.27M  3.55MB/s    남은 시간 41       spark-3.1.2-  36%[======>             ]  79.99M  3.55MB/s    남은 시간 41      spark-3.1.2-b  36%[======>             ]  80.74M  3.56MB/s    남은 시간 41     spark-3.1.2-bi  37%[======>             ]  81.45M  3.56MB/s    남은 시간 40    spark-3.1.2-bin  37%[======>             ]  82.18M  3.55MB/s    남은 시간 40   spark-3.1.2-bin-  37%[======>             ]  82.90M  3.54MB/s    남은 시간 40  spark-3.1.2-bin-h  38%[======>             ]  83.63M  3.55MB/s    남은 시간 40 spark-3.1.2-bin-ha  38%[======>             ]  84.38M  3.56MB/s    남은 시간 40spark-3.1.2-bin-had  38%[======>             ]  85.10M  3.55MB/s    남은 시간 39park-3.1.2-bin-hado  39%[======>             ]  85.80M  3.54MB/s    남은 시간 39ark-3.1.2-bin-hadoo  39%[======>             ]  86.52M  3.54MB/s    남은 시간 39rk-3.1.2-bin-hadoop  40%[=======>            ]  87.30M  3.53MB/s    남은 시간 39k-3.1.2-bin-hadoop3  40%[=======>            ]  88.09M  3.55MB/s    남은 시간 39-3.1.2-bin-hadoop3.  40%[=======>            ]  88.84M  3.55MB/s    남은 시간 383.1.2-bin-hadoop3.2  41%[=======>            ]  89.54M  3.55MB/s    남은 시간 38.1.2-bin-hadoop3.2.  41%[=======>            ]  90.27M  3.55MB/s    남은 시간 381.2-bin-hadoop3.2.t  41%[=======>            ]  90.99M  3.55MB/s    남은 시간 38.2-bin-hadoop3.2.tg  42%[=======>            ]  91.71M  3.55MB/s    남은 시간 382-bin-hadoop3.2.tgz  42%[=======>            ]  92.41M  3.54MB/s    남은 시간 36-bin-hadoop3.2.tgz   42%[=======>            ]  93.16M  3.56MB/s    남은 시간 36bin-hadoop3.2.tgz    43%[=======>            ]  93.88M  3.56MB/s    남은 시간 36in-hadoop3.2.tgz     43%[=======>            ]  94.59M  3.55MB/s    남은 시간 36n-hadoop3.2.tgz      43%[=======>            ]  95.30M  3.55MB/s    남은 시간 36-hadoop3.2.tgz       43%[=======>            ]  95.54M  3.12MB/s    남은 시간 36hadoop3.2.tgz        44%[=======>            ]  97.71M  3.55MB/s    남은 시간 36adoop3.2.tgz         44%[=======>            ]  98.15M  3.47MB/s    남은 시간 36doop3.2.tgz          45%[========>           ]  99.09M  3.53MB/s    남은 시간 36oop3.2.tgz           45%[========>           ]  99.77M  3.52MB/s    남은 시간 36op3.2.tgz            45%[========>           ] 100.37M  3.47MB/s    남은 시간 34p3.2.tgz             46%[========>           ] 101.38M  3.56MB/s    남은 시간 343.2.tgz              46%[========>           ] 101.96M  3.51MB/s    남은 시간 34.2.tgz               47%[========>           ] 102.60M  3.48MB/s    남은 시간 342.tgz                47%[========>           ] 103.55M  3.55MB/s    남은 시간 34.tgz                 47%[========>           ] 104.30M  3.57MB/s    남은 시간 33tgz                  48%[========>           ] 105.01M  3.56MB/s    남은 시간 33gz                   48%[========>           ] 105.73M  3.56MB/s    남은 시간 33z                    48%[========>           ] 106.43M  3.55MB/s    남은 시간 33                     49%[========>           ] 107.18M  3.56MB/s    남은 시간 33                  s  49%[========>           ] 107.90M  3.57MB/s    남은 시간 32                 sp  49%[========>           ] 108.60M  3.94MB/s    남은 시간 32                spa  50%[=========>          ] 109.32M  3.59MB/s    남은 시간 32               spar  50%[=========>          ] 110.07M  3.70MB/s    남은 시간 32              spark  50%[=========>          ] 110.77M  3.70MB/s    남은 시간 32             spark-  51%[=========>          ] 111.49M  3.57MB/s    남은 시간 31            spark-3  51%[=========>          ] 112.20M  3.67MB/s    남은 시간 31           spark-3.  51%[=========>          ] 112.91M  3.64MB/s    남은 시간 31          spark-3.1  52%[=========>          ] 113.70M  3.56MB/s    남은 시간 31         spark-3.1.  52%[=========>          ] 114.40M  3.60MB/s    남은 시간 31        spark-3.1.2  52%[=========>          ] 115.12M  3.56MB/s    남은 시간 30       spark-3.1.2-  53%[=========>          ] 115.82M  3.55MB/s    남은 시간 30      spark-3.1.2-b  53%[=========>          ] 116.57M  3.56MB/s    남은 시간 30     spark-3.1.2-bi  53%[=========>          ] 117.29M  3.57MB/s    남은 시간 30    spark-3.1.2-bin  54%[=========>          ] 117.99M  3.56MB/s    남은 시간 30   spark-3.1.2-bin-  54%[=========>          ] 118.02M  3.13MB/s    남은 시간 29  spark-3.1.2-bin-h  54%[=========>          ] 119.10M  3.24MB/s    남은 시간 29 spark-3.1.2-bin-ha  55%[==========>         ] 120.59M  3.45MB/s    남은 시간 29spark-3.1.2-bin-had  55%[==========>         ] 121.29M  3.45MB/s    남은 시간 29park-3.1.2-bin-hado  55%[==========>         ] 121.77M  3.36MB/s    남은 시간 29ark-3.1.2-bin-hadoo  56%[==========>         ] 122.74M  3.43MB/s    남은 시간 28rk-3.1.2-bin-hadoop  56%[==========>         ] 123.59M  3.47MB/s    남은 시간 28k-3.1.2-bin-hadoop3  56%[==========>         ] 123.98M  3.38MB/s    남은 시간 28-3.1.2-bin-hadoop3.  57%[==========>         ] 124.98M  3.45MB/s    남은 시간 283.1.2-bin-hadoop3.2  57%[==========>         ] 125.60M  3.41MB/s    남은 시간 28.1.2-bin-hadoop3.2.  57%[==========>         ] 126.26M  3.40MB/s    남은 시간 271.2-bin-hadoop3.2.t  58%[==========>         ] 127.29M  3.49MB/s    남은 시간 27.2-bin-hadoop3.2.tg  58%[==========>         ] 127.93M  3.47MB/s    남은 시간 272-bin-hadoop3.2.tgz  58%[==========>         ] 128.60M  3.45MB/s    남은 시간 27-bin-hadoop3.2.tgz   59%[==========>         ] 129.65M  3.55MB/s    남은 시간 27bin-hadoop3.2.tgz    59%[==========>         ] 130.09M  3.92MB/s    남은 시간 25in-hadoop3.2.tgz     60%[===========>        ] 131.01M  3.90MB/s    남은 시간 25n-hadoop3.2.tgz      60%[===========>        ] 131.82M  3.59MB/s    남은 시간 25-hadoop3.2.tgz       60%[===========>        ] 132.52M  3.60MB/s    남은 시간 25hadoop3.2.tgz        61%[===========>        ] 133.24M  3.72MB/s    남은 시간 25adoop3.2.tgz         61%[===========>        ] 134.01M  3.69MB/s    남은 시간 24doop3.2.tgz          61%[===========>        ] 134.73M  3.69MB/s    남은 시간 24oop3.2.tgz           62%[===========>        ] 135.45M  3.72MB/s    남은 시간 24op3.2.tgz            62%[===========>        ] 136.15M  3.71MB/s    남은 시간 24p3.2.tgz             62%[===========>        ] 136.90M  3.71MB/s    남은 시간 243.2.tgz              63%[===========>        ] 137.62M  3.69MB/s    남은 시간 23.2.tgz               63%[===========>        ] 138.32M  3.69MB/s    남은 시간 232.tgz                63%[===========>        ] 139.04M  3.66MB/s    남은 시간 23.tgz                 64%[===========>        ] 139.79M  3.64MB/s    남은 시간 23tgz                  64%[===========>        ] 140.49M  3.67MB/s    남은 시간 23gz                   64%[===========>        ] 141.21M  3.60MB/s    남은 시간 22z                    65%[============>       ] 141.93M  3.56MB/s    남은 시간 22                     65%[============>       ] 142.63M  3.60MB/s    남은 시간 22                  s  65%[============>       ] 143.38M  3.56MB/s    남은 시간 22                 sp  66%[============>       ] 144.09M  3.56MB/s    남은 시간 22                spa  66%[============>       ] 144.80M  3.55MB/s    남은 시간 21               spar  66%[============>       ] 145.52M  3.56MB/s    남은 시간 21              spark  67%[============>       ] 146.26M  3.57MB/s    남은 시간 21             spark-  67%[============>       ] 146.98M  3.60MB/s    남은 시간 21            spark-3  67%[============>       ] 147.68M  3.56MB/s    남은 시간 21           spark-3.  67%[============>       ] 148.40M  3.55MB/s    남은 시간 20          spark-3.1  68%[============>       ] 149.15M  3.57MB/s    남은 시간 20         spark-3.1.  68%[============>       ] 149.85M  3.56MB/s    남은 시간 20        spark-3.1.2  68%[============>       ] 150.57M  3.55MB/s    남은 시간 20       spark-3.1.2-  69%[============>       ] 151.29M  3.56MB/s    남은 시간 20      spark-3.1.2-b  69%[============>       ] 151.99M  3.55MB/s    남은 시간 19     spark-3.1.2-bi  69%[============>       ] 152.74M  3.56MB/s    남은 시간 19    spark-3.1.2-bin  70%[=============>      ] 153.45M  3.56MB/s    남은 시간 19   spark-3.1.2-bin-  70%[=============>      ] 154.16M  3.55MB/s    남은 시간 19  spark-3.1.2-bin-h  70%[=============>      ] 154.88M  3.55MB/s    남은 시간 19 spark-3.1.2-bin-ha  71%[=============>      ] 155.62M  3.56MB/s    남은 시간 18spark-3.1.2-bin-had  71%[=============>      ] 156.37M  3.56MB/s    남은 시간 18park-3.1.2-bin-hado  71%[=============>      ] 157.09M  3.55MB/s    남은 시간 18ark-3.1.2-bin-hadoo  72%[=============>      ] 157.79M  3.55MB/s    남은 시간 18rk-3.1.2-bin-hadoop  72%[=============>      ] 158.54M  3.56MB/s    남은 시간 18k-3.1.2-bin-hadoop3  72%[=============>      ] 159.26M  3.56MB/s    남은 시간 17-3.1.2-bin-hadoop3.  73%[=============>      ] 160.01M  3.56MB/s    남은 시간 173.1.2-bin-hadoop3.2  73%[=============>      ] 160.71M  3.55MB/s    남은 시간 17.1.2-bin-hadoop3.2.  73%[=============>      ] 161.43M  3.55MB/s    남은 시간 171.2-bin-hadoop3.2.t  74%[=============>      ] 162.21M  3.56MB/s    남은 시간 17.2-bin-hadoop3.2.tg  74%[=============>      ] 162.91M  3.56MB/s    남은 시간 162-bin-hadoop3.2.tgz  74%[=============>      ] 163.63M  3.56MB/s    남은 시간 16-bin-hadoop3.2.tgz   75%[==============>     ] 164.35M  3.55MB/s    남은 시간 16bin-hadoop3.2.tgz    75%[==============>     ] 165.13M  3.57MB/s    남은 시간 16in-hadoop3.2.tgz     75%[==============>     ] 165.84M  3.56MB/s    남은 시간 16n-hadoop3.2.tgz      76%[==============>     ] 166.54M  3.55MB/s    남은 시간 15-hadoop3.2.tgz       76%[==============>     ] 167.26M  3.55MB/s    남은 시간 15hadoop3.2.tgz        76%[==============>     ] 168.04M  3.57MB/s    남은 시간 15adoop3.2.tgz         77%[==============>     ] 168.74M  3.56MB/s    남은 시간 15doop3.2.tgz          77%[==============>     ] 169.46M  3.56MB/s    남은 시간 15oop3.2.tgz           77%[==============>     ] 170.16M  3.55MB/s    남은 시간 14op3.2.tgz            78%[==============>     ] 170.96M  3.56MB/s    남은 시간 14p3.2.tgz             78%[==============>     ] 171.66M  3.56MB/s    남은 시간 143.2.tgz              78%[==============>     ] 172.38M  3.56MB/s    남은 시간 14.2.tgz               79%[==============>     ] 173.09M  3.55MB/s    남은 시간 142.tgz                79%[==============>     ] 173.84M  3.55MB/s    남은 시간 13.tgz                 79%[==============>     ] 174.59M  3.56MB/s    남은 시간 13tgz                  80%[===============>    ] 175.29M  3.56MB/s    남은 시간 13gz                   80%[===============>    ] 175.46M  3.13MB/s    남은 시간 13z                    80%[===============>    ] 176.59M  3.25MB/s    남은 시간 12                     81%[===============>    ] 178.16M  3.49MB/s    남은 시간 12                  s  81%[===============>    ] 178.84M  3.48MB/s    남은 시간 12                 sp  82%[===============>    ] 179.46M  3.46MB/s    남은 시간 12                spa  82%[===============>    ] 180.41M  3.51MB/s    남은 시간 12               spar  82%[===============>    ] 180.88M  3.45MB/s    남은 시간 11              spark  83%[===============>    ] 181.80M  3.50MB/s    남은 시간 11             spark-  83%[===============>    ] 182.70M  3.55MB/s    남은 시간 11            spark-3  83%[===============>    ] 183.27M  3.51MB/s    남은 시간 11           spark-3.  84%[===============>    ] 184.15M  3.56MB/s    남은 시간 11          spark-3.1  84%[===============>    ] 184.90M  3.56MB/s    남은 시간 10         spark-3.1.  85%[================>   ] 185.62M  3.56MB/s    남은 시간 10        spark-3.1.2  85%[================>   ] 186.32M  3.56MB/s    남은 시간 10       spark-3.1.2-  85%[================>   ] 187.07M  3.56MB/s    남은 시간 10      spark-3.1.2-b  86%[================>   ] 187.79M  3.56MB/s    남은 시간 10spark-3.1.2-bin-had 100%[===================>] 218.23M  3.53MB/s    / 62s      s 2021-07-27 20:06:07 (3.51 MB/s) - `spark-3.1.2-bin-hadoop3.2.tgz' 저장됨 [228834641/228834641] ninipa@ninipa-VirtualBox:~$

3. Spark를 /opt에 풀기

ninipa@ninipa-VirtualBox:~$ sudo mkdir /opt/spark [sudo] ninipa의 암호:  ninipa@ninipa-VirtualBox:~$ sudo tar -xf spark*.tgz -C /opt/spark --strip-component 1 ninipa@ninipa-VirtualBox:~$ ls -al /opt/spark/ 합계 164 drwxr-xr-x 13 root   root    4096  7월 27 20:10 . drwxr-xr-x  6 root   root    4096  7월 27 20:09 .. -rw-r--r--  1 ninipa ninipa 23235  5월 24 13:45 LICENSE -rw-r--r--  1 ninipa ninipa 57677  5월 24 13:45 NOTICE drwxr-xr-x  3 ninipa ninipa  4096  5월 24 13:45 R -rw-r--r--  1 ninipa ninipa  4488  5월 24 13:45 README.md -rw-r--r--  1 ninipa ninipa   183  5월 24 13:45 RELEASE drwxr-xr-x  2 ninipa ninipa  4096  5월 24 13:45 bin drwxr-xr-x  2 ninipa ninipa  4096  5월 24 13:45 conf drwxr-xr-x  5 ninipa ninipa  4096  5월 24 13:45 data drwxr-xr-x  4 ninipa ninipa  4096  5월 24 13:45 examples drwxr-xr-x  2 ninipa ninipa 20480  5월 24 13:45 jars drwxr-xr-x  4 ninipa ninipa  4096  5월 24 13:45 kubernetes drwxr-xr-x  2 ninipa ninipa  4096  5월 24 13:45 licenses drwxr-xr-x  9 ninipa ninipa  4096  5월 24 13:45 python drwxr-xr-x  2 ninipa ninipa  4096  5월 24 13:45 sbin drwxr-xr-x  2 ninipa ninipa  4096  5월 24 13:45 yarn ninipa@ninipa-VirtualBox:~$ sudo chmod -R 777 /opt/spark ninipa@ninipa-VirtualBox:~$ ls -al /opt/spark/ 합계 164 drwxrwxrwx 13 root   root    4096  7월 27 20:10 . drwxr-xr-x  6 root   root    4096  7월 27 20:09 .. -rwxrwxrwx  1 ninipa ninipa 23235  5월 24 13:45 LICENSE -rwxrwxrwx  1 ninipa ninipa 57677  5월 24 13:45 NOTICE drwxrwxrwx  3 ninipa ninipa  4096  5월 24 13:45 R -rwxrwxrwx  1 ninipa ninipa  4488  5월 24 13:45 README.md -rwxrwxrwx  1 ninipa ninipa   183  5월 24 13:45 RELEASE drwxrwxrwx  2 ninipa ninipa  4096  5월 24 13:45 bin drwxrwxrwx  2 ninipa ninipa  4096  5월 24 13:45 conf drwxrwxrwx  5 ninipa ninipa  4096  5월 24 13:45 data drwxrwxrwx  4 ninipa ninipa  4096  5월 24 13:45 examples drwxrwxrwx  2 ninipa ninipa 20480  5월 24 13:45 jars drwxrwxrwx  4 ninipa ninipa  4096  5월 24 13:45 kubernetes drwxrwxrwx  2 ninipa ninipa  4096  5월 24 13:45 licenses drwxrwxrwx  9 ninipa ninipa  4096  5월 24 13:45 python drwxrwxrwx  2 ninipa ninipa  4096  5월 24 13:45 sbin drwxrwxrwx  2 ninipa ninipa  4096  5월 24 13:45 yarn ninipa@ninipa-VirtualBox:~$

4. 시스템 경로에 Spark 폴더 추가하기

ninipa@ninipa-VirtualBox:~$ nano ~/.bashrc 아래 3 줄 추가 export SPARK_HOME=/opt/spark export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin export PYSPARK_PYTHON=/usr/bin/python3

 추가 후 쉘 스크립트를 실행하여 변경사항을 적용한다.

ninipa@ninipa-VirtualBox:~$ source ~/.bashrc

5. 우분투에서 Apache Spark master 시작하기

ninipa@ninipa-VirtualBox:~$ start-master.sh starting org.apache.spark.deploy.master.Master, logging to /opt/spark/logs/spark-ninipa-org.apache.spark.deploy.master.Master-1-ninipa-VirtualBox.out ninipa@ninipa-VirtualBox:~$

기본으로 master는 7077번 포트를 worker는 8080 포트를 사용한다.

6. 웹 인터페이스에서 Spark master 접속하기

브라우저 실행하고 http://127.0.0.1:8080에 접속하기

이후 /opt/spark/log 경로에 생긴 master 로그 파일과 그 내용은 아래와 같다.

ninipa@ninipa-VirtualBox:/opt/spark/logs$ ls -al 합계 12 drwxrwxr-x  2 ninipa ninipa 4096  7월 28 10:00 . drwxrwxrwx 14 root   root   4096  7월 28 10:00 .. -rw-rw-r--  1 ninipa ninipa 2555  7월 28 10:00 spark-ninipa-org.apache.spark.deploy.master.Master-1-ninipa-VirtualBox.out ninipa@ninipa-VirtualBox:/opt/spark/logs$ cat spark-ninipa-org.apache.spark.deploy.master.Master-1-ninipa-VirtualBox.out  Spark Command: /usr/lib/jvm/java-11-openjdk-amd64/bin/java -cp /opt/spark/conf/:/opt/spark/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host ninipa-VirtualBox --port 7077 --webui-port 8080 ======================================== Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 21/07/28 10:00:54 INFO Master: Started daemon with process name: 38607@ninipa-VirtualBox 21/07/28 10:00:54 INFO SignalUtils: Registering signal handler for TERM 21/07/28 10:00:54 INFO SignalUtils: Registering signal handler for HUP 21/07/28 10:00:54 INFO SignalUtils: Registering signal handler for INT 21/07/28 10:00:54 WARN Utils: Your hostname, ninipa-VirtualBox resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface enp0s3) 21/07/28 10:00:54 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.1.2.jar) to constructor java.nio.DirectByteBuffer(long,int) WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 21/07/28 10:00:55 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 21/07/28 10:00:55 INFO SecurityManager: Changing view acls to: ninipa 21/07/28 10:00:55 INFO SecurityManager: Changing modify acls to: ninipa 21/07/28 10:00:55 INFO SecurityManager: Changing view acls groups to:  21/07/28 10:00:55 INFO SecurityManager: Changing modify acls groups to:  21/07/28 10:00:55 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(ninipa); groups with view permissions: Set(); users  with modify permissions: Set(ninipa); groups with modify permissions: Set() 21/07/28 10:00:55 INFO Utils: Successfully started service 'sparkMaster' on port 7077. 21/07/28 10:00:55 INFO Master: Starting Spark master at spark://ninipa-VirtualBox:7077 21/07/28 10:00:55 INFO Master: Running Spark version 3.1.2 21/07/28 10:00:56 INFO Utils: Successfully started service 'MasterUI' on port 8080. 21/07/28 10:00:56 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://10.0.2.15:8080 21/07/28 10:00:56 INFO Master: I have been elected leader! New state: ALIVE ninipa@ninipa-VirtualBox:/opt/spark/logs$

7. worker 스크립트 실행

문법: start-worker.sh spark://hostname:port 명령: start-worker.sh spark://ninipa-VirtualBox:7077

ninipa@ninipa-VirtualBox:~$ start-worker.sh spark://ninipa-VirtualBox:7077 starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark/logs/spark-ninipa-org.apache.spark.deploy.worker.Worker-1-ninipa-VirtualBox.out ninipa@ninipa-VirtualBox:~$

이후 /opt/spark/log 경로에 생긴 worker로그 파일과 그 내용은 아래와 같다.

ninipa@ninipa-VirtualBox:/opt/spark/logs$ ls -al 합계 16 drwxrwxr-x  2 ninipa ninipa 4096  7월 28 10:17 . drwxrwxrwx 15 root   root   4096  7월 28 10:18 .. -rw-rw-r--  1 ninipa ninipa 2647  7월 28 10:18 spark-ninipa-org.apache.spark.deploy.master.Master-1-ninipa-VirtualBox.out -rw-rw-r--  1 ninipa ninipa 3253  7월 28 10:18 spark-ninipa-org.apache.spark.deploy.worker.Worker-1-ninipa-VirtualBox.out ninipa@ninipa-VirtualBox:/opt/spark/logs$ cat spark-ninipa-org.apache.spark.deploy.worker.Worker-1-ninipa-VirtualBox.out  Spark Command: /usr/lib/jvm/java-11-openjdk-amd64/bin/java -cp /opt/spark/conf/:/opt/spark/jars/* -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://ninipa-VirtualBox:7077 ======================================== Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 21/07/28 10:18:00 INFO Worker: Started daemon with process name: 38979@ninipa-VirtualBox 21/07/28 10:18:00 INFO SignalUtils: Registering signal handler for TERM 21/07/28 10:18:00 INFO SignalUtils: Registering signal handler for HUP 21/07/28 10:18:00 INFO SignalUtils: Registering signal handler for INT 21/07/28 10:18:00 WARN Utils: Your hostname, ninipa-VirtualBox resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface enp0s3) 21/07/28 10:18:00 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.1.2.jar) to constructor java.nio.DirectByteBuffer(long,int) WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 21/07/28 10:18:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 21/07/28 10:18:01 INFO SecurityManager: Changing view acls to: ninipa 21/07/28 10:18:01 INFO SecurityManager: Changing modify acls to: ninipa 21/07/28 10:18:01 INFO SecurityManager: Changing view acls groups to:  21/07/28 10:18:01 INFO SecurityManager: Changing modify acls groups to:  21/07/28 10:18:01 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(ninipa); groups with view permissions: Set(); users  with modify permissions: Set(ninipa); groups with modify permissions: Set() 21/07/28 10:18:01 INFO Utils: Successfully started service 'sparkWorker' on port 33489. 21/07/28 10:18:01 INFO Worker: Worker decommissioning not enabled, SIGPWR will result in exiting. 21/07/28 10:18:01 INFO Worker: Starting Spark worker 10.0.2.15:33489 with 1 cores, 6.8 GiB RAM 21/07/28 10:18:01 INFO Worker: Running Spark version 3.1.2 21/07/28 10:18:01 INFO Worker: Spark home: /opt/spark 21/07/28 10:18:01 INFO ResourceUtils: ============================================================== 21/07/28 10:18:01 INFO ResourceUtils: No custom resources configured for spark.worker. 21/07/28 10:18:01 INFO ResourceUtils: ============================================================== 21/07/28 10:18:02 INFO Utils: Successfully started service 'WorkerUI' on port 8081. 21/07/28 10:18:02 INFO WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://10.0.2.15:8081 21/07/28 10:18:02 INFO Worker: Connecting to master ninipa-VirtualBox:7077... 21/07/28 10:18:02 INFO TransportClientFactory: Successfully created connection to ninipa-VirtualBox/127.0.1.1:7077 after 48 ms (0 ms spent in bootstraps) 21/07/28 10:18:02 INFO Worker: Successfully registered with master spark://ninipa-VirtualBox:7077 ninipa@ninipa-VirtualBox:/opt/spark/logs$

8. Spark shell 사용

다음 명령을 입력하여 Spark 셸을 사용한 프로그래밍을 할 수 있다.

ninipa@ninipa-VirtualBox:~$ spark-shell 21/07/28 10:26:03 WARN Utils: Your hostname, ninipa-VirtualBox resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface enp0s3) 21/07/28 10:26:03 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.1.2.jar) to constructor java.nio.DirectByteBuffer(long,int) WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 21/07/28 10:26:03 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Spark context Web UI available at http://10.0.2.15:4040 Spark context available as 'sc' (master = local[*], app id = local-1627435573621). Spark session available as 'spark'. Welcome to       ____              __      / __/__  ___ _____/ /__     _\ \/ _ \/ _ `/ __/  '_/    /___/ .__/\_,_/_/ /_/\_\   version 3.1.2       /_/           Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 11.0.11) Type in expressions to have them evaluated. Type :help for more information. scala>

Spark 실행 후 브라우저에서 master와 worker의  Web UI 실행한 내용은 아래와 같다.

Worker ID와 그 상태 및 사용 리소스에 대한 정보가 표시된다.