Spark调研笔记

Spark调研笔记

February 21, 2019

理论

streaming 101
streaming 102

spark

选项

spark.io.compression.codec snappy (lz4依赖冲突)
spark.streaming.concurrentjobs (spark streaming实时大数据分析4.4.4)
spark.streaming.receiver.writeaheadlog.enable (spark streaming实时大数据分析5.6节)
spark.sql.shuffle.partitions (default 200, 在单节点测试时,会造成极大的延迟).

pyspark

本地环境搭建

export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)
export SPARK_HOME="$HOME/opt/spark-2.3.2-bin-hadoop2.6"
export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.7-src.zip:$PYTHONPATH
export PATH=$SPARK_HOME/bin:$PATH

python环境

virtualenv -p python3 spark-python3
source spark-python3/bin/activate
pip install pyspark
deactivate(仅从当前venv环境脱离时执行)

spark