1、spark-shell --master local[2]启动出现如下报错:

[hadoop@hadoop002 spark-2.4.2-bin-2.6.0-cdh5.7.0]$ spark-shell --master local[2]
19/07/23 00:52:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
19/07/23 00:52:32 ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot create file/g6_directory/local-1563814350999.inprogress. Name node is in safe mode.
The reported blocks 25 needs additional 10 blocks to reach the threshold 0.9990 of total blocks 35.
The number of live datanodes 1 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.

19/07/23 00:52:33 ERROR repl.Main: Failed to initialize Spark session.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot create file/g6_directory/local-1563814350999.inprogress. Name node is in safe mode.
The reported blocks 25 needs additional 10 blocks to reach the threshold 0.9990 of total blocks 35.
The number of live datanodes 1 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.

定位ERROR:Name node is in safe,那么NameNode为什么在安全模式下?

NameNode发现集群中DataNode丢失达到一定比例(0.01%)时会进入安全模式,此时只允许查看数据不允许对数据进行任何操作。
使用命令:hdfs dfsadmin -safemode leave 离开安全模式

hdfs dfsadmin -safemode enter 进入安全模式
退出安全模式即可重新进入。

2、PK哥上课过程中重启spark-shell报错:
https://blog.csdn.net/zhikanjiani/article/details/97833470
问题:Failed to start database ‘metastore_db’ with class loader org.apache.spark.sql.hive.client

解决:rm -rf metastore_db/

3、更新:2019-08-16
暂时使用hdfs dfsadmin -safemode leave命令离开了安全模式。
在这里插入图片描述

每次启动spark-shell都会报错:2019-08-24
在这里插入图片描述

Logo

有“AI”的1024 = 2048,欢迎大家加入2048 AI社区

更多推荐