一次spark sql 优化的经历： SparkException: Job aborted / spark.yarn.executor.memoryOverhead

问题背景某天跑 sparkSQL 的时候，遇到报错：org.apache.spark.SparkException: Job aborted.at org.apache.spark.sql.execution.datasources.FileFormatWriter.write(FileFormatWriter.scala:198)atorg.apache.spark.sql.hive.exec

??-Samur

7524人浏览 · 2022-02-17 19:26:33

??-Samur · 2022-02-17 19:26:33 发布

问题背景

某天跑 sparkSQL 的时候，遇到报错：
org.apache.spark.SparkException: Job aborted.
at org.apache.spark.sql.execution.datasources.FileFormatWriter $. w r i t e (F i l e F o r m a t W r i t e r . s c a l a : 198) a t o r g . a p a c h e . s p a r k . s q l . h i v e . e x e c u t i o n . S a v e A s H i v e F i l e$ class.saveAsHiveFile(SaveAsHiveFile.scala:86)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.saveAsHiveFile(InsertIntoHiveTable.scala:66)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.processInsert(InsertIntoHiveTable.scala:195)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.run(InsertIntoHiveTable.scala:99)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult $l z y c o m p u t e (c o m m a n d s . s c a l a : 104) a t o r g . a p a c h e . s p a r k . s q l . e x e c u t i o n . c o m m a n d . D a t a W r i t i n g C o m m a n d E x e c . s i d e E f f e c t R e s u l t (c o m m a n d s . s c a l a : 102) a t o r g . a p a c h e . s p a r k . s q l . e x e c u t i o n . c o m m a n d . D a t a W r i t i n g C o m m a n d E x e c . e x e c u t e C o l l e c t (c o m m a n d s . s c a l a : 115) a t o r g . a p a c h e . s p a r k . s q l . D a t a s e t$ $anonfun $6 . a p p l y (D a t a s e t . s c a l a : 194) a t o r g . a p a c h e . s p a r k . s q l . D a t a s e t$ $anonfun $6 . a p p l y (D a t a s e t . s c a l a : 194) a t o r g . a p a c h e . s p a r k . s q l . D a t a s e t$ $anonfun $52 . a p p l y (D a t a s e t . s c a l a : 3370) a t o r g . a p a c h e . s p a r k . s q l . e x e c u t i o n . S Q L E x e c u t i o n$ $a n o n f u n$ withNewExecutionId $1 . a p p l y (S Q L E x e c u t i o n . s c a l a : 80) a t o r g . a p a c h e . s p a r k . s q l . e x e c u t i o n . S Q L E x e c u t i o n$ .withSQLConfPropagated(SQLExecution.scala:127)
at org.apache.spark.sql.execution.SQLExecution $. w i t h N e w E x e c u t i o n I d (S Q L E x e c u t i o n . s c a l a : 75) a t o r g . a p a c h e . s p a r k . s q l . D a t a s e t . o r g$ apache $s p a r k$ sql $D a t a s e t$ $w i t h A c t i o n (D a t a s e t . s c a l a : 3369) a t o r g . a p a c h e . s p a r k . s q l . D a t a s e t . < i n i t > (D a t a s e t . s c a l a : 194) a t o r g . a p a c h e . s p a r k . s q l . D a t a s e t$ .ofRows(Dataset.scala:79)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:643)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)
at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:62)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:371)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver $. m a i n (S p a r k S Q L C L I D r i v e r . s c a l a : 274) a t o r g . a p a c h e . s p a r k . s q l . h i v e . t h r i f t s e r v e r . S p a r k S Q L C L I D r i v e r . m a i n (S p a r k S Q L C L I D r i v e r . s c a l a) a t s u n . r e f l e c t . N a t i v e M e t h o d A c c e s s o r I m p l . i n v o k e 0 (N a t i v e M e t h o d) a t s u n . r e f l e c t . N a t i v e M e t h o d A c c e s s o r I m p l . i n v o k e (N a t i v e M e t h o d A c c e s s o r I m p l . j a v a : 62) a t s u n . r e f l e c t . D e l e g a t i n g M e t h o d A c c e s s o r I m p l . i n v o k e (D e l e g a t i n g M e t h o d A c c e s s o r I m p l . j a v a : 43) a t j a v a . l a n g . r e f l e c t . M e t h o d . i n v o k e (M e t h o d . j a v a : 498) a t o r g . a p a c h e . s p a r k . d e p l o y . J a v a M a i n A p p l i c a t i o n . s t a r t (S p a r k A p p l i c a t i o n . s c a l a : 52) a t o r g . a p a c h e . s p a r k . d e p l o y . S p a r k S u b m i t . o r g$ apache $s p a r k$ deploy $S p a r k S u b m i t$ $runMain(SparkSubmit.scala:845)
at org.apache.spark.deploy.SparkSubmit.doRunMain $1 (S p a r k S u b m i t . s c a l a : 161) a t o r g . a p a c h e . s p a r k . d e p l o y . S p a r k S u b m i t . s u b m i t (S p a r k S u b m i t . s c a l a : 184) a t o r g . a p a c h e . s p a r k . d e p l o y . S p a r k S u b m i t . d o S u b m i t (S p a r k S u b m i t . s c a l a : 86) a t o r g . a p a c h e . s p a r k . d e p l o y . S p a r k S u b m i t$ $anon $2 . d o S u b m i t (S p a r k S u b m i t . s c a l a : 920) a t o r g . a p a c h e . s p a r k . d e p l o y . S p a r k S u b m i t$ .main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 6 in stage 16.0 failed 4 times, most recent failure: Lost task 6.3 in stage 16.0 (TID 478, idc-sql-dms-13, executor 40): ExecutorLostFailure (executor 40 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 11.8 GB of 11 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714.
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org $a p a c h e$ spark $s c h e d u l e r$ DAGScheduler $f a i l J o b A n d I n d e p e n d e n t S t a g e s (D A G S c h e d u l e r . s c a l a : 1925) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . D A G S c h e d u l e r$ anonfun$abortStage $1 . a p p l y (D A G S c h e d u l e r . s c a l a : 1913) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . D A G S c h e d u l e r$ $a n o n f u n$ abortStage $1 . a p p l y (D A G S c h e d u l e r . s c a l a : 1912) a t s c a l a . c o l l e c t i o n . m u t a b l e . R e s i z a b l e A r r a y$ class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1912)
at org.apache.spark.scheduler.DAGScheduler $KaTeX parse error: Can't use function '$' in math mode at position 8: anonfun$̲handleTaskSetFa\dots$ anonfun$handleTaskSetFailed $1 . a p p l y (D A G S c h e d u l e r . s c a l a : 948) a t s c a l a . O p t i o n . f o r e a c h (O p t i o n . s c a l a : 257) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . D A G S c h e d u l e r . h a n d l e T a s k S e t F a i l e d (D A G S c h e d u l e r . s c a l a : 948) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . D A G S c h e d u l e r E v e n t P r o c e s s L o o p . d o O n R e c e i v e (D A G S c h e d u l e r . s c a l a : 2146) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . D A G S c h e d u l e r E v e n t P r o c e s s L o o p . o n R e c e i v e (D A G S c h e d u l e r . s c a l a : 2095) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . D A G S c h e d u l e r E v e n t P r o c e s s L o o p . o n R e c e i v e (D A G S c h e d u l e r . s c a l a : 2084) a t o r g . a p a c h e . s p a r k . u t i l . E v e n t L o o p$ $anon $1 . r u n (E v e n t L o o p . s c a l a : 49) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . D A G S c h e d u l e r . r u n J o b (D A G S c h e d u l e r . s c a l a : 759) a t o r g . a p a c h e . s p a r k . S p a r k C o n t e x t . r u n J o b (S p a r k C o n t e x t . s c a l a : 2061) a t o r g . a p a c h e . s p a r k . s q l . e x e c u t i o n . d a t a s o u r c e s . F i l e F o r m a t W r i t e r$ .write(FileFormatWriter.scala:167)
… 35 more
org.apache.spark.SparkException: Job aborted.
at org.apache.spark.sql.execution.datasources.FileFormatWriter $. w r i t e (F i l e F o r m a t W r i t e r . s c a l a : 198) a t o r g . a p a c h e . s p a r k . s q l . h i v e . e x e c u t i o n . S a v e A s H i v e F i l e$ class.saveAsHiveFile(SaveAsHiveFile.scala:86)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.saveAsHiveFile(InsertIntoHiveTable.scala:66)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.processInsert(InsertIntoHiveTable.scala:195)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.run(InsertIntoHiveTable.scala:99)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult $l z y c o m p u t e (c o m m a n d s . s c a l a : 104) a t o r g . a p a c h e . s p a r k . s q l . e x e c u t i o n . c o m m a n d . D a t a W r i t i n g C o m m a n d E x e c . s i d e E f f e c t R e s u l t (c o m m a n d s . s c a l a : 102) a t o r g . a p a c h e . s p a r k . s q l . e x e c u t i o n . c o m m a n d . D a t a W r i t i n g C o m m a n d E x e c . e x e c u t e C o l l e c t (c o m m a n d s . s c a l a : 115) a t o r g . a p a c h e . s p a r k . s q l . D a t a s e t$ $anonfun $6 . a p p l y (D a t a s e t . s c a l a : 194) a t o r g . a p a c h e . s p a r k . s q l . D a t a s e t$ $anonfun $6 . a p p l y (D a t a s e t . s c a l a : 194) a t o r g . a p a c h e . s p a r k . s q l . D a t a s e t$ $anonfun $52 . a p p l y (D a t a s e t . s c a l a : 3370) a t o r g . a p a c h e . s p a r k . s q l . e x e c u t i o n . S Q L E x e c u t i o n$ $a n o n f u n$ withNewExecutionId $1 . a p p l y (S Q L E x e c u t i o n . s c a l a : 80) a t o r g . a p a c h e . s p a r k . s q l . e x e c u t i o n . S Q L E x e c u t i o n$ .withSQLConfPropagated(SQLExecution.scala:127)
at org.apache.spark.sql.execution.SQLExecution $. w i t h N e w E x e c u t i o n I d (S Q L E x e c u t i o n . s c a l a : 75) a t o r g . a p a c h e . s p a r k . s q l . D a t a s e t . o r g$ apache $s p a r k$ sql $D a t a s e t$ $w i t h A c t i o n (D a t a s e t . s c a l a : 3369) a t o r g . a p a c h e . s p a r k . s q l . D a t a s e t . < i n i t > (D a t a s e t . s c a l a : 194) a t o r g . a p a c h e . s p a r k . s q l . D a t a s e t$ .ofRows(Dataset.scala:79)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:643)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)
at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:62)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:371)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver $. m a i n (S p a r k S Q L C L I D r i v e r . s c a l a : 274) a t o r g . a p a c h e . s p a r k . s q l . h i v e . t h r i f t s e r v e r . S p a r k S Q L C L I D r i v e r . m a i n (S p a r k S Q L C L I D r i v e r . s c a l a) a t s u n . r e f l e c t . N a t i v e M e t h o d A c c e s s o r I m p l . i n v o k e 0 (N a t i v e M e t h o d) a t s u n . r e f l e c t . N a t i v e M e t h o d A c c e s s o r I m p l . i n v o k e (N a t i v e M e t h o d A c c e s s o r I m p l . j a v a : 62) a t s u n . r e f l e c t . D e l e g a t i n g M e t h o d A c c e s s o r I m p l . i n v o k e (D e l e g a t i n g M e t h o d A c c e s s o r I m p l . j a v a : 43) a t j a v a . l a n g . r e f l e c t . M e t h o d . i n v o k e (M e t h o d . j a v a : 498) a t o r g . a p a c h e . s p a r k . d e p l o y . J a v a M a i n A p p l i c a t i o n . s t a r t (S p a r k A p p l i c a t i o n . s c a l a : 52) a t o r g . a p a c h e . s p a r k . d e p l o y . S p a r k S u b m i t . o r g$ apache $s p a r k$ deploy $S p a r k S u b m i t$ $runMain(SparkSubmit.scala:845)
at org.apache.spark.deploy.SparkSubmit.doRunMain $1 (S p a r k S u b m i t . s c a l a : 161) a t o r g . a p a c h e . s p a r k . d e p l o y . S p a r k S u b m i t . s u b m i t (S p a r k S u b m i t . s c a l a : 184) a t o r g . a p a c h e . s p a r k . d e p l o y . S p a r k S u b m i t . d o S u b m i t (S p a r k S u b m i t . s c a l a : 86) a t o r g . a p a c h e . s p a r k . d e p l o y . S p a r k S u b m i t$ $anon $2 . d o S u b m i t (S p a r k S u b m i t . s c a l a : 920) a t o r g . a p a c h e . s p a r k . d e p l o y . S p a r k S u b m i t$ .main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 6 in stage 16.0 failed 4 times, most recent failure: Lost task 6.3 in stage 16.0 (TID 478, idc-sql-dms-13, executor 40): ExecutorLostFailure (executor 40 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 11.8 GB of 11 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714.
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org $a p a c h e$ spark $s c h e d u l e r$ DAGScheduler $f a i l J o b A n d I n d e p e n d e n t S t a g e s (D A G S c h e d u l e r . s c a l a : 1925) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . D A G S c h e d u l e r$ anonfun$abortStage $1 . a p p l y (D A G S c h e d u l e r . s c a l a : 1913) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . D A G S c h e d u l e r$ $a n o n f u n$ abortStage $1 . a p p l y (D A G S c h e d u l e r . s c a l a : 1912) a t s c a l a . c o l l e c t i o n . m u t a b l e . R e s i z a b l e A r r a y$ class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1912)
at org.apache.spark.scheduler.DAGScheduler $KaTeX parse error: Can't use function '$' in math mode at position 8: anonfun$̲handleTaskSetFa\dots$ anonfun$handleTaskSetFailed $1 . a p p l y (D A G S c h e d u l e r . s c a l a : 948) a t s c a l a . O p t i o n . f o r e a c h (O p t i o n . s c a l a : 257) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . D A G S c h e d u l e r . h a n d l e T a s k S e t F a i l e d (D A G S c h e d u l e r . s c a l a : 948) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . D A G S c h e d u l e r E v e n t P r o c e s s L o o p . d o O n R e c e i v e (D A G S c h e d u l e r . s c a l a : 2146) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . D A G S c h e d u l e r E v e n t P r o c e s s L o o p . o n R e c e i v e (D A G S c h e d u l e r . s c a l a : 2095) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . D A G S c h e d u l e r E v e n t P r o c e s s L o o p . o n R e c e i v e (D A G S c h e d u l e r . s c a l a : 2084) a t o r g . a p a c h e . s p a r k . u t i l . E v e n t L o o p$ $anon $1 . r u n (E v e n t L o o p . s c a l a : 49) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . D A G S c h e d u l e r . r u n J o b (D A G S c h e d u l e r . s c a l a : 759) a t o r g . a p a c h e . s p a r k . S p a r k C o n t e x t . r u n J o b (S p a r k C o n t e x t . s c a l a : 2061) a t o r g . a p a c h e . s p a r k . s q l . e x e c u t i o n . d a t a s o u r c e s . F i l e F o r m a t W r i t e r$ .write(FileFormatWriter.scala:167)
… 35 more

通过上面的日志，大概了解到任务失败的原因应该是内存超过限定。 “Container killed by YARN for exceeding memory limits. ”，解决问题的第一思路是 sql 能不能优化下，加内存属于下下策。

解决办法

先讲一下原来的 sql 思路：

SELECT a.name, a.age, b.alias
from a
left join (
    SELECT id, concat_ws(',', COLLECT_LIST(alias)) alias
	from bb
	group by id
) b

这是一个很简单的逻辑，猜测问题应该出现在 collect_ws() 函数，当 b 表根据 id 聚合的时候，如果大量的数据加载到 list （COLLECT_LIST）里面，将导致内存耗尽。

解决思路应该是先把重复数据去掉，再调用 concat_ws(’,’ , COLLECT_LIST(alias))，优化后的sql 如下：

SELECT a.name, a.age, b.alias
from a
left join (
    SELECT id, collect_ws(',', COLLECT_LIST(alias)) alias
	from (
	    SELECT id, alias
		from bb 
		group by id, alias
	)
	group by id
) b

还有更简单的一种写法，就是使用 COLLECT_SET 代替 COLLECT_LIST：

SELECT a.name, a.age, b.alias
from a
left join (
    SELECT id, collect_ws(',', COLLECT_SET(alias)) alias
	from bb
	group by id
) b

哈哈，问题解决！

2048 AI社区

有“AI”的1024 = 2048，欢迎大家加入2048 AI社区

更多推荐

保障AI算力基石：使用NVIDIA DCGM高效管理GPU数据中心

NVIDIA DCGM（数据中心GPU管理器）是NVIDIA推出的企业级GPU集群管理工具，提供远超nvidia-smi的规模化监控能力。它能统一管理数百台服务器中的所有GPU，监测200+深度指标（包括NVLink带宽、ECC错误等），并支持主动诊断和故障隔离。关键特性包括集群级监控视图、策略驱动自动化管理、与Prometheus/Kubernetes生态集成，适用于AI训练集群、云服务等场景。

2048 AI社区

多模态大模型图像识别

本指南涵盖了多模态大模型在计算机视觉各个任务中的应用，从基础理论到实际部署的完整流程。选择合适的架构: 根据任务需求和资源限制选择模型数据质量至关重要: 高质量的标注数据是成功的基础持续迭代: 通过实验和监控不断优化模型性能工程化思维: 考虑部署、维护和扩展性保持更新: 关注最新研究进展和工业实践。