scala - sbt-assembly: Merge Errors - Deduplicate -

i getting these errors using sbt assembly.

i using spark seems @ root of problem.

val spark =  seq(   "org.apache.spark" %% "spark-core" % sparkversion,   "org.apache.spark" %% "spark-sql" % sparkversion,   "org.apache.spark" %% "spark-streaming" % sparkversion ) 

error:

[error] 12 errors encountered during merge [trace] stack trace suppressed: run last corebackend/*:assembly full output. [trace] stack trace suppressed: run last core/*:assembly full output. [trace] stack trace suppressed: run last commons/*:assembly full output. [error] (corebackend/*:assembly) deduplicate: different file contents found in following: [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.osgi/org.osgi.core/org.osgi.core-4.3.1.jar:osgi-opt/bnd.bnd [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.osgi/org.osgi.compendium/org.osgi.compendium-4.3.1.jar:osgi-opt/bnd.bnd [error] deduplicate: different file contents found in following: [error] /volumes/coyote/developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/absent.class [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/absent.class [error] deduplicate: different file contents found in following: [error] /volumes/coyote/developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/function.class [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/function.class [error] deduplicate: different file contents found in following: [error] /volumes/coyote/developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/optional$1$1.class [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/optional$1$1.class [error] deduplicate: different file contents found in following: [error] /volumes/coyote/developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/optional$1.class [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/optional$1.class [error] deduplicate: different file contents found in following: [error] /volumes/coyote/developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/optional.class [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/optional.class [error] deduplicate: different file contents found in following: [error] /volumes/coyote/developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/present.class [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/present.class [error] deduplicate: different file contents found in following: [error] /volumes/coyote/developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/supplier.class [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/supplier.class [error] deduplicate: different file contents found in following: [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-common/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/factories/package-info.class [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-api/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/factories/package-info.class [error] deduplicate: different file contents found in following: [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-common/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/factory/providers/package-info.class [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-api/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/factory/providers/package-info.class [error] deduplicate: different file contents found in following: [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-common/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/util/package-info.class [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-api/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/util/package-info.class [error] deduplicate: different file contents found in following: [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.apache.spark/spark-core_2.11/spark-core_2.11-1.5.1.jar:org/apache/spark/unused/unusedstubclass.class [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.apache.spark/spark-launcher_2.11/spark-launcher_2.11-1.5.1.jar:org/apache/spark/unused/unusedstubclass.class [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.spark-project.spark/unused/unused-1.0.0.jar:org/apache/spark/unused/unusedstubclass.class [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:org/apache/spark/unused/unusedstubclass.class [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.apache.spark/spark-network-shuffle_2.11/spark-network-shuffle_2.11-1.5.1.jar:org/apache/spark/unused/unusedstubclass.class [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.apache.spark/spark-unsafe_2.11/spark-unsafe_2.11-1.5.1.jar:org/apache/spark/unused/unusedstubclass.class [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.apache.spark/spark-sql_2.11/spark-sql_2.11-1.5.1.jar:org/apache/spark/unused/unusedstubclass.class [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.apache.spark/spark-catalyst_2.11/spark-catalyst_2.11-1.5.1.jar:org/apache/spark/unused/unusedstubclass.class [error] /volumes/coyote/developer/tibra/lib_managed/jars/org.apache.spark/spark-streaming_2.11/spark-streaming_2.11-1.5.1.jar:org/apache/spark/unused/unusedstubclass.class [error] (core/*:assembly) deduplicate: different file contents found in following: [error] /users/bryan/.ivy2/cache/org.osgi/org.osgi.core/jars/org.osgi.core-4.3.1.jar:osgi-opt/bnd.bnd [error] /users/bryan/.ivy2/cache/org.osgi/org.osgi.compendium/jars/org.osgi.compendium-4.3.1.jar:osgi-opt/bnd.bnd [error] deduplicate: different file contents found in following: [error] /users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/absent.class [error] /users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/absent.class [error] deduplicate: different file contents found in following: [error] /users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/function.class [error] /users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/function.class [error] deduplicate: different file contents found in following: [error] /users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/optional$1$1.class [error] /users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/optional$1$1.class [error] deduplicate: different file contents found in following: [error] /users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/optional$1.class [error] /users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/optional$1.class [error] deduplicate: different file contents found in following: [error] /users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/optional.class [error] /users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/optional.class [error] deduplicate: different file contents found in following: [error] /users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/present.class [error] /users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/present.class [error] deduplicate: different file contents found in following: [error] /users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/supplier.class [error] /users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/supplier.class [error] deduplicate: different file contents found in following: [error] /users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-common/jars/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/factories/package-info.class [error] /users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-api/jars/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/factories/package-info.class [error] deduplicate: different file contents found in following: [error] /users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-common/jars/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/factory/providers/package-info.class [error] /users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-api/jars/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/factory/providers/package-info.class [error] deduplicate: different file contents found in following: [error] /users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-common/jars/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/util/package-info.class [error] /users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-api/jars/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/util/package-info.class [error] deduplicate: different file contents found in following: [error] /users/bryan/.ivy2/cache/org.apache.spark/spark-core_2.11/jars/spark-core_2.11-1.5.1.jar:org/apache/spark/unused/unusedstubclass.class [error] /users/bryan/.ivy2/cache/org.apache.spark/spark-launcher_2.11/jars/spark-launcher_2.11-1.5.1.jar:org/apache/spark/unused/unusedstubclass.class [error] /users/bryan/.ivy2/cache/org.spark-project.spark/unused/jars/unused-1.0.0.jar:org/apache/spark/unused/unusedstubclass.class [error] /users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:org/apache/spark/unused/unusedstubclass.class [error] /users/bryan/.ivy2/cache/org.apache.spark/spark-network-shuffle_2.11/jars/spark-network-shuffle_2.11-1.5.1.jar:org/apache/spark/unused/unusedstubclass.class [error] /users/bryan/.ivy2/cache/org.apache.spark/spark-unsafe_2.11/jars/spark-unsafe_2.11-1.5.1.jar:org/apache/spark/unused/unusedstubclass.class [error] /users/bryan/.ivy2/cache/org.apache.spark/spark-sql_2.11/jars/spark-sql_2.11-1.5.1.jar:org/apache/spark/unused/unusedstubclass.class [error] /users/bryan/.ivy2/cache/org.apache.spark/spark-catalyst_2.11/jars/spark-catalyst_2.11-1.5.1.jar:org/apache/spark/unused/unusedstubclass.class [error] /users/bryan/.ivy2/cache/org.apache.spark/spark-streaming_2.11/jars/spark-streaming_2.11-1.5.1.jar:org/apache/spark/unused/unusedstubclass.class [error] (commons/*:assembly) deduplicate: different file contents found in following: [error] /users/bryan/.ivy2/cache/org.osgi/org.osgi.core/jars/org.osgi.core-4.3.1.jar:osgi-opt/bnd.bnd [error] /users/bryan/.ivy2/cache/org.osgi/org.osgi.compendium/jars/org.osgi.compendium-4.3.1.jar:osgi-opt/bnd.bnd 

i tried recommend solutions here no luck.

sbt-assembly: deduplication found error

deduplicating commons-validator - sbt assembly

spark + sbt-assembly: "deduplicate: different file contents found in following"

this not answer problem, workaround.

i hope saves few hundred man-hours.

use sbt-native-packager instead of sbt-assembly.

add plugins.sbt:

addsbtplugin("com.typesafe.sbt" % "sbt-native-packager" % "1.0.0") 

and in build.sbt

enableplugins(javaapppackaging) enableplugins(universalplugin) 

to build files multiple scala versions use +

+ universal:packagebin 

the output tell file created.

unfortunately generated jars zipped. not fat jar. (to generate fat jar require sbt-assembly has same issues)

to overcome this, made simple script (in sbt) unzips generated files , writes jar paths file can build spark submit script.

packagebin in txtformat := {      val zippedjar = "core-backend-1.0.zip"     val basepath = target.value / "universal"      // unzip folder of jars     io.unzip(basepath / zippedjar, basepath)      val filemappings = (mappings in universal).value     val sparkscriptout = basepath / s"${packagename.value}.txt"      // append mappings list     filemappings foreach {         case (file, name) => io.append(sparkscriptout, s"core-backend-1.0/$name${io.newline}")     }     sparkscriptout } 

after building zip, use execute task:

+ txtformat:packagebin