- package spark app (maven)
<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>2.3</version> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> </execution> </executions> </plugin>
dependency conflict: Shading allows you to make a second copy of the conflicting package under a different namespace and rewrites your application’s code to use the renamed version.
- run driver in client or one of worker node
- scheduling spark apps
queue with priority and capacity defined. submit job to queue.
Fair Scheduler + queue management;