Spark on k8s


#1

Concerning k8s use cases …

I will be heading an initiative to migrate our (PeopleDataLabs) processing substrate from traditional spark+yarn+hadoop architecture to a native spark+k8s configuration.

Info on how to do this is sparse ATM as the k8s driver was just recently released in spark 2.3, but I have managed to get a native spark+k8s strawman working on CDK! Now to make it accommodate our workload :slight_smile:

Find a reasonably current (at time of writing), high level update on this bleeding edge - https://databricks.com/blog/2018/09/26/whats-new-for-apache-spark-on-kubernetes-in-the-upcoming-apache-spark-2-4-release.html


#2

Interesting stuff. Part of the reason i remembered the Juju K8S charm stuff was because last night I helm installed spark and you get 1.5.x…

I need more staff so I can build out the supported charms on the k8s side as well for big data! :wink:


#3

The number of k8s charms right now is small. If you want some examples to look at, besides the ones I’ve talked about in the posts so far, there’s these:

The ecosystem right now is very embryonic with folks kicking the tyres. So any feedback welcome!