Spark on k8s


Concerning k8s use cases …

I will be heading an initiative to migrate our (PeopleDataLabs) processing substrate from traditional spark+yarn+hadoop architecture to a native spark+k8s configuration.

Info on how to do this is sparse ATM as the k8s driver was just recently released in spark 2.3, but I have managed to get a native spark+k8s strawman working on CDK! Now to make it accommodate our workload :slight_smile:

Find a reasonably current (at time of writing), high level update on this bleeding edge -


Interesting stuff. Part of the reason i remembered the Juju K8S charm stuff was because last night I helm installed spark and you get 1.5.x…

I need more staff so I can build out the supported charms on the k8s side as well for big data! :wink:


The number of k8s charms right now is small. If you want some examples to look at, besides the ones I’ve talked about in the posts so far, there’s these:

The ecosystem right now is very embryonic with folks kicking the tyres. So any feedback welcome!