New Hadoop Charms Available


#1

Hi folks,

Just a note to say that the latest Bigtop charms are now available on the Edge channel.

This include updated:

Hadoop Namenode
Hadoop ResourceManager
Hadoop Slave (that will get renamed after this release)
and Hadoop Client.

I have an updated Spark charm I’m testing locally and will update on the edge channel shortly which will update Spark to 2.2.3

This bumps our Bigtop release from 1.2.x to 1.4.0 and the release notes can be found here:
https://blogs.apache.org/bigtop/entry/release-of-apache-bigtop-13

Going forward I will update the charms for Juju Storage and Network support and will look to bring in the K8S Hadoop charms in line with the Bigtop community activity, I dont’ see much value in rolling our own I’d rather have a few other developers helping out so I shall lend a hand upstream and bring them down to Juju when ready.

Anyway… I’ll leave them in edge and update the bundles and test a few things over the next week or so and slowly bump them through the channels. Unless someone says otherwise I’m gonna run a lazy consensus on this one and if no one squeals I’ll push them into stable in a week or so.

Tom


#2

Sorry for the delay, Spark on Edge channel is 2.2.3!


#3

Hive is now at 2.3.3
Hbase is now at 1.3.2

both on the edge channel. I’ve not tested these yet so YMMV.


#4

Still running the lazy consensus? I paid down some technical debt related to bigtop:

BIGTOP-3015 – remove client from hadoop-spark
BIGTOP-3092 – kafka now has juju storage
BIGTOP-3268 – nagios zookeeper updates

And the grand finale:

BIGTOP-3271 (update all versions/revs; @magicaltrout, i could use a +1 on this)

The bigtop-repo resource has been updated to branch-1.4 as bigtop-repo-8 and relevant artifacts are in edge:

url: cs:~bigdata-charmers/hadoop-namenode-54
url: cs:~bigdata-charmers/hadoop-plugin-52
url: cs:~bigdata-charmers/hadoop-resourcemanager-56
url: cs:~bigdata-charmers/hadoop-slave-55
url: cs:~bigdata-charmers/hbase-45
url: cs:~bigdata-charmers/hive-35
url: cs:~bigdata-charmers/kafka-55
url: cs:~bigdata-charmers/mahout-43
url: cs:~bigdata-charmers/spark-81
url: cs:~bigdata-charmers/zeppelin-52
url: cs:~bigdata-charmers/zookeeper-58

url: cs:~bigdata-charmers/bundle/hadoop-hbase-48
url: cs:~bigdata-charmers/bundle/hadoop-kafka-63
url: cs:~bigdata-charmers/bundle/hadoop-processing-92
url: cs:~bigdata-charmers/bundle/hadoop-spark-66
url: cs:~bigdata-charmers/bundle/spark-processing-84

Smoke tests across all bundles looks good, eg:

$ juju run-action --wait hbase/0 smoke-test
unit-hbase-0:
  id: 180a247f-0c3a-418d-86bc-6150ebd963eb
  results:
    outcome: success
  status: completed
  timing:
    completed: 2019-11-02 14:35:56 +0000 UTC
    enqueued: 2019-11-02 14:35:30 +0000 UTC
    started: 2019-11-02 14:35:34 +0000 UTC
  unit: hbase/0

$ juju run-action --wait namenode/0 smoke-test
unit-namenode-0:
  id: ac6b89a7-e119-4cbe-8e30-f6acadc1be81
  results:
    outcome: success
  status: completed
  timing:
    completed: 2019-11-02 14:36:18 +0000 UTC
    enqueued: 2019-11-02 14:36:03 +0000 UTC
    started: 2019-11-02 14:36:03 +0000 UTC
  unit: namenode/0

$ juju run-action --wait resourcemanager/0 smoke-test
unit-resourcemanager-0:
  id: 2cd09d1c-2fd5-4498-831e-980f511c3c29
  results:
    outcome: success
  status: completed
  timing:
    completed: 2019-11-02 14:37:22 +0000 UTC
    enqueued: 2019-11-02 14:36:28 +0000 UTC
    started: 2019-11-02 14:36:32 +0000 UTC
  unit: resourcemanager/0

$ juju run-action --wait zookeeper/0 smoke-test
unit-zookeeper-0:
  id: 8d7070a3-e00a-447f-896b-b5d7d1cc72e2
  results:
    outcome: success
  status: completed
  timing:
    completed: 2019-11-02 14:38:06 +0000 UTC
    enqueued: 2019-11-02 14:37:32 +0000 UTC
    started: 2019-11-02 14:37:36 +0000 UTC
  unit: zookeeper/0

I also added my release scripts to the bigtop base layer in case they’re helpful. Lmk if you want me to release through additional channels.


#5

You gazumped my release! :slight_smile:
Oh well, good to get the PRs and stuff in. I’ll test them out later this week and let you know.