Charms for centos - lets begin

So, I’ve started to map out the first steps towards a more enjoyable experience using centos7 series with juju. I’ll focus on MAAS here, since thats my use-case. I think this would be very similar to that of other cloud-images, but I have yet to discover that I guess.

Its a strong argument for juju to be able to actually be able to use other os:es better, which is a requirement for me at least. Not all problems are related to “juju”, but rather the differences in how different linux distributions work and operate.

I think that bash-only (non python dependent) might work straight out of the box, but I’m not sure.

Centos images in MAAS

I’ve added two packages from a stock centos net-install image:

  • python3
  • redhat-lsb-core

After that, by “bash-only” reference charm goes along just fine.

juju deploy cs:~erik-lonroth/tiny-bash-centos

Now, my attention is towards “reactive” charms.

The “python3” package pulls in “python3.6” on at least centos7.7. This might work on cento7.6 but I’m sure there will be problems on centos6, but I will not even try to go there. Python3.6 seems to be some kind of minimal requirement.

A change request would be to have Canonical to add in python3 in MAAS builds and rebuilding images would at least go away…

Once the python3.6 prepped image boots up properly in MAAS, my juju controller can deploy bash-hooks-only charms just fine. I test this with my reference charm: https://jujucharms.com/u/erik-lonroth/tiny-bash-centos

Now, I’m turning to reactive charms for centos7.

Getting reactive charms going with centos7

So, first layer-basic has code that is ubuntu only. Have a look here: https://raw.githubusercontent.com/juju-solutions/layer-basic/master/lib/charms/layer/basic.py

What this code basically does, is to bring in needed python packages, which should be easy to replicate on centos7 at least.

Determining the “series” needs to happen here, something like the code I’ve cooked up as an example below:

def lsb_release_centos_ubuntu():
    """series for ubuntu and centos like os:es"""
    series = ""
    if os.path.isfile('/etc/lsb-release'):    # Ubuntu                                                                                                                                                             
        d = {}
        with open('/etc/lsb-release', 'r') as lsb:
            for l in lsb:
                k, v = l.split('=')
                d[k.strip()] = v.strip()
            series = d['DISTRIB_CODENAME']

    elif os.path.isfile('/etc/redhat-release'): # Centos/rhel                                                                                                                                                      
	with open('/etc/redhat-release', 'r') as redhatlsb:
            # CentOS Linux release 7.7.1908 (Core)                                                                                                                                                                 
            # return the major release number from the string.                                                                                                                                                     
            for l in redhatlsb:
                words = l.split()
                release = int(words[3][0])
                series = "centos{}".format(release)
    else:
        series = "unknown"

    return series

There is basically from this a matter of installing the correct packages in the same file (layer/basic.py), using “yum” instead of “apt” and selecting the proper package-names mapped to centos7 instead of ubuntu. This is easy, but require some testing of course. I’ll get going with that this week.

I know “charm helpers” will need some patching I know @hallback has already looked at, so I will come back to this thread along the way.

By the end of all this, I hope the centos7 experience with juju can be more pleasant than it is today.

4 Likes

In this do you mean python3 in the CentOS images it uses by default? As I’ve not used CentOS on MAAS where do those come from I guess? I assumed they were end user uploaded?

1 Like

Yes, Erik means the CentOS 7 images that can be imported from maas.io from within the MAAS web UI. As of CentOS 7.7 Python 3.6 actually is present in the yum repo “base”, making things a bit easier.

The last time I tried the following packages were required:

python3
python36-PyYAML (from EPEL)
python36-virtualenv (from EPEL)

1 Like

(v) [root@nash-3 ~]# yum install python36-PyYAML python36-virtualenv
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile

  • base: mirrors.glesys.net
  • extras: mirrors.glesys.net
  • updates: mirror.fysik.dtu.dk
    No package python36-PyYAML available.
    No package python36-virtualenv available.
    Error: Nothing to do
    (v) [root@nash-3 ~]#

Would like to avoid epel. But perhaps basic.py could make use of “pip install pyyaml” instead that works…

(v) [root@nash-3 ~]# python
Python 3.6.8 (default, Aug  7 2019, 17:28:10) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import yaml
>>>

Adding in “epel” would make “basic.py” move rather far away from a “stock centos”? I’m not sure whats the best path here.

There are currently 2 major versions included of centos in maas, centos6 and centos7. The minor versions doesn’t show, but supposedly, the latest version is the one supported.

Neither of centos6, nor centos7 have native support for juju (!?). To mitigate that today, we use the ability to upload custom images of centos7 which will override the supported version. This has drawbacks.

  1. We can’t have more than one single image named “centos7” despite that maas can handle that. So, we need to decide with maas to either divert from the “stock” image in maas (basically rolling our own centos) - or - support deploying with juju. We cant have both.
  2. If we want to be able to use juju - at all - with maas + centos. We need to modify the image at least to be able to run “python” charms and most notably reactive charms. (This quickly becomes ugly using packer.io for custom image creation.)

The initial thought I have on this, is that the reactive framework of juju really doesn’t stand for itself with juju and any reactive charm. If I decide or need to write a charm for centos, I’m basically left with hooks and bash or a heavily custom centos-image. That’s problematic. For one - it prevents juju from reaching out to anyone having a problem that is not based on Ubuntu. Thats a pitty.

From my limited knowledge of all the interiors of juju, I would suggest that perhaps make the juju machine agent provide all the needed bits-and-pieces to run reactive charms regardless of linux distro - at least be able to stand up “layer-basic” and perhaps write a “layer-yum” that resembles “layer-apt” which would at least be providing the very basics of a reactive framework for yum based distros. Experienced programmers might even think of a generic “layer-package-installer”, but I’m not that person =)

I’ll keep you all posted on the progress and findings.

Below the “images” pane which shows the the two default options for centos in MAAS.

So, at the moment I’m stuck more specifically on not having an available “yaml.py” module from the stock Centos7 image used by MAAS.

This further strengthen the suggestion I have to somehow make the juju agent provide the full needed support for at least getting a proper python environment ready for everything needed by:

  • includes: [‘layer:options’] —> This is where “yaml.py” first causes problems
  • charmhelpers
  • charms.reactive

What is installed in basic.py

I unstuck myself by adding in a yaml.py module from centos EPEL repos as @hallback suggested.

sudo yum install epel-release
sudo yum install python36-yaml

But, looking in basic.py, there are a number of packages that’s being installed in. Why or for what purpose some of them are getting installed is unclear to me… “build-essentials” for example?

I do however need to modify the stock centos image (!!!) to add in 3 packages I think:

  1. python3
  2. epel-release
  3. python36-PyYAML

This will likely produce an image that takes this a bit further along with a patch I need to layer-basic.

@rick_h Who can I approach to improve this considering I’m digging into layer-basic at the moment?

1 Like

This is the normal ubuntu package that brings in make and other build tools like a compiler on Ubuntu. It lets you bring in Python deps and have the start of the tools needed to build ones with c-modules.

@cory_fu is the main driver but this is really interesting feedback for him as well as folks like @jameinel as they work on improving the charming experience. This idea of Juju providing some sort of bootstrap on the units as we standardize is interesting. The main issues would be delivery since we would want to change/improve that bootstrap w/o having to be too tied to juju releases themselves though.

Can this be done by setting default cloud-init data in MAAS? In this way all instance would have this setup?

If not there, it can be done with custom cloud-init data with Juju itself.

My point is that leveraging cloud-init means you don’t have to customize the image, but basically create a “pre step” during deployments.

1 Like

I will try this above since it seems to be able to provide a shorter path to getting at first “yaml” on to the nodes in models using centos.

But I still think that basic reactive charms would be possible to get going without this preparation. Oh, given python3 perhaps as a single requirement =/ At least for charms.reactive kind of things…

At the moment, I’m down to trying to understand how layer-options could be made to include “pyyaml” as a wheelhouse installation. But so far I haven’t been able to figure out how.

https://github.com/juju-solutions/layer-options/blob/master/lib/charms/layer/options.py

After that, @hallback:s fixes and some in layer-basic should perhaps be enough to get a better start for centos-like: distros.

I tried your recommendation, with no avail. Below is the situation.

  • Added config to the model
  • Juju deploy a minimal centos7 reactive charm to maas
  • The node come alive
  • Errors related to “yaml” (ignore that for now)
  • The package “epel-release” which I hoped would be installed is not.

I’ll keep going =)

… This could perhaps have to do with that the package “epel-release” needs to import pgp-keys… ???

I’m making another attempt like this now, trying to get a modified “basic.py” and @hallback fixes to reactive.charms

Following the advice here: https://serverfault.com/questions/988874/how-can-i-enable-epel-release-before-installing-packages-with-cloud-init

I really need to get some LXD instances up to be able to test this faster… Machines takes forever to boot.

K, can you file a bug on the centos not working with the additional packages? That seems worth making right with our current setup.

I was definitely thinking you’d have to leverage the preruncmd to get that setup ahead of time, but could make it consistent and automated using that.

Definitely agree that testing that setup up for charming on lxd is great.

1 Like

Hey @rick_h, I will update on this soon as I actually got something that worked efter a few tries with your suggestions.

I have basically used the model config adding in a repo, added in a few packages, modified layer-basic and the patch from @hallback.

With all that, I deployed the minimal centos reactive charm.

I would consider that a very complex activity today but could be greatly reduced with the code fixes.

But I think it highlights the need to work on getting charms out of being a ‘Ubuntu only’ tool.

1 Like

No doubt, not defending doing nothing here at all. I think this is just really good that we can use this to help work out and validate what needs doing, what doesn’t, etc.

For speeding up development I wonder if the cloud-init config can be made a model-default value so that all models on the controller have it and kind of allow for quicker iteration, but again, that’s a hack not a solution.

1 Like

Thanx @rick_h I’m not holding you guilty =) I’m ready to pitch in with my findings and contribute with thoughts on getting juju stronger. Its a great tool with potential to be even better. A good thought is to strive to get juju linux agnostic if possible.

Did anyone say ‘snap’?

2 Likes

@rick_h

I have produced and tested a patch for layer-basic https://github.com/juju-solutions/layer-basic/pull/148

This works along-side with this patch:https://github.com/erik78se/charm-helpers/tree/hallback/centosfixes which @hallback might want to create a PR for.

I’m not sure I should make the PR against master, but I’ll fix that if needed.

Here is a charm that deploys on my local maas with NO modifications to the Centos7 image:

The charm has the patches I mentioned above and the model contains only this.

$ juju model-config cloudinit-userdata
yum_repos:
  epel-release:
      baseurl: http://download.fedoraproject.org/pub/epel/7/$basearch
      enabled: true
      failovermethod: priority
      gpgcheck: true
      gpgkey: http://download.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-7
      name: Extra Packages for Enterprise Linux 7 - Release
packages:
  - 'python3'
  - 'python36-PyYAML'

However, the direct drawback, is that having BOTH centos and ubuntu series within the same model will become impossible as package-names will differ and cloud-init will bork.

OK, @rick_h @hallback with the following “hack” I have managed to get running a model with mixed centos/ubuntu in a lxd localhost cloud behind a corporate proxy.

The experience achieving this was horrible, but at least its possible without messing up the stock images with centos.

It deploys a centos7 reactive charm I made for the purpose: cs~erik-lonroth/tiny-reactive-centos along with the cs~erik-lonroth/tiny-bash, which is a non reactive ubuntu charm.

The trick I had to use, was to make yum use the proxy config with the ‘–setopt=’ argument. This is needed to happen ‘before’ the preruncmd is executed. Installing packages from the “EPEL” repo is not possible otherwise since the proxy is not set at this moment. Patching cloud-init to allow the yum proxy to be configured as a configuration item in the ‘cloudinit-userdata’ would have made this alot nicer.

Also, a configuration for “yum” in the model config might make sense down the road.

This is how I end up for now prepping my model and deploying charms:

modelconfig.yaml

cloudinit-userdata: |
  yum_repos:
    epel-release:
      baseurl: http://download.fedoraproject.org/pub/epel/7/$basearch
      enabled: true
      failovermethod: priority
      gpgcheck: true
      gpgkey: http://download.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-7
      name: Extra Packages for Enterprise Linux 7 - Release
  preruncmd:
    - yum --setopt=proxy=http://some.proxy.com:8080 -y install epel-release redhat-lsb-core sudo openssh-server python36-virtualenv python36-PyYAML python3 snapd
    - echo "proxy=http://some.proxy.com:8080" >> /etc/yum.conf
  postruncmd:
    - usermod -p '*' ubuntu
    - 'systemctl start sshd'
    - 'systemctl enable snapd'

http-proxy: http://some.proxy.com:8080
https-proxy: http://some.proxy.com:8080
snap-http-proxy: http://some.proxy.com:8080
snap-https-proxy: http://some.proxy.com:8080

After takin this path, I’ve now come to try to deploy a centos-charm. A very basic one really. But that is not working out so good.

I started off with a few attempts like:
“charm create -t python-basic subsub”.
“charm create -t python subsub”

The outcome renders always into a fail/error on centos.

unit-subsub-0: 15:27:05 DEBUG unit.subsub/0.install Traceback (most recent call last):
unit-subsub-0: 15:27:05 DEBUG unit.subsub/0.install File “/var/lib/juju/agents/unit-subsub-0/charm/hooks/install”, line 8, in
unit-subsub-0: 15:27:05 DEBUG unit.subsub/0.install from charmhelpers.core import (
unit-subsub-0: 15:27:05 DEBUG unit.subsub/0.install ImportError: No module named charmhelpers.core
unit-subsub-0: 15:27:05 ERROR juju.worker.uniter.operation hook “install” failed: exit status 1
unit-subsub-0: 15:28:27 DEBUG unit.subsub/0.install Traceback (most recent call last):
unit-subsub-0: 15:28:27 DEBUG unit.subsub/0.install File “/var/lib/juju/agents/unit-subsub-0/charm/hooks/install”, line 8, in
unit-subsub-0: 15:28:27 DEBUG unit.subsub/0.install from charmhelpers.core import (
unit-subsub-0: 15:28:27 DEBUG unit.subsub/0.install ImportError: No module named charmhelpers.core
unit-subsub-0: 15:28:27 ERROR juju.worker.uniter.operation hook “install” failed: exit status 1

Even if this error might be for something not directly related to centos, the three most critical pieces that all fails today are:

  1. charmhelpers is broken for any other OS but ubuntu. (!)
  2. layer-basic is broken for any other OS but ubuntu.
  3. charm-tools (e.g. charm create -t ) will never produce a working charm for anything but Ubuntu.

There are a number of PR:s out for charmhelpers and layer-basic, which would perhaps restore some support for non-ubuntu charms, but there is not much activity in those repos or response on the PR:s made?

https://github.com/juju-solutions/layer-basic/pull/148

https://github.com/juju/charm-helpers/pull/400

It’s very difficult to get a proper path working with all obstacles needing to be covered for to be able to juju with centos.

The current state of juju is that it does not support anything but Ubuntu. I think that is something that MUST change.

I think any future framework for charming needs to carefully consider a much more general linux support for juju/charms rather than Ubuntu only.

For now, is the only remedy is to use hooks only and bash to create my charms?

@rick_h @hallback @Dmitrii @timClicks