I’m trying to deploy a charm mentioned in post Getting Started that involves the aws-integrator charm:
juju bootstrap --credential juju-k8s --config charmstore-url=https://api.staging.jujucharms.com/charmstore aws aws-k8s
juju deploy cs:~johnsca/kube-core-aws
juju trust aws-integrator
I end up with a failure.
Credential ‘juju-k8s’ is associated with the AWS ‘AdministratorAccess’ policy and the custom policy I created specifically for ‘aws-integrator’.
Any ideas?
The credential access rights look correct to me. What’s the content of the hook error as seen in the debug logs? That will help the aws integrator charm folks diagnose the issue.
the first thing I would do is use the latest aws-integrator charm. Current revision is 7, and you seem to have 4
juju upgrade charm should do it
Doesn’t help. I guess the staging Charm Store, which I need, doesn’t have the newer revision.
juju upgrade-charm aws-integrator
ERROR already running latest charm "cs:~johnsca/aws-integrator-4"The JAAS team helpfully sync’d the ~containers space. Try using the charm straight from there. It should be the latest version as of yesterday.
I downloaded the bundle I was using and replaced the default aws-integrator charm with, as Rick suggested:
charm: "cs:~containers/aws-integrator"
This helped, yet I still get a permission error:
As a test I’ve granted to my Juju credential the IAM AdministratorAccess role. I also, probably futilely, added all IAM “actions” within the “EC2 service”. I’m not sure how to give more than that.
That error is coming from this query:
subnet_id = _aws('ec2', 'describe-instances',
'--instance-ids', instance_id,
'--region', region,
'--query', 'Reservations[*]'
'.Instances[*]'
'.SubnetId[] | [0]')
I’ve never seen that fail before and I’m not sure what could cause it. Possibly the credentials provided to the integrator charm lacking the EC2 DescribeInstances permission, or maybe some sort of issue with the subnet being attached? Maybe you can try manually running that query and provide the results?
I have given my user all the permissions I can think of (as stated earlier). This is the user whose credentials Juju is using to bootstrap.
As for manually querying, I presume I need to install an AWS CLI client. I’ll need to set that up.
You could also just ssh into the aws-integrator unit and run the query from there. If you do the following, the credential will be already configured:
sudo su
export HOME=/root
Also, make sure to leave off the final | [0] bit of the query to see all of the subnet info, and possibly all of the .SubnetId[] | [0] bit to see all instance info.
I’m afraid this is going to betray my cluelessness.
I tried running the function as is to begin and got an error:
# cd /var/lib/juju/agents/unit-aws-integrator-0/charm/lib/charms/layer
# python3 -c 'from aws import *; print tag_instance_subnet(instance_id, region, tags)'
File "<string>", line 1
from aws import *; print tag_instance_subnet(instance_id, region, tags)
^
SyntaxError: invalid syntaxI made a change to how I called the function and got at least some semblance of next steps:
python3 -c 'from aws import *; print(tag_instance_subnet(instance_id, region, tags))'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/var/lib/juju/agents/unit-aws-integrator-0/charm/lib/charms/layer/aws.py", line 14, in <module>
from charmhelpers.core import hookenv
ModuleNotFoundError: No module named 'charmhelpers'The ModuleNotFound error seems like you need to ensure that the charm’s venv is activated. You could try using charm-env python3 ... instead. However, even if that runs, I suspect that it would just give you the same error that’s in the Juju log. Instead, I think you need to run a modified form of the query directly, to see what values the query is coming back with for the subnets.
Would you mind pasting the results of the following, filling in the $REGION and $INSTANCE_ID from the output of juju status:
juju run --unit aws-integrator/0 -- aws ec2 describe-instances --profile juju --output json --instance-ids $INSTANCE_ID --region $REGION --query 'Reservations[*].Instances[*].SubnetId[]'I’d first like to re-explain my environment since my initial post. Since we spoke (on IRC) about bundle ‘kube-core-aws’ I have since moved to just using bundle ‘canonical-kubernetes’. I then installed ‘aws-integrator’ separately (juju deploy cs:~containers/aws-integrator).
After having run juju trust aws-integrator the final result is here.
Pre-pending charm-env gave me a different error:
charm-env python3 -c 'from aws import *; print(tag_instance_subnet(instance_id, region, tags))'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/var/lib/juju/agents/unit-aws-integrator-0/charm/lib/charms/layer/aws.py", line 22, in <module>
MODEL_UUID = os.environ['JUJU_MODEL_UUID']
File "/var/lib/juju/agents/unit-aws-integrator-0/.venv/lib/python3.6/os.py", line 669, in __getitem__
raise KeyError(key) from None
KeyError: 'JUJU_MODEL_UUID'
Details here.
Now the result of the juju run command:
juju run --unit aws-integrator/0 -- aws ec2 describe-instances --profile juju --output json --instance-ids i-09128b6fbacebdb32 --region us-east-1 --query 'Reservations[*].Instances[*].SubnetId[]'
/tmp/juju-exec418336576/script.sh: line 1: aws: command not found
I remain vexed by the aws-integrator charm. I left off last time with my only hope being that my
AWS account was somehow borked due to it being of some legacy nature. Well I’m using a fresh
account with Administrator privileges (arn:aws:iam::aws:policy/AdministratorAccess)
and it still doesn’t work.
juju bootstrap aws aws-k8s
juju deploy kubernetes-core
juju deploy cs:~containers/aws-integrator
juju trust aws-integrator
juju status
juju scp kubernetes-master/0:config ~/.kube/config
juju add-k8s k8s-cloud
juju add-model k8s-model k8s-cloud
juju create-storage-pool operator-storage kubernetes \
storage-class=juju-operator-storage \
storage-provisioner=kubernetes.io/aws-ebs parameters.type=gp2
juju deploy cs:~juju/gitlab-k8s
juju status
kubectl -n k8s-model describe sc,pv,pvc
Kubernetes error:
Failed to provision volume with StorageClass "k8s-model-juju-operator-storage": Failed to get AWS Cloud Provider. GetCloudProvider returned <nil> instead
I ran the aws tool that’s installed on the aws-integrator/0 unit machine to describe the
instance hosting units etcd/0 and kubernetes-master/0:
juju run --unit aws-integrator/0 -- HOME=/root \
/snap/bin/aws ec2 describe-instances \
--profile juju --output json --instance-ids i-047adb8f82f0fbf7e --region us-east-1
The last time I did this I got an empty set but this time it gave me full output:
Anyone? @cory_fu?
I don’t see where you added the relation between the aws-integrator and both of kubernetes-master and kubernetes-worker. Was that just an omission from the copy-paste here?
It’s also worth noting that, while the charm should generally handle a late addition of an integrator charm gracefully, there have been some known issues within Kubernetes itself having trouble switching over (possibly due to some corner-cases requiring restarting of some of the control-plane pods which we haven’t pinned down and added to the charms yet). So it’s usually recommended to deploy the bundle with an overlay, as recommended in the integrator charm’s README.
Oh yes, there was a typo. I tried again and this time I got both a failure and a success. I then tried with an overlay and it worked. Gratitude.