The new default instance-type for AWS seems to have changed to a t3.micro, which only has < 1G ram, causing applications that previously deployed successfully with the default instance-type to now fail.
Example: Any java application that sets the default jvm to 1G [0] fails with [1].
James is this in 2.5? I know we setup guards to make sure instance sizes stayed consistent with the new instance types in 2.3/2.4 but we are planning to evaluate if we should be adjusting these defaults before 2.5 goes final now that there’s a new array of instance types available. The baked in constraints have always been 1gb of ram but there wasn’t an instance that matched that so historically folks got 2. Personally I think that 2gb seems a reasonable default but I wanted to double check other clouds and see what the cost/etc would be.
To replicate, juju deploy cs:~omnivector/bundle/elk-0 on an AWS model running 2.4.4.
I think its the controller version that matters, not the model. I can switch to models that haven’t been upgraded to 2.4.4 yet and deploy something without constraints and get a t3.micro. I’ll try with an older controller.
@jameinel@rick_h right so I spun up a 2.4.3 controller and this is what I got
$ juju bootstrap aws/us-west-2
$ juju deploy ubuntu
$ juju status
Model Controller Cloud/Region Version SLA Timestamp
default aws-us-west-2 aws/us-west-2 2.4.3 unsupported 16:02:32-07:00
App Version Status Scale Charm Store Rev OS Notes
ubuntu 18.04 active 1 ubuntu jujucharms 12 ubuntu
Unit Workload Agent Machine Public address Ports Message
ubuntu/0* active idle 0 52.12.220.196 ready
Machine State DNS Inst id Series AZ Message
0 started 52.12.220.196 i-0f635740ef6291497 bionic us-west-2a running
$ juju run --application ubuntu "curl http://169.254.169.254/latest/meta-data/instance-type"
m3.medium
This shows that the default instance-type previously was a m3.medium.
We went from a 4G of ram default instance-type in juju < 2.4.4 to 1G default instance-type in 2.4.4.
Possibly the t3.small is a good compromise moving forward?
Juju controllers always use a custom request, something like “at least 3.5GB of ram and 2CPU”. That is different from the default workload request which is “1GB of RAM, 1 CPU”.
I believe you can also just do a model-config for default constraints if you know you always want 2GB.
I get this problem all the time, I have now successfully beef-caked the ELK, both ELK versions and in both AWS and GCE. I have failed so many times now - I feel sorry for anyone trying deploy a simple ELK stack in AWS or GCE.
For the ELK stack, both elasticsearch and logstash runs out of memory.
In AWS, I get away with setting elasticsearch and logstash constraints: mem=4G (That took me two days to resolve only partially)
@rick_h Would it be a good idea to add “Recommended cpu,mem,disk” options for charms to fire off warnings to someone deploying charms with “too small” units as this causes very random problems that are very difficult for users to mitigate and even understand? It took me a long time to figure out if it was me, the bundle, the charms, the software - that in fact causes me to fail deploying stuff. A warning “Not meeting recommended resources” would at least have given me a hint that something would be very wrong.
I just encountered it again. This is how its presented to a user:
erik@pinky:~$ juju status
Model Controller Cloud/Region Version SLA Timestamp
tut jaas aws/eu-west-1 2.4.7 unsupported 21:56:33+01:00
App Version Status Scale Charm Store Rev OS Notes
filebeat 5.6.13 waiting 1 filebeat jujucharms 19 ubuntu
logstash blocked 1 logstash jujucharms 1 ubuntu
pyapp-snapped 0.4 active 1 pyapp-snapped jujucharms 6 ubuntu
Unit Workload Agent Machine Public address Ports Message
logstash/1* blocked idle 2 34.242.194.175 Unable to install packages logstash
pyapp-snapped/0* active idle 0 34.241.251.172 Ready pyapp-0.4 (edge)
filebeat/0* waiting idle 34.241.251.172 Waiting for: elasticsearch, logstash or kafka.
Machine State DNS Inst id Series AZ Message
0 started 34.241.251.172 i-023ab4f845168b571 bionic eu-west-1a running
2 started 34.242.194.175 i-042590c0c98bac05c bionic eu-west-1c running
Basically, no clue whatsoever that this would be a resource problem. Just a mystical “Unable to install packages logstash”. Good luck for a beginner user to figure our whats the problem. (E.g. not going to happen)
So, then I do this instead (adding constraints):
erik@pinky:~$ juju deploy cs:~omnivector/logstash-1 **--constraints mem=4G**
Located charm "cs:~omnivector/logstash-1".
Deploying charm "cs:~omnivector/logstash-1".
erik@pinky:~$ juju status --relations
Model Controller Cloud/Region Version SLA Timestamp
tut jaas aws/eu-west-1 2.4.7 unsupported 22:06:40+01:00
App Version Status Scale Charm Store Rev OS Notes
filebeat 5.6.13 active 1 filebeat jujucharms 19 ubuntu
logstash 6.5.2 active 1 logstash jujucharms 1 ubuntu
pyapp-snapped 0.4 active 1 pyapp-snapped jujucharms 6 ubuntu
Unit Workload Agent Machine Public address Ports Message
logstash/2* active idle 3 34.247.84.128 Logstash running - version 6.5.2
pyapp-snapped/0* active idle 0 34.241.251.172 Ready pyapp-0.4 (edge)
filebeat/0* active idle 34.241.251.172 Filebeat ready.
Machine State DNS Inst id Series AZ Message
0 started 34.241.251.172 i-023ab4f845168b571 bionic eu-west-1a running
3 started 34.247.84.128 i-098fa1156a84f631e bionic eu-west-1b running
Relation provider Requirer Interface Type Message
logstash:beat filebeat:logstash elastic-beats regular
pyapp-snapped:juju-info filebeat:beats-host juju-info subordinate
… Then all works just fine.
Conclusion: We really need some kind of facility to help users get an indication on how to scale a unit for load in terms of hardware: “cpu,memory,disk,gpu, etc.”
This is my fault/bad practice as a charm author. I should notate in the readme the resources needed for the application, and make a handler in the charm that sets a blocked status and clearly communicates a meaningful message to the user if the amount of needed resource is not available.
I think its a mistake that everyone will make since there is no clear way of structurally present it to either the developer or the user.
Adding a charm metadata option that indicate constraints on resources (cpu,mem,disk,net,cpu, etc?) would help everyone. Also, it would be possible to fire off early warnings from a juju deploy if the indicated resources are not met.
