Default AWS instance-type t3.micro too small


#1

The new default instance-type for AWS seems to have changed to a t3.micro, which only has < 1G ram, causing applications that previously deployed successfully with the default instance-type to now fail.

Example: Any java application that sets the default jvm to 1G [0] fails with [1].

[0] https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html#heap-size
[1] https://paste.ubuntu.com/p/YnPTbYtsGj/

How do people feel about using a t3.small for the default instance-type on aws, and similarly 2core 2G ram instance-type on other clouds?


#2

Can’t you just set that as an application constraint? So doing “juju deploy JAVAAPP --constraints mem=2G” ?


#3

James is this in 2.5? I know we setup guards to make sure instance sizes stayed consistent with the new instance types in 2.3/2.4 but we are planning to evaluate if we should be adjusting these defaults before 2.5 goes final now that there’s a new array of instance types available. The baked in constraints have always been 1gb of ram but there wasn’t an instance that matched that so historically folks got 2. Personally I think that 2gb seems a reasonable default but I wanted to double check other clouds and see what the cost/etc would be.


#4

Yes, you can. I guess that isn’t a bad modification to carry along.

From a logical standpoint, don’t we want our things to have at least 2G of ram by default anyway?


#5

@rick_h 2.4.4

To replicate, juju deploy cs:~omnivector/bundle/elk-0 on an AWS model running 2.4.4.

I think its the controller version that matters, not the model. I can switch to models that haven’t been upgraded to 2.4.4 yet and deploy something without constraints and get a t3.micro. I’ll try with an older controller.


#6

I’m pretty sure it is whether the controller version is aware of the new AWS instance types.


#7

@jameinel @rick_h right so I spun up a 2.4.3 controller and this is what I got

$ juju bootstrap aws/us-west-2
$ juju deploy ubuntu
$ juju status
Model    Controller     Cloud/Region   Version  SLA          Timestamp
default  aws-us-west-2  aws/us-west-2  2.4.3    unsupported  16:02:32-07:00

App     Version  Status  Scale  Charm   Store       Rev  OS      Notes
ubuntu  18.04    active      1  ubuntu  jujucharms   12  ubuntu  

Unit       Workload  Agent  Machine  Public address  Ports  Message
ubuntu/0*  active    idle   0        52.12.220.196          ready

Machine  State    DNS            Inst id              Series  AZ          Message
0        started  52.12.220.196  i-0f635740ef6291497  bionic  us-west-2a  running

$ juju run --application ubuntu "curl http://169.254.169.254/latest/meta-data/instance-type"
m3.medium

This shows that the default instance-type previously was a m3.medium.
We went from a 4G of ram default instance-type in juju < 2.4.4 to 1G default instance-type in 2.4.4.

Possibly the t3.small is a good compromise moving forward?


#8

Juju controllers always use a custom request, something like “at least 3.5GB of ram and 2CPU”. That is different from the default workload request which is “1GB of RAM, 1 CPU”.
I believe you can also just do a model-config for default constraints if you know you always want 2GB.

juju model-config constraints=“mem=2G”


#9

^that isn’t a controller , it’s just an instance deployed to the default space following a bootstrap.


#10

Excellent. This should do the trick for the most part.

Thx


#11

I get this problem all the time, I have now successfully beef-caked the ELK, both ELK versions and in both AWS and GCE. I have failed so many times now - I feel sorry for anyone trying deploy a simple ELK stack in AWS or GCE.

For the ELK stack, both elasticsearch and logstash runs out of memory.

In AWS, I get away with setting elasticsearch and logstash constraints: mem=4G (That took me two days to resolve only partially)

@rick_h Would it be a good idea to add “Recommended cpu,mem,disk” options for charms to fire off warnings to someone deploying charms with “too small” units as this causes very random problems that are very difficult for users to mitigate and even understand? It took me a long time to figure out if it was me, the bundle, the charms, the software - that in fact causes me to fail deploying stuff. A warning “Not meeting recommended resources” would at least have given me a hint that something would be very wrong.

It could go in here as a new type of charm-metadata: https://docs.jujucharms.com/2.4/en/authors-charm-metadata


#12

I just encountered it again. This is how its presented to a user:

erik@pinky:~$ juju status
Model  Controller  Cloud/Region   Version  SLA          Timestamp
tut    jaas        aws/eu-west-1  2.4.7    unsupported  21:56:33+01:00

App            Version  Status   Scale  Charm          Store       Rev  OS      Notes
filebeat       5.6.13   waiting      1  filebeat       jujucharms   19  ubuntu  
logstash                blocked      1  logstash       jujucharms    1  ubuntu  
pyapp-snapped  0.4      active       1  pyapp-snapped  jujucharms    6  ubuntu  

Unit              Workload  Agent  Machine  Public address  Ports  Message
logstash/1*       blocked   idle   2        34.242.194.175         Unable to install packages logstash
pyapp-snapped/0*  active    idle   0        34.241.251.172         Ready pyapp-0.4 (edge)
  filebeat/0*     waiting   idle            34.241.251.172         Waiting for: elasticsearch, logstash or kafka.

Machine  State    DNS             Inst id              Series  AZ          Message
0        started  34.241.251.172  i-023ab4f845168b571  bionic  eu-west-1a  running
2        started  34.242.194.175  i-042590c0c98bac05c  bionic  eu-west-1c  running

Basically, no clue whatsoever that this would be a resource problem. Just a mystical “Unable to install packages logstash”. Good luck for a beginner user to figure our whats the problem. (E.g. not going to happen)

So, then I do this instead (adding constraints):

erik@pinky:~$ juju deploy cs:~omnivector/logstash-1 **--constraints mem=4G**
Located charm "cs:~omnivector/logstash-1".
Deploying charm "cs:~omnivector/logstash-1".

erik@pinky:~$ juju status --relations
Model  Controller  Cloud/Region   Version  SLA          Timestamp
tut    jaas        aws/eu-west-1  2.4.7    unsupported  22:06:40+01:00

App            Version  Status  Scale  Charm          Store       Rev  OS      Notes
filebeat       5.6.13   active      1  filebeat       jujucharms   19  ubuntu  
logstash       6.5.2    active      1  logstash       jujucharms    1  ubuntu  
pyapp-snapped  0.4      active      1  pyapp-snapped  jujucharms    6  ubuntu  

Unit              Workload  Agent  Machine  Public address  Ports  Message
logstash/2*       active    idle   3        34.247.84.128          Logstash running - version 6.5.2
pyapp-snapped/0*  active    idle   0        34.241.251.172         Ready pyapp-0.4 (edge)
  filebeat/0*     active    idle            34.241.251.172         Filebeat ready.

Machine  State    DNS             Inst id              Series  AZ          Message
0        started  34.241.251.172  i-023ab4f845168b571  bionic  eu-west-1a  running
3        started  34.247.84.128   i-098fa1156a84f631e  bionic  eu-west-1b  running

Relation provider        Requirer             Interface      Type         Message
logstash:beat            filebeat:logstash    elastic-beats  regular      
pyapp-snapped:juju-info  filebeat:beats-host  juju-info      subordinate  

… Then all works just fine.

Conclusion: We really need some kind of facility to help users get an indication on how to scale a unit for load in terms of hardware: “cpu,memory,disk,gpu, etc.”


#13

This is my fault/bad practice as a charm author. I should notate in the readme the resources needed for the application, and make a handler in the charm that sets a blocked status and clearly communicates a meaningful message to the user if the amount of needed resource is not available.

Feel free to make an issue on layer-elastic-base for this.


#14

This is my fault/bad practice as a charm author.

I think its a mistake that everyone will make since there is no clear way of structurally present it to either the developer or the user.

Adding a charm metadata option that indicate constraints on resources (cpu,mem,disk,net,cpu, etc?) would help everyone. Also, it would be possible to fire off early warnings from a juju deploy if the indicated resources are not met.