ERROR cannot log into controller

auth
openstack
help-needed

#1

I was using the admin account default credential for too long. I decided to create some users and passwords. Having done so, I can no longer log in:

routergod@juju:~$ juju login -u routergod
please enter password for routergod on openstack:
ERROR cannot log into controller "openstack": cannot get discharge from "https://192.168.200.233:17070/auth": failed to acquire macaroon after waiting: third party refused discharge: discharging macaroon: interaction required

Any clues please? Many thanks in advance!


#2

Can you try logging out and then logging back in? Macaroons only have a 24h shelf life I believe:

juju logout -u routergod
juju login -u routergod

#3

Sure! No cigar unfortunately…

routergod@juju:~$ juju logout -u routergod
ERROR option provided but not defined: -u
routergod@juju:~$ juju logout
Logged out. You are no longer logged into any controllers.
routergod@juju:~$ juju login -u routergod
please enter password for routergod on openstack:
ERROR cannot log into controller "openstack": invalid entity name or password
routergod@juju:~$ juju login -u routergod
please enter password for routergod on openstack:
ERROR cannot log into controller "openstack": cannot get discharge from "https://192.168.200.233:17070/auth": failed to acquire macaroon after waiting: third party refused discharge: discharging macaroon: interaction required

I purposely used the wrong password the first attempt there to show that there is apparently some password check going on.

I had a poke around but can’t find any log of the event on the controller.


#4

Is the controller setup to use an external identity provider?

Juju does do password checks, but it returns a macaroon to the client.

  • Is the ip address shown 192.168.200.233 an ip address of the controller? I think it should be because port 17070 is the default API port on the controller.
  • The third party refused discharge is a bit strange to me. Clearly interaction required is being returned from somewhere. I wonder if there are missing environment variables in the client terminal. However I’m not sure what it would be looking for.

@uros-jovanovic or @cmars do either of you have more insight to this error?


#5

Yes the controller is 192.168.200.233. The controller is not knowingly set to use external identity, can I check that somehow?


#6

Still no resolution for this, can someone suggest debugging approach please?


#7

I prodded someone earlier today. Juju uses macaroons for remembering the login details rather than storing the password locally.

The bakery (part of the macaroon internals) tries to call out interactively to a browser.

Could you try again with this option set?

-B, --no-browser-login  (= false)
    Do not use web browser for authentication

Hopefully this will address the issue.


#8

Thank you but unfortunately I get the same result with -B added.

Apologies for also raising a bug for this - desperation on my part.


#9

Don’t appologise for raising a bug. It looks like there may well be a bug in there.

Are you sure you have the password right? I’ve not come across this edge case before. Are you able to log back in as admin?


#10

I have a bit of a problem there in that I set the admin password but somehow neglected to save the Keepass file! I was planning to try using the instructions that were posted here recently to recover that, once I understand this login issue. Before I lost the password though, attempting to log in as admin produced the same macaroon error.

I am using the correct password for the routergod user. If I purposely use the wrong password I get a more straightforward error message;

ERROR cannot log into controller "openstack": invalid entity name or password

I also created a user for a colleague, indeed that was my first action. At that time I had not attempted to set the admin password. Whatever default token was in place for the admin user was still working at that point. It seems the authorization bit is somehow not working where a password is set.


#11

When a controller is bootstrapped, the admin password is saved in the accounts.yaml file. If you logout and then login, this is removed and turned into a macroon (I think). However, you should be able to find it there.

$ cat ~/.local/share/juju/accounts.yaml 
controllers:
  test:
    user: admin
    password: bd2aca662af1c962934368dc90fa6533
    last-known-access: superuser

#12

Ha! Yeah that got trashed already :roll_eyes: I only have controllers.yaml here now.

I followed Bui Ha’s instruction to reset the admin password in the database, which appears to have worked in that I get the macaroon error when I log in as admin with the new password I set;

routergod@juju:~/.local/share/juju$ juju login -u admin
please enter password for admin on openstack:
ERROR cannot log into controller "openstack": cannot get discharge from "https://192.168.200.233:17070/auth": failed to acquire macaroon after waiting: third party refused discharge: discharging macaroon: interaction required

#13

@routergod :confused:

By “Bui Ha’s instruction”, do you mean this tutorial?


#14

@routergod I went looking for the bug as @cmars asked for debug output. Something is clearly not working here, and I’d love to know what.

I take it you have ssh access to the controller machine since you were able to reset the admin password. Is that right?

If so, is there anything in /var/log/juju/machine-x.log (where x is the machine number) on the controller around the times that the macaroon discharge fails?


#15

@timClicks yes that is the process I used

@thumper I looked in /var/log/juju/machine-0.log initially, there is nothing in there related to this.

I can create error messages in that file by forging connections to controller:17070 with curl, whereby I cause some websocket errors. But attempting to log in using the juju client produces no message at all. This is where my confusion started. Is this a client error or the result of a server one? I have no evidence either way unfortunately. The latest messages in there right now appear to be from the last time I rebooted the controller;

root@bigmetal01:~# tail /var/log/juju/machine-0.log
2019-10-08 19:02:49 ERROR juju.worker.dependency engine.go:663 "api-caller" manifold worker returned unexpected error: [b5fb8e] "machine-0" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused
2019-10-08 19:02:50 WARNING juju.mongo open.go:160 mongodb connection failed, will retry: dial tcp 127.0.0.1:37017: connect: connection refused
2019-10-08 19:02:50 WARNING juju.mongo open.go:160 mongodb connection failed, will retry: dial tcp 192.168.200.233:37017: connect: connection refused
2019-10-08 19:03:25 INFO juju.cmd supercommand.go:57 running jujud [2.5.8 gc go1.11.11]
2019-10-08 19:03:25 DEBUG juju.cmd supercommand.go:58   args: []string{"/var/lib/juju/tools/machine-0/jujud", "machine", "--data-dir", "/var/lib/juju", "--machine-id", "0", "--debug", "--verbose"}
2019-10-08 19:03:25 DEBUG juju.utils gomaxprocs.go:24 setting GOMAXPROCS to 4
2019-10-08 19:03:25 DEBUG juju.agent agent.go:545 read agent config, format "2.0"
2019-10-08 19:03:25 INFO juju.cmd.jujud agent.go:133 setting logging config to "<root>=WARNING;unit=DEBUG"
2019-10-08 19:03:26 ERROR juju.worker.dependency engine.go:663 "api-caller" manifold worker returned unexpected error: [b5fb8e] "machine-0" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused
2019-10-08 19:03:30 ERROR juju.worker.dependency engine.go:663 "api-caller" manifold worker returned unexpected error: [b5fb8e] "machine-0" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused

Oh, you may notice I added a ‘–verbose’ to jujud, it was a shot in the dark and apparently achieved nothing.


#16

@thumper, @timClicks, are you able to suggest anything further please? I am really stuck unable to add compute nodes to this cluster and I really hope not to have to rebuild everything.

Many thanks in advance!


#17

Not one to give up… @cmars @timClicks @thumper CFI

I have managed to replace the jujud on my controller with an unstripped version and I can attach the debugger (dlv). I have traced the authentication flow as far as this;

> github.com/juju/juju/apiserver/stateauthenticator.(*localLoginHandlers).serveLoginPost() ./locallogin.go:114 (PC: 0x2063782)
Warning: debugging optimized function
   109:         }
   110:
   111:         // Provide the client with a macaroon that they can use to
   112:         // prove that they have logged in, and obtain a discharge
   113:         // macaroon.
=> 114:         m, err := h.authCtxt.CreateLocalLoginMacaroon(userTag)
   115:         if err != nil {
   116:                 return nil, err
   117:         }
   118:         cookie, err := httpbakery.NewCookie(macaroon.Slice{m})
   119:         if err != nil {

At this point the client times out with a “context deadline exceeded” message. I will persist, but if there are any clues as to where I should be looking please post them :slight_smile:


#18

@cmars @timClicks @thumper @uros-jovanovic

Here is where I got to, although I don’t really understand what I’m looking at yet.

> github.com/juju/juju/vendor/gopkg.in/macaroon-bakery.v2-unstable/bakery.checkNeedDeclared() /home/ubuntu/go/src/github.com/juju/juju/vendor/gopkg.in/macaroon-bakery.v2-unstable/bakery/service.go:375 (PC: 0x8efaf6)
Warning: debugging optimized function
   370:         if len(needDeclared) == 0 {
   371:                 return nil, fmt.Errorf("need-declared caveat with no required attributes")
   372:         }
   373:         cavInfo.Condition = arg[i+1:]
   374:         caveats, err := checker.CheckThirdPartyCaveat(cavInfo)
=> 375:         if err != nil {
   376:                 return nil, errgo.Mask(err, errgo.Any)
   377:         }
   378:         declared := make(map[string]bool)
   379:         for _, cav := range caveats {
   380:                 if cav.Location != "" {
(dlv) whatis cavInfo
*github.com/juju/juju/vendor/gopkg.in/macaroon-bakery.v2-unstable/bakery.ThirdPartyCaveatInfo
(dlv) whatis cavInfo.Condition
string
(dlv) print cavInfo.Condition
"is-authenticated-user admin"
(dlv) print err
error(*github.com/juju/juju/vendor/gopkg.in/macaroon-bakery.v2-unstable/httpbakery.Error) *{
    Code: "interaction required",
    Message: "interaction required",
    Info: *github.com/juju/juju/vendor/gopkg.in/macaroon-bakery.v2-unstable/httpbakery.ErrorInfo {
            Macaroon: *github.com/juju/juju/vendor/gopkg.in/macaroon%2ev2-unstable.Macaroon nil,
            MacaroonPath: "",
            CookieNameSuffix: "",
            VisitURL: "/auth/login?waitid=dbe4c0905d38656bc6daa20f",
            WaitURL: "/auth/wait?waitid=dbe4c0905d38656bc6daa20f",},
    version: version0 (0),}

I “is-authenticated-user”. Is this perhaps something upstream?


#19

Ok, ignore that nonsense above. I understand the macaroons better now and can read the Golang :blush:

When I follow the login process on jujud, it runs through two iterations - once before and once after I am prompted for my password. On both iterations it ends up at;

github.com/juju/juju/vendor/gopkg.in/macaroon-bakery.v2-unstable/httpbakery/client.go:701

Apparently there is no macaroon presented on the second pass? This despite jujud apparently having authenticated me. My attention is now on what the client is doing.