Connecting to a Running Phoenix Application Deployed on Dokku

2023/08/22

I deploy my Elixir Phoenix applications to a personal VPS managed via Dokku.

Recently, I needed to run a script to populate the production database with geographical data. The best practise in Phoenix is to not put data in migrations, so I set up a seed data file (called priv/repo/seeds.exs):

require Logger

alias MyApp.Geo

Logger.info "Upserting areas..."

nord_ovest = Geo.get_or_create_area(%{name: "Nord-ovest"})
...

In development, this file can be run via mix:

$ mix run priv/repo/seeds.exs
[info] Upserting areas...

You can also run Elixir scripts from within iex:

$ iex -S mix
c "priv/repo/seeds.exs"
[info] Upserting areas...

You can't just run mix or iex -S mix in production, for one of two reasons, which I'll explain below.

dokku enter - Part 1

Dokku has the dokku enter command, but as (at least in my setup) it doesn't echo commands, I have tended to avoid it.

Running Mix in a Dokku Docker Container

So, I tried accessing in the container and running mix from there:

$ docker exec -ti my_app.web.1 bash
$ cd app
$ mix run priv/repo/seeds.exs
bash: mix: command not found

Unfortunately, mix was not in the PATH.

Dokku sets this stuff up via scripts under /app/.profile.d/ which need to be sourced.

So the magic invocation to use is

$ . <(cat /app/.profile.d/*)

Mix Won't Run

When I tried to run it exactly as above, Mix tried to start the application before running the script, and startup failed as the Cowboy server was already running on the designated port:

$ docker exec -ti my_app.web.1 bash
$ cd /app
$ . <(cat /app/.profile.d/*)
$ mix run priv/repo/seeds.exs
06:34:49.673 [error] Failed to start Ranch listener MyAppWeb.Endpoint.HTTP in :ranch_tcp:listen([cacerts: :..., key: :..., cert: :..., ip: {0, 0, 0, 0, 0, 0, 0, 0}, port: 5000]) for reason :eaddrinuse (address already in use)

If, on the other hand, I passed the --no-start parameter, it failed as the process I was starting wasn't starting a database connection:

$ mix run --no-start priv/repo/seeds.exs

06:37:51.168 [info] Upserting areas...
** (RuntimeError) could not lookup Ecto repo MyApp.Repo because it was not started or it does not exist
    (ecto 3.10.3) lib/ecto/repo/registry.ex:22: Ecto.Repo.Registry.lookup/1
    (ecto 3.10.3) lib/ecto/repo/supervisor.ex:160: Ecto.Repo.Supervisor.tuplet/2
    (my_app 0.1.0) lib/colla/repo.ex:2: MyApp.Repo.get_by/3
    (my_app 0.1.0) lib/colla/geo.ex:30: MyApp.Geo.get_or_create_area/1
    priv/repo/seeds.exs:7: (file)

Distributed Elixir

So, what I needed to do was connect to the existing Phoenix process. That way mix wouldn't try to connect to the webserver port and would have the database connection available.

I changed my Procfile to add the sname ("short name") my_app for the node.

web: elixir --sname my_app -S mix phx.server

But, that didn't work:

$ iex --sname console --remsh my_app
                                                              
Erlang/OTP 25 [erts-13.0.4] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [jit:ns]

Could not contact remote node my_app@13a522884ed8, reason: :nodedown. Aborting...

Actually, that :nodedown is not quite true, as I could see from the application logs:

$ dokku logs $DOKKU_APP
2023-08-22T07:43:17.082683150Z app[web.1]: 07:43:17.075 [error] ** Connection attempt from node :console@13a522884ed8 rejected. Invalid challenge reply. **

The remote node was actually receiving the request for connection but is refusing it.

Wrong Cookie

The problem turned out to be the magic cookie.

The Phoenix process is run as herokuishuser:

$ ps afux
root         243  0.0  0.0   4248  3464 pts/0    Ss   06:54   0:00 bash
root         256  0.0  0.0   4524  2976 pts/0    S    06:54   0:00  \_ su - herokuishuser

When the Phoenix application starts, as it has a node name, Erlang generates a file .erlang.cookie in /app, which is herokuishuser's home directory:

$ cat /app/.erlang.cookie
ZHGOXPNPCCICDESCLHCD

I was execing into the Docker container as root.

If I ran iex with a node name, a new cookie was generated for root, which didn't match the one for herokuishuser:

$ iex --sname ciao
$ cat /root/.erlang.cookie 
PANTWFLLYXARWYBRTNUU

So, when I tried to run a remote shell, the correction was refused.

herokuishuser

So, all I needed to do was to invoke Docker with the correct user

$ docker exec -u herokuishuser -ti my_app.web.1 bash
$ cd /app
$ . <(cat /app/.profile.d/*)
$ iex --sname console --remsh my_app
Erlang/OTP 25 [erts-13.0.4] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [jit:ns]

Interactive Elixir (1.14.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(my_app@35fb427c9010)1>

And I finally got a working remote shell in the Phoenix process.

From there, I could run my seed script:

c "priv/repo/seeds.exs"
[info] Upserting areas...
...

Back to dokku enter

Armed with my understanding of what was going on, I can now run dokku enter and have remote iex access to the Phoenix process from my development machine:

$ dokku enter $DOKKU_APP
$ iex --sname console --remsh my_app
Erlang/OTP 25 [erts-13.0.4] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [jit:ns]

Interactive Elixir (1.14.1) - press Ctrl+C to exit (type h() ENTER for help)
c "priv/repo/seeds.exs"

08:53:26.853 [info] Upserting areas...
...

No messing about with setting up the user or PATH.

So, in summary, the solution when using Dokku is:

  1. Give the application a node name in Procfile,

  2. Use dokku enter,

  3. Invoke iex --remsh with that node name.

If you're not using Dokku, and are using containers, the trick is to invoke iex as the same user that started the application.