Docker limitation: Can't run a new command in a stopped container

[davidonlaptop]

Problem

Docker does not (currently as of v1.5) allows a user to run a new command in a stopped container.

If the container is running, we can

• use docker exec to spawn a new command in the container from the host
• use docker attach or nsenter to "enter" the container and manually run a command

If the container is stopped, we can

• use docker start to restart the container with the same command used to launch it (via docker run, docker create, or as specified by the Dockerfile instruction CMD or ENTRYPOINT).
• docker restart works the same way as docker start except for a running container.

The use case would be to start a shell to debug the log files and other state on the disk after the process has crashed. (Or to explore a data-only container.)

See https://github.com/jpetazzo/nsenter/issues/27, moby/moby#1437, moby/moby#1228

Explanation

Containers should be ephemeral
The container produced by the image your Dockerfile defines should be as ephemeral as possible. By “ephemeral,” we mean that it can be stopped and destroyed and a new one built and put in place with an absolute minimum of set-up and configuration.

https://docs.docker.com/articles/dockerfile_best-practices/

Solution

All the container state (data, logging) that requires persistence MUST be stored outside the container's main layer.

• For the data, we should be using the Dockerfile instruction "VOLUME"
• For the logging, we need to explore solutions.

[flangelier]

For logging, can't we just use another "volume" instruction?

[davidonlaptop]

@flangelier: that would work. But I think it is worth looking into the docker logging best practices to understand the whole picture. It seems people have been using more elaborate solutions - why?: https://www.google.ca/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=docker%20logging

[davidonlaptop]

for reference, here's how to configure the log directory for Hadoop: http://stackoverflow.com/questions/22856958/hadoop-logs-value-of-environment-variable-hadoop-log-dir

All the container state

I've got news for you buddy, the containers state is the entire filesystem it resides on. Any part of that may be of interest to someone for whatever reason...if your main command starts crashing for some reason you may want to go on as /bin/bash and have a real look around and see what happened. Docker makes that impossible for no good reason.

[mikefaille]

Docker makes that impossible for no good reason.

To get shell :

$ docker ps   # check container interesting container
$ docker exec -ti <container_id> /bin/bash

Cheers

[davidonlaptop]

@mikefaille,

This command would not work if you're container crashes on start. If that
happens you're stuck, there is no way you can exec into the container. For
this to happen, docker would need to allow overriding the entrypoint
parameter on docker start command.

On Wed, Sep 2, 2015 at 8:55 AM, mikefaille notifications@github.com wrote:

Docker makes that impossible for no good reason.
To get shell :
$ docker ps # check container interesting container
$ docker exec -ti /bin/bash

Cheers

—
Reply to this email directly or view it on GitHub
#13 (comment).

[mikefaille]

You're right. I was in potatoes. (french expression)

[sebastienbonami]

I'm guessing the problem only occurs when the entrypoint is not interactive, right?

A workaround would be to always have an interactive entrypoint like /bin/bash to keep the container alive. Then, use this to enter a shell for a stopped container:
$ docker start -ai <container_id>

[davidonlaptop]

That's fine for debugging, but not in production.

I think the best solution is to just mount (using "-v") everything that can have a debugging value: data, log files, etc.

[sebastienbonami]

The /var partition could be in fact mounted.

Also, you can look at the "docker logs" command (https://docs.docker.com/reference/commandline/logs/) that shows the output. Can be useful to understand why a container has crashed.

[mikefaille]

docker logs show only ouput from PID1's stdout. My 2 cents. It could be useful for sure.

Maybe, starting ENTRYPOINT manually from docker exec -ti <image_name> /usr/bin/bash could reveal some info too.

Also, while each docker images is identical, for most case, we must expect same behavior between stopped/crashed container and container started from docker exec as I point.