Docker

If possible we aim to dockerize (containerize, virtualize) the applications. The main advantage of this approach is that we can provide identical environments across servers, including development, staging and production.

We regularly have Docker meetings to discuss open issues, and transform them into policies.

1. Best practices (external)

​http://developerblog.redhat.com/2016/02/24/10-things-to-avoid-in-docker-containers/

2. Open issues

1. How to build containers? Packer vs. Dockerfile?
2. Where to store application and container configuration/build information, and how?
3. Should we maintain our own Docker Registry?
4. How do we manage logging from within containers?
5. Which Linux distro for containers? Security aspects? Alpine Linux v. other?
6. Backup strategy? Disk quota? Snapshots, tarballs within container, etc.?
7. ...

3. Policies

3.1. Building images

3.1.1. Layering inheritance

• One base image. If possible, this image is both a demo and production image.
• If not possible: separate demo and production images that inherit from the base image.

3.1.2. Packer

...

3.1.3. Git

1. Repo naming?
2. Dependencies on other images?

3.1.4. Application packaging

• We will always use ​dumb-init as the entry point process inside containers.

## Deploying into containers

1. docker rename existing and possibly running containers, volumes, networks, etc.
2. Create containers.
3. docker stop old_container && docker start new_container.

3.2. Data

Kind of data	How to use it with Docker
Variable data on which app configuration does not directly depend	on volume per content set
Application configuration	within image of application
Global configuration (used by more than one container)	on volume per type of configuration
Secrets (passwords, private key files)	on isolated volume (exactly the same across hypervisors)

3.2.1. Backup data in a container

Using this method you can create backups outside your container of any data stored in a volume.

General command:

docker run -ti --rm --volumes-from <container_name> -v <host directory>:<container directory> ubuntu <backup command>

Examples:

### Docker registry backup ###
docker run -ti --rm --volumes-from registry_volume -v /scratch:/backup ubuntu tar-pczvf /backup/registry_data.tgz /etc/registry /srv/registry-data

#### Nexus backup ####
docker run -ti --rm --volumes-from nexus_volume -v /scratch:/backup ubuntu tar -pczvf /backup/nexus_data.tgz /sonatype-work

#### Nginx backup ####
docker run -ti --rm --volumes-from nginx_volume -v /scratch:/backup ubuntu tar -pczvf /backup/nginx_data.tgz /etc/nginx /usr/share/nginx/html /var/log/nginx

3.2.2. Restore data in a container

## Naming

## Volumes: ... Volume containers: ... Containers: ... Images: ... Git repositories:

3.3. Managing containers

• Willem has worked on creating shell scripts to manage Docker containers based on images and Dockerfiles.
• Sander has worked on creating shell scripts to manage Docker containers based on Packerfiles.

Listing running containers:

docker ps

Listing all (including stopped) containers:

docker ps -a

Starting, stopping and restarting containers:

docker [start|stop|restart] <container_name>

Connecting to a container to e.g. look at log files or modify configuration:

docker exec -ti <container_name> /bin/bash

Tailing the container output:

docker logs -f --tail=100 <container_name>

Kill all running containers:

docker kill $(docker ps -q)

3.4. Managing images

Listing all images:

docker images

Cleaning up unused (untagged/dangling) images:

docker rmi $(docker images -q -f dangling=true)

3.5. Docker registry at ​https://docker.clarin.eu/

...

4. Dockerized applications

4.1. CLARIN private Docker registry

## pull from Docker registry
docker pull registry:latest
## or import from image export
docker load -i docker_registry.tgz

## Create volume container
docker create --name registry_volume -v /etc/registry -v /srv/registry-data tianon/true

## Create application container
docker create --name registry --volumes-from registry_volume -p 127.0.0.1:5000:5000 -e GUNICORN_OPTS=["--preload"] registry:latest

## Optionally restore data into the volume container
docker run -ti --rm --volumes-from registry_volume -v /data/backup/:/backup debian tar -xzf /backup/registry_data.tgz -C /

## Start the registry container
docker start registry

## Check running containers and registry container state
docker ps
docker logs registry

4.1.1. Past issues

After moving the clarin docker registry from stoor146 to clarinvm and following the above instructions, we ran into the following error when starting the registry container:

OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'

As documented in ​#892, adding the -e GUNICORN_OPTS=["--preload"] resolved the issue

4.2. CLARIN Nexus repository

## Pull from Docker registry
docker pull sonatype/nexus:latest
## or import from image export
docker load -i docker_nexus.tgz

## Create volume container
docker create --name nexus_volume -v /sonatype-work tianon/true

## Create application container
docker create --name nexus --volumes-from nexus_volume -p 127.0.0.1:8081:8081 sonatype/nexus:latest

## Optionally restore data into the volume container
docker run -ti --rm --volumes-from nexus_volume -v /data/backup/:/backup debian tar -xzf /backup/nexus_data.tgz -C /

## Start the registry container
docker start nexus

## Check running containers and registry container state
docker ps
docker logs nexus

4.3. NGinx proxy

## Pull from docker registry
docker pull clarin:nginx
## or import from image export
docker load -i docker_nginx.tgz

## Create volume container
docker create --name nginx_volume -v /etc/nginx -v /etc/nginx/ssl -v /usr/share/nginx/html -v /var/log/nginx tianon/true

## Create application container
docker create --name nginx --volumes-from nginx_volume -v /root/certstore/wildcard-clarin-eu/bundle.cer:/etc/nginx/ssl/bundle.cer -v /root/certstore/wildcard-clarin-eu/privateKey.key:/etc/nginx/ssl/privateKey.key -p 80:80 -p 443:443 --link nexus:nexus --link registry:registry clarin/nginx

## Optionally restore data into the volume container
docker run -ti --rm --volumes-from nginx_volume -v /data/backup/:/backup debian tar -xzf /backup/nginx_data.tgz -C /

## Start the registry container
docker start nginx

## Check running containers and registry container state
docker ps
docker logs nginx

5. References

• Docker volumes
  • ​https://medium.com/@ramangupta/why-docker-data-containers-are-good-589b3c6c749e
• Docker Application configuration
  • ​https://dantehranian.wordpress.com/2015/03/25/how-should-i-get-application-configuration-into-my-docker-containers/
• Docker containers and images visually explained:
  • ​http://merrigrove.blogspot.nl/2015/10/visualizing-docker-containers-and-images.html