Contents

1. Tickets
2. 1. Hosts
   1. 1.1. Internally managed
      1. 1.1.1. Production (clarin.eu)
      2. 1.1.2. Beta / Development (clarin-dev.eu)
      3. 1.1.3. Unused
   2. 1.2. Externally managed, with central services
   3. 1.3. Decommissioned
3. 2. DNS entries and TLS certificates
4. 3. Getting access
   1. 3.1. Security
5. 4. Default VM setup
   1. 4.1. Connections
   2. 4.2. Centos / Scientific Linux
   3. 4.3. SLES 11
   4. 4.4. Ubuntu
6. 5. Deploying and running services
   1. 5.1. Deploy a service
   2. 5.2. Initialize a service
   3. 5.3. Start the service
7. 6. Infrastructure and service status information
8. 7. Documents
9. 8. Services
10. 9. Updates
11. 10. Known issues
    1. 10.1. Docker
       1. 10.1.1. dial tcp: lookup index.docker.io: no such host
          1. 10.1.1.1. Error
          2. 10.1.1.2. Symptoms
          3. 10.1.1.3. Solution
          4. 10.1.1.4. References
12. 11. GitLab
    1. 11.1. Managing a git repository on a server with a deploy key

Tickets

Ticket	Priority	Summary	Owner	Created	Modified
#1086	minor	Problem with admin messages from lists.clarin.eu	André Moreira	4 years ago	4 years ago
#1054	major	Bad URL encoding in catalog.clarin.eu/vlo -> vlo.clarin.eu redirects	André Moreira	6 years ago	6 years ago
#1010	minor	Piwik overlay	André Moreira	7 years ago	7 years ago
#1000	major	Review server timezone	André Moreira	8 years ago	8 years ago
#996	major	catalog.clarin.eu SP: aa-statistics error message on login	André Moreira	8 years ago	8 years ago
#981	major	Add DKIM, DMARC DNS RRs for clarin.eu	Sander Maijers	8 years ago	8 years ago
#887	major	Configure local firewall for idp1-clarin and idp2-clarin	Sander Maijers	9 years ago	8 years ago
#881	minor	Migrate lists.clarin.eu	Sander Maijers	9 years ago	8 years ago
#891	critical	dmeventd for LVM hogs CPU on CentOS 7	Willem Elbers	9 years ago	8 years ago
#955	major	Add Reverse DNS records for TransIP VMs	Sander Maijers	8 years ago	8 years ago
#902	major	Grant read access to httpd logs	Sander Maijers	9 years ago	9 years ago
#885	major	Django security update to 1.9.3	Sander Maijers	9 years ago	9 years ago
#884	critical	Security-critical instructions to data centre support should have integrity	Sander Maijers	9 years ago	9 years ago
#883	major	Devise automatic tty I/O auditing setup & policy	Sander Maijers	9 years ago	9 years ago
#865	major	Make host pages adhere to host template in Trac	Sander Maijers	9 years ago	9 years ago
#864	major	Automated collection of running Docker containers	Sander Maijers	9 years ago	9 years ago
#863	minor	Automate collection of host info	Sander Maijers	9 years ago	9 years ago

1. Hosts

1.1. Internally managed

1.1.1. Production (clarin.eu)

Canonical FQDN	Aliases	Services	Ports	Service Type	IPv4-address	OS	(v)CPUs	Memory (GiB)	Storage (GiB)	Hoster	Type	Responsible	Support	Collectd	Fluentd	Docker	Compose
rs238144.rs.hosteurope.de	vlo.clarin.eu	​VLO	8181, 8143	PRODUCTION	91.250.82.71	CentOS	8	40	800	HostEurope	Root Server XL	​sysops@clarin.eu	​support@hosteurope.de	5.8.0	td-agent 1.2.2	18.03.1	1.21.2
clarinvm.cesnet.cz	catalog.clarin.eu docker.clarin.eu nexus.clarin.eu office.clarin.eu	​Netkernel CLARIN EU trac CLARIN NL trac SVN Metrics ​Component Registry ​Discovery service		PRODUCTION PRODUCTION PRODUCTION PRODUCTION PRODUCTION PRODUCTION PRODUCTION	78.128.216.72	CentOS 7.1.1503	8	32	500	CESNET		​sysops@clarin.eu	​cesnet-virtual@cesnet.cz	5.5	n/a	1.8.2	n/a
idp1-clarin.esc.rzg.mpg.de	idm.clarin.eu	​Unity IDM		PRODUCTION PRODUCTION 1	130.183.206.196	Scientific Linux 7.4	2	4	20	MPCDF		​sysops@clarin.eu	​clarin-support@rzg.mpg.de	5.8.0	td-agent 1.2.2	18.06.0	1.22.0
idp2-clarin.esc.rzg.mpg.de	aai2.clarin.eu beta-stats.clarin.eu	​Infra SPF MD pipelines ​Infra static webserver ​Piwik	8082, 44344, 44345 44343 44325	PRODUCTION 2 PRODUCTION 2 PRODUCTION (Backup)	130.183.206.33	Scientific Linux 7.4	2	4	20	MPCDF		​sysops@clarin.eu	​clarin-support@rzg.mpg.de	5.8.0	td-agent 1.2.2	18.06.0	1.22.0
centres2-clarin.esc.rzg.mpg.de	centres.clarin.eu	Centre Registry	44335 44325 (!) to be closed	PRODUCTION (Backup)	130.183.206.40	Scientific Linux 7.5	2	4	18	MPCDF		​sysops@clarin.eu	​clarin-support@rzg.mpg.de	5.8.1	td-agent 1.2.6	18.09.0	1.22.0
149-210-236-86.colo.transip.net	clarineu-vps2	​Reverse proxy	80, 443	PRODUCTION (Primary)	149.210.236.86 Priv. net.: 192.168.1.3	CentOS 7-5.1804.4.el7	2	4	150	TransIP	VPS X4 @AMS0 (Amsterdam)	​sysops@clarin.eu	​CP, 2nd best: ​support@transip.nl Status: ​TransNOC	5.8.0	td-agent 1.2.2	18.06.0	1.22.0
136-144-215-36.colo.transip.net	clarineu-vps6 (clarineu-vps5) original	​Reverse proxy	80, 443	PRODUCTION (Backup)	136.144.215.36 Priv. net.: 192.168.1.1	CentOS 7-5.1804.4.el7	2	4	150	TransIP	VPS X4 @RTM0 (Delft)	​sysops@clarin.eu	​CP, 2nd best: ​support@transip.nl Status: ​TransNOC	5.8.0	td-agent 1.2.2	18.06.0	1.22.0
37-97-184-230.colo.transip.net	clarineu-vps4 www.clarin.eu	​Main Website	44305	PRODUCTION (Primary)	37.97.184.230	CentOS 7-6.1810.2.el7	2	4	150	TransIP	VPS X4 @AMS0 (Amsterdam)	​sysops@clarin.eu	​CP, 2nd best: ​support@transip.nl Status: ​TransNOC	5.8.1	td-agent 1.4.2	18.09.8	1.24.1
136-144-221-254.colo.transip.net	clarineu-vps8 www.clarin.eu	​Main Website	44305	PRODUCTION (Backup)	136.144.221.254	CentOS 7-6.1810.2.el7	2	4	150	TransIP	VPS X4 @RTM0 (Delft)	​sysops@clarin.eu	​CP, 2nd best: ​support@transip.nl Status: ​TransNOC	5.8.1	td-agent 1.4.2	18.09.8	1.24.1
37.97.220.172.colo.transip.net	clarineu-vps5 (clarineu-vps) original	​Discovery service ​Infra SPF MD pipelines ​Infra static webserver ​Unity IDM (2.8.x)	8444 44344, 44345 44343	PRODUCTION 2 PRODUCTION 1 PRODUCTION 1 PRODUCTION	37.97.220.172	CentOS 7-5.1804.4.el7	2	4	150	TransIP	VPS X4 @AMS0 (Amsterdam)	​sysops@clarin.eu	​CP, 2nd best: ​support@transip.nl Status: ​TransNOC	5.8.0	td-agent 1.2.2	18.06.0	1.22.0
136-144-199-95.colo.transip.net	clarineu-vps7 (clarineu-vps6) original stats.clarin.eu switchboard.clarin.eu	​Piwik Centre Registry Swithboard	8082, 4425 44335 44399	PRODUCTION (Primary) PRODUCTION (Primary) PRODUCTION	136.144.199.95	CentOS 7-6.1810.2.el7	2	4	150	TransIP	VPS X4 @RTM0 (Delft)	​sysops@clarin.eu	​CP, 2nd best: ​support@transip.nl Status: ​TransNOC	5.8.1	td-agent 1.4.2	18.09.8	1.24.1
136-144-208-88.colo.transip.net	clarineu-backups (clarineu-vps7) original	BACKUPS		PRODUCTION	136.144.208.88	CentOS	1	1	2TB	TransIP	VPS X1 @RTM0 (Delft)	​sysops@clarin.eu	​CP, 2nd best: ​support@transip.nl Status: ​TransNOC	5.8.1	td-agent 1.3.3
CLARINEU-HAIP	High available IP address				136.144.144.150	-	-	-	-	TransIP		​sysops@clarin.eu	​CP, 2nd best: ​support@transip.nl Status: ​TransNOC	-	-	-	-
CLARINEU-HAIP-DEV	High available IP address				136.144.144.52		-	-	-	TransIP		​sysops@clarin.eu	​CP, 2nd best: ​support@transip.nl Status: ​TransNOC	-	-	-	-
clarin-vcr.ids-mannheim.de	VCR		443	PRODUCTION	193.196.8.26	CentOS	4	8	100	IDS		​sysops@clarin.eu	Oliver Schonefeld CLARIN Slack

1.1.2. Beta / Development (clarin-dev.eu)

Canonical FQDN	Aliases	Services	Ports	Service Type	IPv4-address	OS	(v)CPUs	Memory (GiB)	Storage (GiB)	Hoster	Type	Responsible	Support	Collectd	Fluentd	Docker	Compose
rs236235.rs.hosteurope.de	alpha-vlo.clarin.eu logs.clarin.eu	​Virtual Collection Registry ​Virtual Language Observatory docker-runner-hosteurope-1 docker-runner-hosteurope-2 discovery	5601	ALPHA ALPHA BUILD BUILD ALPHA BETA	91.250.80.240	CentOS 7.1.1503	4	18	750	HostEurope	Root Server M	​sysops@clarin.eu	​support@hosteurope.de	5.8.0	td-agent 1.2.6	18.06.0-ce	1.22.0
dev-idp-clarin.esc.rzg.mpg.de	dev-idp.clarin.eu dev-sp.clarin.eu	docker-runner-rzg-1 docker-runner-rzg-2 compreg (beta)		BUILD BUILD BETA	130.183.206.39	Scientific Linux 7.5	2	4	20	MPCDF		​sysops@clarin.eu	​clarin-support@rzg.mpg.de	5.8.1	td-agent 1.2.6	18.09.0	1.23.2
centres-clarin.esc.rzg.mpg.de	staging-centres.clarin.eu	Centre Registry	44335	STAGING BETA (Backup)	130.183.206.32	Scientific Linux 7.5	1	2	18	MPCDF		​sysops@clarin.eu	​clarin-support@rzg.mpg.de	5.8.1	td-agent 1.2.6	18.09.0	1.22.0
beta-vlo-clarin.esc.rzg.mpg.de	beta-vlo.clarin.eu	VLO		BETA	130.183.206.198	Scientific Linux 7.2	8	16	1000	MPCDF		​sysops@clarin.eu	​clarin-support@rzg.mpg.de	5.5	n/a	17.05.0-ce	1.17.0
alpha-vlo-clarin.esc.rzg.mpg.de		proxy-beta	80, 443	BETA	130.183.206.35	Scientific Linux 7.4	4	15	125	MPCDF		Twan Goosen	​clarin-support@rzg.mpg.de	5.4.2	n/a	n/a	n/a
37-97-154-156.colo.transip.net	transip-vps3 dev-www.clarin.eu	Main Website (dev) idm idm-delegation-pilot	4430, 4431 4432 2443, 1000	DEVELOPMENT DEVELOPMENT BETA	37.97.154.156 Priv. net. dev: 192.168.2.3	CentOS 7.4.1708	2	4	150	TransIP	VPS X4 @AMS0 (Amsterdam)	​sysops@clarin.eu	​CP, 2nd best: ​support@transip.nl Status: ​TransNOC	5.8.0	td-agent 1.2.6	18.06.0-ce	1.22.0
193.144.35.162	eosc-cesga-vps1				193.144.35.162	CentOs? 7.5.1804	12	24	800	CESGA		​sysops@clarin.eu	Ruben Diez ​rdiez@cesga.es
78.128.250.25	eosc-cesnet-vps1	Elastic search/Kibana			78.128.250.25		12	24		CESNET			https://dashboard.cloud.muni.cz/auth/login/?next=/
	eosc-recas-vps1				90.147.170.203		12	24		RECAS			https://cloud.recas.ba.infn.it

1.1.3. Unused

Canonical FQDN	Aliases	Services	Ports	Service Type	IPv4-address	OS	(v)CPUs	Memory (GiB)	Storage (GiB)	Hoster	Type	Responsible	Support	Collectd	Fluentd	Docker	Compose
lvps92-51-161-129.dedicated.hosteurope.de Contract termination requested: scheduled for 31-10-2018	vlo.clarin.eu				92.51.161.129	CentOS 7.1.1503	8	32	500	HostEurope	Virtual Server Linux Unlimited High I/O 8.0	​sysops@clarin.eu	​support@hosteurope.de	5.5	n/a	n/a	n/a

1.2. Externally managed, with central services

Canonical FQDN	Aliases	IPv4-address	OS	Docker	(v)CPUs	Memory (GiB)	Storage (GiB)	Hoster	Responsible
vz07-clarin-list?.im.hum.uu.nl	lists.clarin.eu newlists.clarin.eu	131.211.143.192	Debian 6	n/a	?	?	?	UU	​ictenmedia@uu.nl - Official (generic) ​r.vanvalkenburg@uu.nl - Direct to René van Valkenburg
fsd-cloud22.zam.kfa-juelich.de	monitoring.clarin.eu	134.94.199.42	Ubuntu 14.04.4 LTS	n/a				FZJ?	​CLARIN-support@fz-juelich.de
clarin.fz-juelich.de	-	134.94.199.71		n/a				FZJ?	​CLARIN-support@fz-juelich.de
clarin.ids-mannheim.de	clarin.ids-mannheim.de	193.196.8.17	CentOS 7.4	n/a	4	16	64	IDS?	Oliver Schonefeld
weblicht.sfs.uni-tuebingen.de	weblicht.sfs.uni-tuebingen.de	130.183.206.38	Ubuntu 16.04	1.12.3	4	64	500	UTU?	​emanuel.dima@uni-tuebingen.de
spraakbanken.gu.se/ws/fcs/2.0/aggregator/	contentsearch.clarin.eu	130.241.42.13						Språkbanken	​leif-joran.olsson@svenska.gu.seadded aa

1.3. Decommissioned

Canonical FQDN	Aliases	IPv4-address	OS	Docker	Hoster	Responsible
clarinvm.ics.muni.cz		147.251.9.199	CentOS 7.1.1503	??	CESNET	​sysops@clarin.eu
ems04.mpi.nl		192.87.79.165	Ubuntu 12.04.5 LTS	n/a	MPI-PL	​sysops@clarin.eu
idp-clarin.esc.rzg.mpg.de	-	130.183.206.37	SLES 11.3	n/a	MPCDF	​sysops@clarin.eu
stoor146.meta.zcu.cz	-	147.228.242.146	CentOS 7.1.1503	1.5.0	CESNET	​sysops@clarin.eu
catalog-clarin?.esc.rzg.mpg.de		192.87.79.171	SLES 11.2	n/a	MPI-PL	​sysops@clarin.eu
im-linux-clarin-eu?.im.hum.uu.nl	www.clarin.eu	131.211.143.212	Debian 8	n/a	UU	​web team Sander Maijers ​ictenmedia@uu.nl
im-linux-dev-clarin-eu.hum.uu.nl	-	131.211.143.192	Debian 8	n/a	UU	​web team Sander Maijers ​ictenmedia@uu.nl
vz07-clarin-eu?.im.hum.uu.nl	-	131.211.143.186	Debian 8	n/a	UU	​web team Sander Maijers ​ictenmedia@uu.nl
lvps83-169-5-155.dedicated.hosteurope.de		83.169.5.155	CentOS	n/a	HostEurope	Decommissioned per 31.05.2018

2. DNS entries and TLS certificates

Hosted by ​TransIP

admins: Dieter Van Uytvanck, Andre Moreira, Willem Elbers

3. Getting access

Shell access to the CLARIN hosts is only possible via key-based SSH.

Contact ​sysops@clarin.eu to request access to a host. Make sure to include your public SSH key.

Instructions and guidelines on how to create your OpenSSH key pair can be found here.

3.1. Security

4. Default VM setup

These instructions describe how we install/provision/configure each host by default.

4.1. Connections

service	port	type	direction
ssh	22	tcp	incoming
ssh	22	tcp	outgoing to gitlab.com
collectd	25826	tcp	outgoing
fluentd	24224	tcp	outgoing

4.2. Centos / Scientific Linux

Some notes on administering Centos / Scientific linux hosts.

4.3. SLES 11

We are in the process of migration our SLES 11 machines to CentOS/Scientific Linux. We collect some notes on administering SLES hosts.

4.4. Ubuntu

We are in the process of migration our Ubuntu machines to CentOS/Scientific Linux.

5. Deploying and running services

Repositories:

• Deploy script: ​https://gitlab.com/CLARIN-ERIC/deploy-script

• Control script: ​https://gitlab.com/CLARIN-ERIC/control-script

5.1. Deploy a service

In the deploy users home directory (/home/deploy):

sh deploy.sh --name service-name --git git-repo-name --tag 1.0.0

Updates are performed by running the same command with a different tag and then using the control.sh script to restart the service.

5.2. Initialize a service

In the deploy users home directory (/home/deploy):

sh control.sh service-name init

Customize <service-name>/.env as needed.

5.3. Start the service

In the deploy users home directory (/home/deploy):

sh control.sh service-name start

Other commands available: stop, restart, backup, restore, ...

6. Infrastructure and service status information

A manually curated service status overview including planned maintenance is kept at ​clarin.eu/status.

Service availability statistics (sourced by ​StatusCake) are available at ​status.clarin.eu. Incidents are also posted automatically to the private ​sysalert channel on Slack.

Maintainers of services, in particular core services and A-services are requested to ​submit expected downtime information timely. For more information, see Service status guidelines.

7. Documents

• ​CLARIN Infrastructure Overview
• ​Docker Workflow and best practices
• Proposal: ​High Availability for the CLARIN infrastructure
• ​Sysops - infrastructure management

8. Services

• CE Archive

9. Updates

Bi-weekly on Tuesdays.

Canonical FQDN	OS	Updates			Comments
		Kernel	Packages	Docker
clarinvm.cesnet.cz	CentOS 7.1.1503	12-01-2018	12-01-2018	26-03-2019	Kernel: 3.10.0-229.el7.x86_64
dev-idp-clarin.esc.rzg.mpg.de	Scientific Linux 7.4	12-01-2018	12-01-2018	12-01-2018	Kernel: 3.10.0-693.11.6.el7.x86_64
centres2-clarin.esc.rzg.mpg.de	Scientific Linux 7.4	20-08-2018	20-08-2018	20-08-2018	Kernel: 3.10.0-862.6.3.el7.x86_64
beta-vlo-clarin.esc.rzg.mpg.de	Scientific Linux 7.2	12-01-2018	12-01-2018	10-11-2017	Kernel: 3.10.0-693.11.6.el7.x86_64
lvps92-51-161-129.dedicated.hosteurope.de	CentOS 7.1.1503	12-01-2018	12-01-2018	N/A	Kernel: 3.10.0-042stab127.2 (​OpenVZ)
rs236235.rs.hosteurope.de	CentOS 7.4.1708	12-01-2018	12-01-2018	12-01-2018	Kernel: 3.10.0-693.11.6.el7.x86_64
idp2-clarin.esc.rzg.mpg.de	Scientific Linux 7.2	16-08-2018	21-08-2018	16-08-2018	Kernel: 3.10.0-862.3.2.el7.x86_64
idp1-clarin.esc.rzg.mpg.de	Scientific Linux 7.2	12-01-2018	12-01-2018	PENDING	Kernel: 3.10.0-693.11.6.el7.x86_64
149-210-236-86.colo.transip.net	CentOS 7-5.1804.4.el7	16-08-2018	21-08-2018	16-08-2018	Kernel: 3.10.0-862.11.6.el7.x86_64
37-97-154-156.colo.transip.net	CentOS 7-5.1804.4.el7	21-08-2018	21-08-2018	21-08-2018	Kernel: 3.10.0-862.11.6.el7.x86_64
37-97-184-230.colo.transip.net	CentOS 7	12-01-2018	12-01-2018	14-11-2017	kernel: 3.10.0-693.11.6.el7.x86_64
37.97.220.172.colo.transip.net	CentOS 7-5.1804.4.el7	16-08-2018	21-08-2018	16-08-2018	Kernel: 3.10.0-862.11.6.el7.x86_64
136-144-215-36.colo.transip.net	CentOS 7-5.1804.4.el7	15-08-2018	21-08-2018	15-08-2018	kernel: 3.10.0-862.11.6.el7.x86_64
centres-clarin.esc.rzg.mpg.de	SLES 11.3	12-01-2018	12-01-2018	N/A	Kernel: 3.0.101-108.21.1.x86_64
alpha-vlo-clarin.esc.rzg.mpg.de	Scientific Linux 7.3	12-01-2018	12-01-2018	N/A	Kernel: 3.10.0-693.11.6.el7.x86_64

10. Known issues

10.1. Docker

10.1.1. dial tcp: lookup index.docker.io: no such host

10.1.1.1. Error

dial tcp: lookup index.docker.io: no such host

10.1.1.2. Symptoms

While using Docker a user is unable to perform tasks such as pull new image or search for new images while the following error message appears:

# docker pull debian:8
Pulling repository debian
FATA[0053] Get https://index.docker.io/v1/repositories/library/debian/images: dial tcp: lookup index.docker.io: no such host

10.1.1.3. Solution

No good solution available at this time.

10.1.1.4. References

• ​https://linuxconfig.org/docker-dial-tcp-lookup-index-docker-io-no-such-host-fix
• ​https://robinwinslow.uk/2016/06/23/fix-docker-networking-dns/
• ​https://stackoverflow.com/questions/29266560/docker-container-can-reach-dns-but-not-resolve-hosts
• ​https://github.com/moby/moby/issues/13381

11. GitLab

11.1. Managing a git repository on a server with a deploy key

1. Enable deploy key in gitlab reository

• Goto Settings - Repository
• Expand "Deploy Keys"
• Enable the CLARIN keys (make sure to not use the public ones!)

2. Configure a ssh connection for gitlab on the service

• Add the private part of the deploy key to /home/deploy/.ssh/id_rsa_gitlab_deploy
• Edit /home/deploy/.ssh/config
• Add:

  #Deploys
  Host gitlab.com
      User git
      HostName gitlab.com
      IdentityFile ~/.ssh/id_rsa_gitlab_deploy
  

3. Use the SSH location to clone the repository

• Example

  git clone git@gitlab.com:CLARIN-ERIC/compose_transip_vps5.git