wiki:SystemAdministration

Version 483 (modified by Willem Elbers, 3 years ago) (diff)

--

CVEs

Tickets

Ticket Priority Summary Owner Created Modified
#1086 minor Problem with admin messages from lists.clarin.eu André Moreira 4 years ago 4 years ago
#1054 major Bad URL encoding in catalog.clarin.eu/vlo -> vlo.clarin.eu redirects André Moreira 7 years ago 7 years ago
#1010 minor Piwik overlay André Moreira 7 years ago 7 years ago
#1000 major Review server timezone André Moreira 8 years ago 8 years ago
#996 major catalog.clarin.eu SP: aa-statistics error message on login André Moreira 8 years ago 8 years ago
#981 major Add DKIM, DMARC DNS RRs for clarin.eu Sander Maijers 8 years ago 8 years ago
#887 major Configure local firewall for idp1-clarin and idp2-clarin Sander Maijers 9 years ago 8 years ago
#881 minor Migrate lists.clarin.eu Sander Maijers 9 years ago 8 years ago
#891 critical dmeventd for LVM hogs CPU on CentOS 7 Willem Elbers 9 years ago 8 years ago
#955 major Add Reverse DNS records for TransIP VMs Sander Maijers 8 years ago 8 years ago
#902 major Grant read access to httpd logs Sander Maijers 9 years ago 9 years ago
#885 major Django security update to 1.9.3 Sander Maijers 9 years ago 9 years ago
#884 critical Security-critical instructions to data centre support should have integrity Sander Maijers 9 years ago 9 years ago
#883 major Devise automatic tty I/O auditing setup & policy Sander Maijers 9 years ago 9 years ago
#865 major Make host pages adhere to host template in Trac Sander Maijers 9 years ago 9 years ago
#864 major Automated collection of running Docker containers Sander Maijers 9 years ago 9 years ago
#863 minor Automate collection of host info Sander Maijers 9 years ago 9 years ago

1. Hosts

1.1. Internally managed

1.1.1. Production (clarin.eu)

Canonical FQDN Aliases Services Ports Service Type IPv4-address OS (v)CPUs Memory
(GiB)
Storage
(GiB)
Hoster Type Responsible Support Collectd Fluentd Docker Compose
clarinvm.cesnet.cz catalog.clarin.eu
docker.clarin.eu
nexus.clarin.eu
office.clarin.eu
Netkernel
CLARIN EU trac
CLARIN NL trac
SVN
Metrics
Component Registry
Discovery service
PRODUCTION
PRODUCTION
PRODUCTION
PRODUCTION
PRODUCTION
PRODUCTION
PRODUCTION
78.128.216.72 CentOS
7.1.1503
8 32 500 CESNET sysops@clarin.eu cesnet-virtual@cesnet.cz 5.5 n/a 1.8.2 n/a
idp2-clarin.esc.rzg.mpg.de aai2.clarin.eu
beta-stats.clarin.eu
Infra SPF MD pipelines
Infra static webserver
Piwik
8082, 44344, 44345
44343
44325
PRODUCTION 2
PRODUCTION 2
PRODUCTION (Backup)
130.183.206.33 Scientific Linux
7.4
2 4 20 MPCDF sysops@clarin.eu clarin-support@rzg.mpg.de 5.8.0 td-agent
1.2.2
18.06.0 1.22.0
centres2-clarin.esc.rzg.mpg.de centres.clarin.eu Centre Registry 44335
44325 (!) to be closed
PRODUCTION (Backup) 130.183.206.40 Scientific Linux
7.5
2 4 18 MPCDF sysops@clarin.eu clarin-support@rzg.mpg.de 5.8.1 td-agent
1.2.6
18.09.0 1.22.0
149-210-236-86.colo.transip.net clarineu-vps2 Reverse proxy 80, 443 PRODUCTION (Primary) 149.210.236.86
Priv. net.: 192.168.1.3
CentOS
7-5.1804.4.el7
2 4 150 TransIP VPS X4
@AMS0 (Amsterdam)
sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.0 td-agent
1.2.2
18.06.0 1.22.0
136-144-215-36.colo.transip.net clarineu-vps6
(clarineu-vps5) original
Reverse proxy 80, 443 PRODUCTION (Backup) 136.144.215.36
Priv. net.: 192.168.1.1
CentOS
7-5.1804.4.el7
2 4 150 TransIP VPS X4
@RTM0 (Delft)
sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.0 td-agent
1.2.2
18.06.0 1.22.0
149-210-250-181.colo.transip.net clarineu-vps9
www.clarin.eu
Main Website 44305 PRODUCTION (Primary) 149.210.250.181 CentOS
7-9.2009.1.el7
4 8 300 TransIP VPS X8
@AMS0 (Amsterdam)
sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.1 td-agent
1.11.5
20.10.7 1.29.2
136-144-221-254.colo.transip.net clarineu-vps8
www.clarin.eu
Main Website 44305 PRODUCTION (Backup) 136.144.221.254 CentOS
7-9.2009.1.el7
4 8 300 TransIP VPS X8
@RTM0 (Delft)
sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.1 td-agent
1.11.5
20.10.7 1.29.2
37.97.220.172.colo.transip.net clarineu-vps5
(clarineu-vps) original
Discovery service
Infra SPF MD pipelines
Infra static webserver
Unity IDM (2.8.x)
8444
44344, 44345
44343
PRODUCTION 2
PRODUCTION 1
PRODUCTION 1
PRODUCTION
37.97.220.172 CentOS
7-5.1804.4.el7
2 4 150 TransIP VPS X4
@AMS0 (Amsterdam)
sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.0 td-agent
1.2.2
18.06.0 1.22.0
136-144-199-95.colo.transip.net clarineu-vps7
(clarineu-vps6) original
stats.clarin.eu
switchboard.clarin.eu
Piwik
Centre Registry
Switchboard
8082, 4425
44335
44399
PRODUCTION (Primary)
PRODUCTION (Primary)
PRODUCTION
136.144.199.95 CentOS
7-6.1810.2.el7
2 4 150 TransIP VPS X4
@RTM0 (Delft)
sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.1 td-agent
1.4.2
18.09.8 1.24.1
136-144-208-88.colo.transip.net clarineu-backups
(clarineu-vps7) original
BACKUPS PRODUCTION 136.144.208.88 CentOS 1 1 2TB TransIP VPS X1

@RTM0 (Delft)
sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.1 td-agent 1.3.3
CLARINEU-HAIP High available IP address 136.144.144.150 - - - - TransIP sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
- - - -
CLARINEU-HAIP-DEV High available IP address 136.144.144.52 - - - TransIP sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
- - - -
clarin-vcr.ids-mannheim.de collections.clarin.eu VCR 443 PRODUCTION 193.196.8.26 CentOS 4 8 100 IDS sysops@clarin.eu Oliver Schonefeld
CLARIN Slack
hetzner-vps3 vlo.clarin.eu VLO PRODUCTION 157.90.1.116 CentOS 7 24 128 1800GB Hetzner AX61-NVMe sysops@clarin.eu https://robot.your-server.de/server

1.1.2. Beta / Development (clarin-dev.eu)

Canonical FQDN Aliases Services Ports Service Type IPv4-address OS (v)CPUs Memory
(GiB)
Storage
(GiB)
Hoster Type Responsible Support Collectd Fluentd Docker Compose
dev-idp-clarin.esc.rzg.mpg.de dev-idp.clarin.eu
dev-sp.clarin.eu
docker-runner-rzg-1
docker-runner-rzg-2
compreg (beta)
BUILD
BUILD
BETA
130.183.206.39 Scientific Linux
7.5
2 4 20 MPCDF sysops@clarin.eu clarin-support@rzg.mpg.de 5.8.1 td-agent
1.2.6
18.09.0 1.23.2
centres-clarin.esc.rzg.mpg.de staging-centres.clarin.eu Centre Registry 44335 STAGING
BETA (Backup)
130.183.206.32 Scientific Linux
7.5
1 2 18 MPCDF sysops@clarin.eu clarin-support@rzg.mpg.de 5.8.1 td-agent
1.2.6
18.09.0 1.22.0
alpha-vlo-clarin.esc.rzg.mpg.de proxy-beta 80, 443 BETA 130.183.206.35 Scientific Linux
7.4
4 15 125 MPCDF Twan Goosen clarin-support@rzg.mpg.de 5.4.2 n/a n/a n/a
37-97-154-156.colo.transip.net clarineu-vps3
dev-www.clarin.eu
Main Website (dev primary)
idm
idm-delegation-pilot
4430, 4431
4432
2443, 1000
DEVELOPMENT
DEVELOPMENT
BETA
37.97.154.156
Priv. net. dev: 192.168.2.3
CentOS
7.4.1708
2 4 150 TransIP VPS X4
@AMS0 (Amsterdam)
sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.0 td-agent
1.2.6
18.06.0-ce 1.22.0
37-97-184-230.colo.transip.net clarineu-vps4
legacy-d8b3-www.clarin-dev.eu
Main Website (legacy reference instance: Drupal 8 Bootstrap 3) 44305 DEVELOPMENT 37.97.184.230 CentOS
7-9.2009.1.el7
2 6 150 TransIP VPS X4
@AMS0 (Amsterdam)
sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.1 td-agent
1.11.5
20.10.7 1.29.2
193.144.46.251 eosc-cesga-vps1
legacy-d7b3-www.clarin-dev.eu
dev-www.clarin.eu
beta-switchboard.clarin.eu
nextcloud.clarin-dev.eu
Main Website (legacy reference instance: Drupal 7 Bootstrap 3)
Main Website (dev backup)
Switchboard beta
Nextcloud instance with direct SSH access on port 22422 (B2Drop Switchoboard plugin dev)

44305
44315
44399
44355
22422
193.144.46.251 CentOs? 7.5.1804 12 24 800 CESGA sysops@clarin.eu https://fedcloud-osservices.egi.cesga.es/
(Login via EGI sso)

Ruben Diez: rdiez@cesga.es
5.8.1 td-agent
1.2.2
18.09.0 1.24.1
78.128.250.25 eosc-cesnet-vps1
logs.clarin.eu
Elastic search/Kibana
| Digital Object Gate

44332
78.128.250.25 12 64 1000 CESNET sysops@clarin.eu https://dashboard.cloud.muni.cz/auth/login/?next=/

(Select EGI checkin, then login via EGI sso)
hetzner-vps1 beta-vlo.clarin.eu
beta-collections.clarin.eu
-
fcs.clarin-dev.eu
-
-
VLO beta and curation beta
VCR Beta
Discovery service Alpha
FCS
Netkernel
Gitlab Runners
BETA
BETA
DEVELOPMENT
BETA
?
TOOL
168.119.38.169 CentOS 7 12 128 934GB Hetzner EX52-NVMe sysops@clarin.eu https://robot.your-server.de/server - - - -
hetzner-vps2 alpha-vlo.clarin.eu
alpha-collections.clarin.eu
VLO alpha and curation alpha
VCR Alpha
DEVELOPMENT
DEVELOPMENT
136.243.133.121 CentOS 7 12 128 934GB Hetzner EX52-NVMe sysops@clarin.eu https://robot.your-server.de/server

1.1.3. Unused

Canonical FQDN Aliases Services Ports Service Type IPv4-address OS (v)CPUs Memory
(GiB)
Storage
(GiB)
Hoster Type Responsible Support Collectd Fluentd Docker Compose
beta-vlo-clarin.esc.rzg.mpg.de beta-vlo.clarin.eu 130.183.206.198 Scientific Linux
7.2
8 16 1000 MPCDF sysops@clarin.eu clarin-support@rzg.mpg.de 5.5 n/a 17.05.0-ce 1.17.0
idp1-clarin.esc.rzg.mpg.de idm.clarin.eu
130.183.206.196 Scientific Linux
7.4
2 4 20 MPCDF sysops@clarin.eu clarin-support@rzg.mpg.de 5.8.0 td-agent
1.2.2
18.06.0 1.22.0

1.2. Externally managed, with central services

Canonical FQDN Aliases IPv4-address OS Docker (v)CPUs Memory
(GiB)
Storage
(GiB)
Hoster Responsible
vz07-clarin-list?.im.hum.uu.nl lists.clarin.eu
newlists.clarin.eu
131.211.143.192 Debian
6
n/a ? ? ? UU ictenmedia@uu.nl - Official (generic)
r.vanvalkenburg@uu.nl - Direct to René van Valkenburg
fsd-cloud22.zam.kfa-juelich.de monitoring.clarin.eu 134.94.199.42 Ubuntu 14.04.4 LTS n/a FZJ? CLARIN-support@fz-juelich.de
clarin.fz-juelich.de - 134.94.199.71 n/a FZJ? CLARIN-support@fz-juelich.de
clarin.ids-mannheim.de clarin.ids-mannheim.de 193.196.8.17 CentOS
7.4
n/a 4 16 64 IDS? Oliver Schonefeld
weblicht.sfs.uni-tuebingen.de weblicht.sfs.uni-tuebingen.de 130.183.206.38 Ubuntu
16.04
1.12.3 4 64 500 UTU? emanuel.dima@uni-tuebingen.de
spraakbanken.gu.se/ws/fcs/2.0/aggregator/ contentsearch.clarin.eu 130.241.42.13 Språkbanken leif-joran.olsson@svenska.gu.seadded aa

1.3. Decommissioned

Canonical FQDN Aliases IPv4-address OS Docker Hoster Decommisioning notes
clarinvm.ics.muni.cz 147.251.9.199 CentOS
7.1.1503
?? CESNET sysops@clarin.eu
ems04.mpi.nl 192.87.79.165 Ubuntu
12.04.5 LTS
n/a MPI-PL sysops@clarin.eu
idp-clarin.esc.rzg.mpg.de - 130.183.206.37 SLES
11.3
n/a MPCDF sysops@clarin.eu
stoor146.meta.zcu.cz - 147.228.242.146 CentOS
7.1.1503
1.5.0 CESNET sysops@clarin.eu
catalog-clarin?.esc.rzg.mpg.de 192.87.79.171 SLES
11.2
n/a MPI-PL sysops@clarin.eu
im-linux-clarin-eu?.im.hum.uu.nl www.clarin.eu 131.211.143.212 Debian
8
n/a UU web team
Sander Maijers
ictenmedia@uu.nl
im-linux-dev-clarin-eu.hum.uu.nl - 131.211.143.192 Debian
8
n/a UU web team
Sander Maijers
ictenmedia@uu.nl
vz07-clarin-eu?.im.hum.uu.nl - 131.211.143.186 Debian
8
n/a UU web team
Sander Maijers
ictenmedia@uu.nl
lvps83-169-5-155.dedicated.hosteurope.de 83.169.5.155 CentOS n/a HostEurope Decommissioned per 31.05.2018
lvps92-51-161-129.dedicated.hosteurope.de vlo.clarin.eu 92.51.161.129 CentOS
7.1.1503
n/a HostEurope Decomissioned per 31-10-2018
cloud-90-147-170-203.cloud.ba.infn.it eosc-recas-vps1 90.147.170.203 CentOS RECAS Fatal Crash April 2021, never recommissioned
rs238144.rs.hosteurope.de vlo.clarin.eu 91.250.82.71 CentOS HostEurope Decomissioned July 2021
rs236235.rs.hosteurope.de alpha-vlo.clarin.eu
Virtual Collection Registry
Virtual Language Observatory
docker-runner-hosteurope-1
docker-runner-hosteurope-2
discovery
FCS beta
91.250.80.240 CentOS HostEurope Decomissioned July 2021
249811.rs.hosteurope.de Link Checker 5.35.250.44 CentOS HostEurope Decomissioned July 2021

2. DNS entries and TLS certificates

Hosted by TransIP

admins: Dieter Van Uytvanck, Andre Moreira, Willem Elbers

3. Getting access

Shell access to the CLARIN hosts is only possible via key-based SSH.

Contact sysops@clarin.eu to request access to a host. Make sure to include your public SSH key.

Instructions and guidelines on how to create your OpenSSH key pair can be found here.

3.1. Security

4. Default VM setup

These instructions describe how we install/provision/configure each host by default.

4.1. Connections

service port type direction
ssh 22 tcp incoming
ssh 22 tcp outgoing to gitlab.com
collectd 25826 tcp outgoing
fluentd 24224 tcp outgoing

4.2. Centos / Scientific Linux

Some notes on administering Centos / Scientific linux hosts.

4.3. SLES 11

We are in the process of migration our SLES 11 machines to CentOS/Scientific Linux. We collect some notes on administering SLES hosts.

4.4. Ubuntu

We are in the process of migration our Ubuntu machines to CentOS/Scientific Linux.

5. Deploying and running services

Repositories:

5.1. Deploy a service

In the deploy users home directory (/home/deploy):

sh deploy.sh --name service-name --git git-repo-name --tag 1.0.0

Updates are performed by running the same command with a different tag and then using the control.sh script to restart the service.

5.2. Initialize a service

In the deploy users home directory (/home/deploy):

sh control.sh service-name init

Customize <service-name>/.env as needed.

5.3. Start the service

In the deploy users home directory (/home/deploy):

sh control.sh service-name start

Other commands available: stop, restart, backup, restore, ...

6. Infrastructure and service status information

A manually curated service status overview including planned maintenance is kept at clarin.eu/status.

Service availability statistics (sourced by StatusCake) are available at status.clarin.eu. Incidents are also posted automatically to the private sysalert channel on Slack.

Maintainers of services, in particular core services and A-services are requested to submit expected downtime information timely. For more information, see Service status guidelines.

7. Documents

8. Services

9. Updates

(to be updated)

10. Metrics

10.1. Latency Checks

10.1.1. Latency dashboard

The latency check dashboard is available here: https://metrics.clarin.eu/d/000000019/latency

10.1.2. Managing latency checks

On transip-vps6 there is a collectd configuration in place that pings some of our VPSes. Ideally one for each provider / geographical area.

To edit the latency checks edit /etc/collectd.d/ping.conf:

LoadPlugin ping
<Plugin ping>
        Host "rs238144.rs.hosteurope.de"
 	Host "clarinvm.cesnet.cz"
	Host "idp1-clarin.esc.rzg.mpg.de"
	Host "clarin-vcr.ids-mannheim.de"
	Host "hetzner-vps1"
</Plugin>

If a host is not properly reachable via a hostname, add a IP to hostname mapping in /etc/hosts. E.g.:

168.119.38.169	hetzner-vps1

11. Known issues

11.1. Docker

11.1.1. dial tcp: lookup index.docker.io: no such host

11.1.1.1. Error

dial tcp: lookup index.docker.io: no such host

11.1.1.2. Symptoms

While using Docker a user is unable to perform tasks such as pull new image or search for new images while the following error message appears:

# docker pull debian:8
Pulling repository debian
FATA[0053] Get https://index.docker.io/v1/repositories/library/debian/images: dial tcp: lookup index.docker.io: no such host

11.1.1.3. Solution

No good solution available at this time.

11.1.1.4. References

12. GitLab

12.1. Managing a git repository on a server with a deploy key

  1. Enable deploy key in gitlab reository
  • Goto Settings - Repository
  • Expand "Deploy Keys"
  • Enable the CLARIN keys (make sure to not use the public ones!)
  1. Configure a ssh connection for gitlab on the service
  • Add the private part of the deploy key to /home/deploy/.ssh/id_rsa_gitlab_deploy
  • Edit /home/deploy/.ssh/config
  • Add:
    #Deploys
    Host gitlab.com
        User git
        HostName gitlab.com
        IdentityFile ~/.ssh/id_rsa_gitlab_deploy
    
  1. Use the SSH location to clone the repository
  • Example
    git clone git@gitlab.com:CLARIN-ERIC/compose_transip_vps5.git