!!!!PLEASE DO NOT EDIT!!!!
THIS PAGE HAS BEEN MIGRATED TO THE Dev Google Drive
!!!!PLEASE DO NOT EDIT!!!!
Contents
CVEs
- Log4Shell: CVE-2021-44228
Tickets
1. Hosts
1.1. Internally managed
1.1.1. Production (clarin.eu)
Canonical FQDN | Aliases | Services | Ports | Service Type | IPv4-address | OS | (v)CPUs | Memory (GiB) | Storage (GiB) | Hoster | Type | Responsible | Support | Collectd | Fluentd | Docker | Compose |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
clarinvm.cesnet.cz | catalog.clarin.eu docker.clarin.eu nexus.clarin.eu office.clarin.eu | Netkernel CLARIN EU trac CLARIN NL trac SVN Metrics Component Registry Discovery service | PRODUCTION PRODUCTION PRODUCTION PRODUCTION PRODUCTION PRODUCTION PRODUCTION | 78.128.216.72 | CentOS 7.1.1503 | 8 | 32 | 500 | CESNET | sysops@clarin.eu | cesnet-virtual@cesnet.cz | 5.5 | n/a | 1.8.2 | n/a | ||
149-210-236-86.colo.transip.net | clarineu-vps2 | Reverse proxy | 80, 443 | PRODUCTION (Primary) | 149.210.236.86 Priv. net.: 192.168.1.3 | CentOS 7-5.1804.4.el7 | 2 | 4 | 150 | TransIP | VPS X4 @AMS0 (Amsterdam) | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | 5.8.0 | td-agent 1.2.2 | 18.06.0 | 1.22.0 |
136-144-215-36.colo.transip.net | clarineu-vps6 (clarineu-vps5) original | Reverse proxy | 80, 443 | PRODUCTION (Backup) | 136.144.215.36 Priv. net.: 192.168.1.1 | CentOS 7-5.1804.4.el7 | 2 | 4 | 150 | TransIP | VPS X4 @RTM0 (Delft) | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | 5.8.0 | td-agent 1.2.2 | 18.06.0 | 1.22.0 |
149-210-250-181.colo.transip.net | clarineu-vps9 www.clarin.eu | Main Website | 44305 | PRODUCTION (Primary) | 149.210.250.181 | CentOS 7-9.2009.1.el7 | 4 | 8 | 300 | TransIP | VPS X8 @AMS0 (Amsterdam) | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | 5.8.1 | td-agent 1.11.5 | 20.10.7 | 1.29.2 |
136-144-221-254.colo.transip.net | clarineu-vps8 www.clarin.eu | Main Website | 44305 | PRODUCTION (Backup) | 136.144.221.254 | CentOS 7-9.2009.1.el7 | 4 | 8 | 300 | TransIP | VPS X8 @RTM0 (Delft) | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | 5.8.1 | td-agent 1.11.5 | 20.10.7 | 1.29.2 |
37.97.220.172.colo.transip.net | clarineu-vps5 (clarineu-vps) original | Discovery service Infra SPF MD pipelines Infra static webserver Unity IDM (2.8.x) | 8444 44344, 44345 44343 | PRODUCTION 2 PRODUCTION 1 PRODUCTION 1 PRODUCTION | 37.97.220.172 | CentOS 7-5.1804.4.el7 | 4 | 8 | 150 (300 but needs growfs) | TransIP | VPS X8 @AMS0 (Amsterdam) | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | 5.8.0 | td-agent 1.2.2 | 18.06.0 | 1.22.0 |
136-144-199-95.colo.transip.net | clarineu-vps7 (clarineu-vps6) original stats.clarin.eu switchboard.clarin.eu | Matomo Centre Registry Switchboard | 8082, 4425 44335 44399 | PRODUCTION (Primary) PRODUCTION (Primary) PRODUCTION | 136.144.199.95 | CentOS 7-9.2009.1.el7 | 4 | 8 | 150 (300 available but FS not grown) | TransIP | VPS X8 @RTM0 (Delft) | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | 5.8.1 | td-agent 1.11.15 | 23.0.4 | 1.24.1 |
136-144-208-88.colo.transip.net | clarineu-backups (clarineu-vps7) original | BACKUPS | PRODUCTION | 136.144.208.88 | CentOS | 1 | 1 | 2TB | TransIP | VPS X1 @RTM0 (Delft) | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | 5.8.1 | td-agent 1.3.3 | |||
CLARINEU-HAIP | High available IP address | 136.144.144.150 | - | - | - | - | TransIP | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | - | - | - | - | ||||
CLARINEU-HAIP-DEV | High available IP address | 136.144.144.52 | - | - | - | TransIP | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | - | - | - | - | |||||
clarin-vcr.ids-mannheim.de | collections.clarin.eu | VCR | 443 | PRODUCTION | 193.196.8.26 | CentOS | 4 | 8 | 100 | IDS | sysops@clarin.eu | Oliver Schonefeld CLARIN Slack | |||||
hetzner-vps3 | vlo.clarin.eu | VLO Curation module/linkchecker | PRODUCTION | 157.90.1.116 | CentOS 7 | 24 | 128 | 1800GB | Hetzner | AX61-NVMe | sysops@clarin.eu | https://robot.your-server.de/server | |||||
hetzner-vps4 | Infra SPF MD pipelines Infra static webserver Matomo Centre Registry | 8082, 44344, 44345 44343 44325 44335 | PRODUCTION 2 PRODUCTION 2 PRODUCTION (Backup) PRODUCTION (Backup) | 95.216.225.96 | CentOS 7 | 8 | 64 | 512GB | Hetzner | EX42-NVMe (Finland) | sysops@clarin.eu | https://robot.your-server.de/server | 5.8.1 | td-agent 1.11.5 | 20.10.12 | 1.29.2 | |
transip-vps10 | Unity IDM (3.x.x) | PRODUCTION | 37.97.132.227 | AlmaLinux? 8 | 4 | 8 | 300GB | TransIP | VPS X8 @AMS0 (Amsterdam) | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | 5.9.0 | td-agent 1.11.5 | 23.0.0 | 2.2.3 |
1.1.2. Beta / Development (clarin-dev.eu)
Canonical FQDN | Aliases | Services | Ports | Service Type | IPv4-address | OS | (v)CPUs | Memory (GiB) | Storage (GiB) | Hoster | Type | Responsible | Support | Collectd | Fluentd | Docker | Compose |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
37-97-154-156.colo.transip.net | clarineu-vps3 dev-www.clarin.eu | Main Website (dev primary) idm idm-delegation-pilot | 4430, 4431 4432 2443, 1000 | DEVELOPMENT DEVELOPMENT BETA | 37.97.154.156 Priv. net. dev: 192.168.2.3 | CentOS 7.9.2009 | 2 | 4 | 150 | TransIP | VPS X4 @AMS0 (Amsterdam) | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | 5.8.0 | td-agent 1.2.6 | 20.10.12 | 2.2.3 |
37-97-184-230.colo.transip.net | clarineu-vps4 legacy-d8b3-www.clarin-dev.eu centres-staging.clarin.eu alpha-d4-centres.clarin.eu | Main Website (legacy reference instance: Drupal 8 Bootstrap 3) idm Centre Registry GitLab runners | 44305 44335 44334 | DEVELOPMENT DEVELOPMENT DEVELOPMENT DEVELOPMENT TOOL | 37.97.184.230 Priv. net. dev: 192.168.2.4 | AlmaLinux 8.5-4.el8 | 4 | 8 | 300 | TransIP | VPS X8 @AMS0 (Amsterdam) | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | 5.8.1 | td-agent 1.11.5 | 20.10.7 | 1.29.2 |
193.144.46.251 | eosc-cesga-vps1 legacy-d7b3-www.clarin-dev.eu dev-www.clarin.eu beta-switchboard.clarin.eu nextcloud.clarin-dev.eu | Main Website (legacy reference instance: Drupal 7 Bootstrap 3) Main Website (dev backup) Switchboard beta Nextcloud instance with direct SSH access on port 22422 (B2Drop Switchoboard plugin dev) (OPENSTACK BASED - Non Standard MTU) | 44305 44315 44399 44355 22422 | 193.144.46.251 | CentOS 7.5.1804 | 12 | 24 | 800 | CESGA | sysops@clarin.eu | https://fedcloud-osservices.egi.cesga.es/ (Login via EGI sso) Ruben Diez: rdiez@cesga.es | 5.8.1 | td-agent 1.2.2 | 18.09.0 | 1.24.1 | ||
78.128.250.25 | eosc-cesnet-vps1 logs.clarin.eu | Elastic search/Kibana | Digital Object Gate (OPENSTACK BASED - Non Standard MTU) | 44332 | 78.128.250.25 | 12 | 64 | 1000 | CESNET | sysops@clarin.eu | https://dashboard.cloud.muni.cz/auth/login/?next=/ (Select EGI checkin, then login via EGI sso) | |||||||
hetzner-vps1 | beta-vlo.clarin.eu beta-collections.clarin.eu - fcs.clarin-dev.eu - - | VLO beta and curation beta VCR Beta Discovery service Alpha FCS Netkernel Gitlab Runners | BETA BETA DEVELOPMENT BETA ? TOOL | 168.119.38.169 | CentOS 7 | 12 | 128 | 934GB | Hetzner | EX52-NVMe (Germany) | sysops@clarin.eu | https://robot.your-server.de/server | - | - | - | - | |
hetzner-vps2 | alpha-vlo.clarin.eu alpha-curation.clarin.eu alpha-collections.clarin.eu europeana-oai.clarin.eu | VLO alpha and curation alpha VCR Alpha Europeana OAI-PMH aggregator | DEVELOPMENT DEVELOPMENT | 136.243.133.121 | CentOS 7 | 12 | 128 | 934GB | Hetzner | EX52-NVMe (Germany) | sysops@clarin.eu | https://robot.your-server.de/server | |||||
hetzner-vps5 | DEVELOPMENT | 65.108.201.6 | AlmaLinux? 8 | 16 | 64 | 8000GB | AX51 (Helsinki) | sysops@clarin.eu | https://robot.your-server.de/server |
1.2. Externally managed, with central services
Canonical FQDN | Aliases | IPv4-address | OS | Docker | (v)CPUs | Memory (GiB) | Storage (GiB) | Hoster | Responsible |
---|---|---|---|---|---|---|---|---|---|
vz07-clarin-list?.im.hum.uu.nl | lists.clarin.eu newlists.clarin.eu | 131.211.143.192 | Debian 6 | n/a | ? | ? | ? | UU | ictenmedia@uu.nl - Official (generic) r.vanvalkenburg@uu.nl - Direct to René van Valkenburg |
fsd-cloud22.zam.kfa-juelich.de | monitoring.clarin.eu | 134.94.199.42 | Ubuntu 14.04.4 LTS | n/a | FZJ? | CLARIN-support@fz-juelich.de | |||
clarin.fz-juelich.de | - | 134.94.199.71 | n/a | FZJ? | CLARIN-support@fz-juelich.de | ||||
clarin.ids-mannheim.de | clarin.ids-mannheim.de | 193.196.8.17 | CentOS 7.4 | n/a | 4 | 16 | 64 | IDS? | Oliver Schonefeld |
weblicht.sfs.uni-tuebingen.de | weblicht.sfs.uni-tuebingen.de | 130.183.206.38 | Ubuntu 16.04 | 1.12.3 | 4 | 64 | 500 | UTU? | emanuel.dima@uni-tuebingen.de |
spraakbanken.gu.se/ws/fcs/2.0/aggregator/ | contentsearch.clarin.eu | 130.241.42.13 | Språkbanken | leif-joran.olsson@svenska.gu.seadded aa |
1.3. Decommissioned
Canonical FQDN | Aliases | IPv4-address | OS | Docker | Hoster | Decommisioning notes |
---|---|---|---|---|---|---|
dev-idp-clarin.esc.rzg.mpg.de | dev-idp.clarin.eu dev-sp.clarin.eu | 130.183.206.39 | Scientific Linux 7.5 | 18.09.0 | MPCDF | sysops@clarin.eu |
alpha-vlo-clarin.esc.rzg.mpg.de | 130.183.206.35 | Scientific Linux 7.4 | n/a | MPCDF | sysops@clarin.eu | |
beta-vlo-clarin.esc.rzg.mpg.de | beta-vlo.clarin.eu | 130.183.206.198 | Scientific Linux 7.2 | 17.05.0-ce | MPCDF | sysops@clarin.eu |
idp1-clarin.esc.rzg.mpg.de | idm.clarin.eu | 130.183.206.196 | Scientific Linux 7.4 | 18.06.0 | MPCDF | sysops@clarin.eu |
centres-clarin.esc.rzg.mpg.de | staging-centres.clarin.eu | 130.183.206.32 | Scientific Linux 7.5 | 18.09.0 | MPCDF | sysops@clarin.eu |
centres2-clarin.esc.rzg.mpg.de | centres.clarin.eu | 130.183.206.32 | Scientific Linux 7.5 | 18.09.0 | MPCDF | sysops@clarin.eu |
idp2-clarin.esc.rzg.mpg.de | - | 130.183.206.33 | Scientific Linux 7.4 | 18.06.0 | MPCDF | sysops@clarin.eu |
clarinvm.ics.muni.cz | 147.251.9.199 | CentOS 7.1.1503 | ?? | CESNET | sysops@clarin.eu | |
ems04.mpi.nl | 192.87.79.165 | Ubuntu 12.04.5 LTS | n/a | MPI-PL | sysops@clarin.eu | |
idp-clarin.esc.rzg.mpg.de | - | 130.183.206.37 | SLES 11.3 | n/a | MPCDF | sysops@clarin.eu |
stoor146.meta.zcu.cz | - | 147.228.242.146 | CentOS 7.1.1503 | 1.5.0 | CESNET | sysops@clarin.eu |
catalog-clarin?.esc.rzg.mpg.de | 192.87.79.171 | SLES 11.2 | n/a | MPI-PL | sysops@clarin.eu | |
im-linux-clarin-eu?.im.hum.uu.nl | www.clarin.eu | 131.211.143.212 | Debian 8 | n/a | UU | web team Sander Maijers ictenmedia@uu.nl |
im-linux-dev-clarin-eu.hum.uu.nl | - | 131.211.143.192 | Debian 8 | n/a | UU | web team Sander Maijers ictenmedia@uu.nl |
vz07-clarin-eu?.im.hum.uu.nl | - | 131.211.143.186 | Debian 8 | n/a | UU | web team Sander Maijers ictenmedia@uu.nl |
lvps83-169-5-155.dedicated.hosteurope.de | 83.169.5.155 | CentOS | n/a | HostEurope | Decommissioned per 31.05.2018 | |
lvps92-51-161-129.dedicated.hosteurope.de | vlo.clarin.eu | 92.51.161.129 | CentOS 7.1.1503 | n/a | HostEurope | Decomissioned per 31-10-2018 |
cloud-90-147-170-203.cloud.ba.infn.it | eosc-recas-vps1 | 90.147.170.203 | CentOS | RECAS | Fatal Crash April 2021, never recommissioned | |
rs238144.rs.hosteurope.de | vlo.clarin.eu | 91.250.82.71 | CentOS | HostEurope | Decomissioned July 2021 | |
rs236235.rs.hosteurope.de | alpha-vlo.clarin.eu Virtual Collection Registry Virtual Language Observatory docker-runner-hosteurope-1 docker-runner-hosteurope-2 discovery FCS beta | 91.250.80.240 | CentOS | HostEurope | Decomissioned July 2021 | |
249811.rs.hosteurope.de | Link Checker | 5.35.250.44 | CentOS | HostEurope | Decomissioned July 2021 |
2. DNS entries and TLS certificates
Hosted by TransIP
admins: Dieter Van Uytvanck, Andre Moreira, Willem Elbers
3. Getting access
Shell access to the CLARIN hosts is only possible via key-based SSH.
Contact sysops@clarin.eu to request access to a host. Make sure to include your public SSH key.
Instructions and guidelines on how to create your OpenSSH key pair can be found here.
3.1. Security
4. Default VM setup
These instructions describe how we install/provision/configure each host by default.
4.1. Connections
service | port | type | direction |
ssh | 22 | tcp | incoming |
ssh | 22 | tcp | outgoing to gitlab.com |
collectd | 25826 | tcp | outgoing |
fluentd | 24224 | tcp | outgoing |
4.2. Centos / Scientific Linux
Some notes on administering Centos / Scientific linux hosts.
4.3. SLES 11
We are in the process of migration our SLES 11 machines to CentOS/Scientific Linux. We collect some notes on administering SLES hosts.
4.4. Ubuntu
We are in the process of migration our Ubuntu machines to CentOS/Scientific Linux.
5. Deploying and running services
Repositories:
- Deploy script: https://gitlab.com/CLARIN-ERIC/deploy-script
- Control script: https://gitlab.com/CLARIN-ERIC/control-script
5.1. Deploy a service
In the deploy users home directory (/home/deploy):
sh deploy.sh --name service-name --git git-repo-name --tag 1.0.0
Updates are performed by running the same command with a different tag and then using the control.sh script to restart the service.
5.2. Initialize a service
In the deploy users home directory (/home/deploy):
sh control.sh service-name init
Customize <service-name>/.env as needed.
5.3. Start the service
In the deploy users home directory (/home/deploy):
sh control.sh service-name start
Other commands available: stop, restart, backup, restore, ...
6. Infrastructure and service status information
A manually curated service status overview including planned maintenance is kept at clarin.eu/status.
Service availability statistics (sourced by StatusCake) are available at status.clarin.eu. Incidents are also posted automatically to the private sysalert channel on Slack.
Maintainers of services, in particular core services and A-services are requested to submit expected downtime information timely. For more information, see Service status guidelines.
7. Documents
- CLARIN Infrastructure Overview
- Docker Workflow and best practices
- Proposal: High Availability for the CLARIN infrastructure
- Sysops - infrastructure management
8. Services
9. Updates
9.1. Schedule
The schedule is kept in this Google Doc. Don't forget to check the last field once you've done the updates.
9.2. Workflow
- Install security updates for beta servers on the monday. Reboot if neccessary, no announcement required
- Install security updates for production servers on Tuesday. If a reboot is neccessary make a proper announcement for the Thursday.
In exceptional cases spread the update over two weeks. Half of the servers in week 1, the other half on week 2.These cases include:
- exceptionally many updates
- updates to the kernel (might affect the docker daemon)
- updates to the docker daemon
9.3. Usefull commands
Task | CentOS (yum) | AlmaLinux (dnf) |
List all available | yum updateinfo list available | dnf list available
|
List available security updates | yum updateinfo list security | dnf updateinfo list --security
|
dnf updateinfo list --security --sec-severity [Severity]
| ||
Install all available secutiry updates | yum update --security | dnf upgrade --security
|
Install specific CVE | yum update --cve <CVE> | dnf upgrade --cve CVE-xxxx-xxxx,CVE-yyyy-yyyy
|
Check if a reboot is required | needs-restarting -r | dnf needs-restarting -r
|
Check for kernel and docker updates, pay special attention and exclude in case of doubt.
10. Metrics
10.1. Latency Checks
10.1.1. Latency dashboard
The latency check dashboard is available here: https://metrics.clarin.eu/d/000000019/latency
10.1.2. Managing latency checks
On transip-vps6 there is a collectd configuration in place that pings some of our VPSes. Ideally one for each provider / geographical area.
To edit the latency checks edit /etc/collectd.d/ping.conf:
LoadPlugin ping <Plugin ping> Host "rs238144.rs.hosteurope.de" Host "clarinvm.cesnet.cz" Host "idp1-clarin.esc.rzg.mpg.de" Host "clarin-vcr.ids-mannheim.de" Host "hetzner-vps1" </Plugin>
If a host is not properly reachable via a hostname, add a IP to hostname mapping in /etc/hosts. E.g.:
168.119.38.169 hetzner-vps1
11. Known issues
11.1. Docker
11.1.1. dial tcp: lookup index.docker.io: no such host
11.1.1.1. Error
dial tcp: lookup index.docker.io: no such host
11.1.1.2. Symptoms
While using Docker a user is unable to perform tasks such as pull new image or search for new images while the following error message appears:
# docker pull debian:8 Pulling repository debian FATA[0053] Get https://index.docker.io/v1/repositories/library/debian/images: dial tcp: lookup index.docker.io: no such host
11.1.1.3. Solution
No good solution available at this time.
11.1.1.4. References
- https://linuxconfig.org/docker-dial-tcp-lookup-index-docker-io-no-such-host-fix
- https://robinwinslow.uk/2016/06/23/fix-docker-networking-dns/
- https://stackoverflow.com/questions/29266560/docker-container-can-reach-dns-but-not-resolve-hosts
- https://github.com/moby/moby/issues/13381
12. GitLab
12.1. Managing a git repository on a server with a deploy key
- Enable deploy key in gitlab reository
- Goto Settings - Repository
- Expand "Deploy Keys"
- Enable the CLARIN keys (make sure to not use the public ones!)
- Configure a ssh connection for gitlab on the service
- Add the private part of the deploy key to /home/deploy/.ssh/id_rsa_gitlab_deploy
- Edit /home/deploy/.ssh/config
- Add:
#Deploys Host gitlab.com User git HostName gitlab.com IdentityFile ~/.ssh/id_rsa_gitlab_deploy
- Use the SSH location to clone the repository
- Example
git clone git@gitlab.com:CLARIN-ERIC/compose_transip_vps5.git