Version 483 (modified by 3 years ago) (diff) | ,
---|
Contents
CVEs
- Log4Shell: CVE-2021-44228
Tickets
1. Hosts
1.1. Internally managed
1.1.1. Production (clarin.eu)
Canonical FQDN | Aliases | Services | Ports | Service Type | IPv4-address | OS | (v)CPUs | Memory (GiB) | Storage (GiB) | Hoster | Type | Responsible | Support | Collectd | Fluentd | Docker | Compose |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
clarinvm.cesnet.cz | catalog.clarin.eu docker.clarin.eu nexus.clarin.eu office.clarin.eu | Netkernel CLARIN EU trac CLARIN NL trac SVN Metrics Component Registry Discovery service | PRODUCTION PRODUCTION PRODUCTION PRODUCTION PRODUCTION PRODUCTION PRODUCTION | 78.128.216.72 | CentOS 7.1.1503 | 8 | 32 | 500 | CESNET | sysops@clarin.eu | cesnet-virtual@cesnet.cz | 5.5 | n/a | 1.8.2 | n/a | ||
idp2-clarin.esc.rzg.mpg.de | aai2.clarin.eu beta-stats.clarin.eu | Infra SPF MD pipelines Infra static webserver Piwik | 8082, 44344, 44345 44343 44325 | PRODUCTION 2 PRODUCTION 2 PRODUCTION (Backup) | 130.183.206.33 | Scientific Linux 7.4 | 2 | 4 | 20 | MPCDF | sysops@clarin.eu | clarin-support@rzg.mpg.de | 5.8.0 | td-agent 1.2.2 | 18.06.0 | 1.22.0 | |
centres2-clarin.esc.rzg.mpg.de | centres.clarin.eu | Centre Registry | 44335 44325 (!) to be closed | PRODUCTION (Backup) | 130.183.206.40 | Scientific Linux 7.5 | 2 | 4 | 18 | MPCDF | sysops@clarin.eu | clarin-support@rzg.mpg.de | 5.8.1 | td-agent 1.2.6 | 18.09.0 | 1.22.0 | |
149-210-236-86.colo.transip.net | clarineu-vps2 | Reverse proxy | 80, 443 | PRODUCTION (Primary) | 149.210.236.86 Priv. net.: 192.168.1.3 | CentOS 7-5.1804.4.el7 | 2 | 4 | 150 | TransIP | VPS X4 @AMS0 (Amsterdam) | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | 5.8.0 | td-agent 1.2.2 | 18.06.0 | 1.22.0 |
136-144-215-36.colo.transip.net | clarineu-vps6 (clarineu-vps5) original | Reverse proxy | 80, 443 | PRODUCTION (Backup) | 136.144.215.36 Priv. net.: 192.168.1.1 | CentOS 7-5.1804.4.el7 | 2 | 4 | 150 | TransIP | VPS X4 @RTM0 (Delft) | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | 5.8.0 | td-agent 1.2.2 | 18.06.0 | 1.22.0 |
149-210-250-181.colo.transip.net | clarineu-vps9 www.clarin.eu | Main Website | 44305 | PRODUCTION (Primary) | 149.210.250.181 | CentOS 7-9.2009.1.el7 | 4 | 8 | 300 | TransIP | VPS X8 @AMS0 (Amsterdam) | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | 5.8.1 | td-agent 1.11.5 | 20.10.7 | 1.29.2 |
136-144-221-254.colo.transip.net | clarineu-vps8 www.clarin.eu | Main Website | 44305 | PRODUCTION (Backup) | 136.144.221.254 | CentOS 7-9.2009.1.el7 | 4 | 8 | 300 | TransIP | VPS X8 @RTM0 (Delft) | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | 5.8.1 | td-agent 1.11.5 | 20.10.7 | 1.29.2 |
37.97.220.172.colo.transip.net | clarineu-vps5 (clarineu-vps) original | Discovery service Infra SPF MD pipelines Infra static webserver Unity IDM (2.8.x) | 8444 44344, 44345 44343 | PRODUCTION 2 PRODUCTION 1 PRODUCTION 1 PRODUCTION | 37.97.220.172 | CentOS 7-5.1804.4.el7 | 2 | 4 | 150 | TransIP | VPS X4 @AMS0 (Amsterdam) | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | 5.8.0 | td-agent 1.2.2 | 18.06.0 | 1.22.0 |
136-144-199-95.colo.transip.net | clarineu-vps7 (clarineu-vps6) original stats.clarin.eu switchboard.clarin.eu | Piwik Centre Registry Switchboard | 8082, 4425 44335 44399 | PRODUCTION (Primary) PRODUCTION (Primary) PRODUCTION | 136.144.199.95 | CentOS 7-6.1810.2.el7 | 2 | 4 | 150 | TransIP | VPS X4 @RTM0 (Delft) | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | 5.8.1 | td-agent 1.4.2 | 18.09.8 | 1.24.1 |
136-144-208-88.colo.transip.net | clarineu-backups (clarineu-vps7) original | BACKUPS | PRODUCTION | 136.144.208.88 | CentOS | 1 | 1 | 2TB | TransIP | VPS X1 @RTM0 (Delft) | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | 5.8.1 | td-agent 1.3.3 | |||
CLARINEU-HAIP | High available IP address | 136.144.144.150 | - | - | - | - | TransIP | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | - | - | - | - | ||||
CLARINEU-HAIP-DEV | High available IP address | 136.144.144.52 | - | - | - | TransIP | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | - | - | - | - | |||||
clarin-vcr.ids-mannheim.de | collections.clarin.eu | VCR | 443 | PRODUCTION | 193.196.8.26 | CentOS | 4 | 8 | 100 | IDS | sysops@clarin.eu | Oliver Schonefeld CLARIN Slack | |||||
hetzner-vps3 | vlo.clarin.eu | VLO | PRODUCTION | 157.90.1.116 | CentOS 7 | 24 | 128 | 1800GB | Hetzner | AX61-NVMe | sysops@clarin.eu | https://robot.your-server.de/server |
1.1.2. Beta / Development (clarin-dev.eu)
Canonical FQDN | Aliases | Services | Ports | Service Type | IPv4-address | OS | (v)CPUs | Memory (GiB) | Storage (GiB) | Hoster | Type | Responsible | Support | Collectd | Fluentd | Docker | Compose |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
dev-idp-clarin.esc.rzg.mpg.de | dev-idp.clarin.eu dev-sp.clarin.eu | docker-runner-rzg-1 docker-runner-rzg-2 compreg (beta) | BUILD BUILD BETA | 130.183.206.39 | Scientific Linux 7.5 | 2 | 4 | 20 | MPCDF | sysops@clarin.eu | clarin-support@rzg.mpg.de | 5.8.1 | td-agent 1.2.6 | 18.09.0 | 1.23.2 | ||
centres-clarin.esc.rzg.mpg.de | staging-centres.clarin.eu | Centre Registry | 44335 | STAGING BETA (Backup) | 130.183.206.32 | Scientific Linux 7.5 | 1 | 2 | 18 | MPCDF | sysops@clarin.eu | clarin-support@rzg.mpg.de | 5.8.1 | td-agent 1.2.6 | 18.09.0 | 1.22.0 | |
alpha-vlo-clarin.esc.rzg.mpg.de | proxy-beta | 80, 443 | BETA | 130.183.206.35 | Scientific Linux 7.4 | 4 | 15 | 125 | MPCDF | Twan Goosen | clarin-support@rzg.mpg.de | 5.4.2 | n/a | n/a | n/a | ||
37-97-154-156.colo.transip.net | clarineu-vps3 dev-www.clarin.eu | Main Website (dev primary) idm idm-delegation-pilot | 4430, 4431 4432 2443, 1000 | DEVELOPMENT DEVELOPMENT BETA | 37.97.154.156 Priv. net. dev: 192.168.2.3 | CentOS 7.4.1708 | 2 | 4 | 150 | TransIP | VPS X4 @AMS0 (Amsterdam) | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | 5.8.0 | td-agent 1.2.6 | 18.06.0-ce | 1.22.0 |
37-97-184-230.colo.transip.net | clarineu-vps4 legacy-d8b3-www.clarin-dev.eu | Main Website (legacy reference instance: Drupal 8 Bootstrap 3) | 44305 | DEVELOPMENT | 37.97.184.230 | CentOS 7-9.2009.1.el7 | 2 | 6 | 150 | TransIP | VPS X4 @AMS0 (Amsterdam) | sysops@clarin.eu | CP, 2nd best: support@transip.nl Status: TransNOC | 5.8.1 | td-agent 1.11.5 | 20.10.7 | 1.29.2 |
193.144.46.251 | eosc-cesga-vps1 legacy-d7b3-www.clarin-dev.eu dev-www.clarin.eu beta-switchboard.clarin.eu nextcloud.clarin-dev.eu | Main Website (legacy reference instance: Drupal 7 Bootstrap 3) Main Website (dev backup) Switchboard beta Nextcloud instance with direct SSH access on port 22422 (B2Drop Switchoboard plugin dev) | 44305 44315 44399 44355 22422 | 193.144.46.251 | CentOs? 7.5.1804 | 12 | 24 | 800 | CESGA | sysops@clarin.eu | https://fedcloud-osservices.egi.cesga.es/ (Login via EGI sso) Ruben Diez: rdiez@cesga.es | 5.8.1 | td-agent 1.2.2 | 18.09.0 | 1.24.1 | ||
78.128.250.25 | eosc-cesnet-vps1 logs.clarin.eu | Elastic search/Kibana | Digital Object Gate | 44332 | 78.128.250.25 | 12 | 64 | 1000 | CESNET | sysops@clarin.eu | https://dashboard.cloud.muni.cz/auth/login/?next=/ (Select EGI checkin, then login via EGI sso) | |||||||
hetzner-vps1 | beta-vlo.clarin.eu beta-collections.clarin.eu - fcs.clarin-dev.eu - - | VLO beta and curation beta VCR Beta Discovery service Alpha FCS Netkernel Gitlab Runners | BETA BETA DEVELOPMENT BETA ? TOOL | 168.119.38.169 | CentOS 7 | 12 | 128 | 934GB | Hetzner | EX52-NVMe | sysops@clarin.eu | https://robot.your-server.de/server | - | - | - | - | |
hetzner-vps2 | alpha-vlo.clarin.eu alpha-collections.clarin.eu | VLO alpha and curation alpha VCR Alpha | DEVELOPMENT DEVELOPMENT | 136.243.133.121 | CentOS 7 | 12 | 128 | 934GB | Hetzner | EX52-NVMe | sysops@clarin.eu | https://robot.your-server.de/server |
1.1.3. Unused
Canonical FQDN | Aliases | Services | Ports | Service Type | IPv4-address | OS | (v)CPUs | Memory (GiB) | Storage (GiB) | Hoster | Type | Responsible | Support | Collectd | Fluentd | Docker | Compose |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
beta-vlo-clarin.esc.rzg.mpg.de | beta-vlo.clarin.eu | 130.183.206.198 | Scientific Linux 7.2 | 8 | 16 | 1000 | MPCDF | sysops@clarin.eu | clarin-support@rzg.mpg.de | 5.5 | n/a | 17.05.0-ce | 1.17.0 | ||||
idp1-clarin.esc.rzg.mpg.de | idm.clarin.eu | | 130.183.206.196 | Scientific Linux 7.4 | 2 | 4 | 20 | MPCDF | sysops@clarin.eu | clarin-support@rzg.mpg.de | 5.8.0 | td-agent 1.2.2 | 18.06.0 | 1.22.0 |
1.2. Externally managed, with central services
Canonical FQDN | Aliases | IPv4-address | OS | Docker | (v)CPUs | Memory (GiB) | Storage (GiB) | Hoster | Responsible |
---|---|---|---|---|---|---|---|---|---|
vz07-clarin-list?.im.hum.uu.nl | lists.clarin.eu newlists.clarin.eu | 131.211.143.192 | Debian 6 | n/a | ? | ? | ? | UU | ictenmedia@uu.nl - Official (generic) r.vanvalkenburg@uu.nl - Direct to René van Valkenburg |
fsd-cloud22.zam.kfa-juelich.de | monitoring.clarin.eu | 134.94.199.42 | Ubuntu 14.04.4 LTS | n/a | FZJ? | CLARIN-support@fz-juelich.de | |||
clarin.fz-juelich.de | - | 134.94.199.71 | n/a | FZJ? | CLARIN-support@fz-juelich.de | ||||
clarin.ids-mannheim.de | clarin.ids-mannheim.de | 193.196.8.17 | CentOS 7.4 | n/a | 4 | 16 | 64 | IDS? | Oliver Schonefeld |
weblicht.sfs.uni-tuebingen.de | weblicht.sfs.uni-tuebingen.de | 130.183.206.38 | Ubuntu 16.04 | 1.12.3 | 4 | 64 | 500 | UTU? | emanuel.dima@uni-tuebingen.de |
spraakbanken.gu.se/ws/fcs/2.0/aggregator/ | contentsearch.clarin.eu | 130.241.42.13 | Språkbanken | leif-joran.olsson@svenska.gu.seadded aa |
1.3. Decommissioned
Canonical FQDN | Aliases | IPv4-address | OS | Docker | Hoster | Decommisioning notes |
---|---|---|---|---|---|---|
clarinvm.ics.muni.cz | 147.251.9.199 | CentOS 7.1.1503 | ?? | CESNET | sysops@clarin.eu | |
ems04.mpi.nl | 192.87.79.165 | Ubuntu 12.04.5 LTS | n/a | MPI-PL | sysops@clarin.eu | |
idp-clarin.esc.rzg.mpg.de | - | 130.183.206.37 | SLES 11.3 | n/a | MPCDF | sysops@clarin.eu |
stoor146.meta.zcu.cz | - | 147.228.242.146 | CentOS 7.1.1503 | 1.5.0 | CESNET | sysops@clarin.eu |
catalog-clarin?.esc.rzg.mpg.de | 192.87.79.171 | SLES 11.2 | n/a | MPI-PL | sysops@clarin.eu | |
im-linux-clarin-eu?.im.hum.uu.nl | www.clarin.eu | 131.211.143.212 | Debian 8 | n/a | UU | web team Sander Maijers ictenmedia@uu.nl |
im-linux-dev-clarin-eu.hum.uu.nl | - | 131.211.143.192 | Debian 8 | n/a | UU | web team Sander Maijers ictenmedia@uu.nl |
vz07-clarin-eu?.im.hum.uu.nl | - | 131.211.143.186 | Debian 8 | n/a | UU | web team Sander Maijers ictenmedia@uu.nl |
lvps83-169-5-155.dedicated.hosteurope.de | 83.169.5.155 | CentOS | n/a | HostEurope | Decommissioned per 31.05.2018 | |
lvps92-51-161-129.dedicated.hosteurope.de | vlo.clarin.eu | 92.51.161.129 | CentOS 7.1.1503 | n/a | HostEurope | Decomissioned per 31-10-2018 |
cloud-90-147-170-203.cloud.ba.infn.it | eosc-recas-vps1 | 90.147.170.203 | CentOS | RECAS | Fatal Crash April 2021, never recommissioned | |
rs238144.rs.hosteurope.de | vlo.clarin.eu | 91.250.82.71 | CentOS | HostEurope | Decomissioned July 2021 | |
rs236235.rs.hosteurope.de | alpha-vlo.clarin.eu Virtual Collection Registry Virtual Language Observatory docker-runner-hosteurope-1 docker-runner-hosteurope-2 discovery FCS beta | 91.250.80.240 | CentOS | HostEurope | Decomissioned July 2021 | |
249811.rs.hosteurope.de | Link Checker | 5.35.250.44 | CentOS | HostEurope | Decomissioned July 2021 |
2. DNS entries and TLS certificates
Hosted by TransIP
admins: Dieter Van Uytvanck, Andre Moreira, Willem Elbers
3. Getting access
Shell access to the CLARIN hosts is only possible via key-based SSH.
Contact sysops@clarin.eu to request access to a host. Make sure to include your public SSH key.
Instructions and guidelines on how to create your OpenSSH key pair can be found here.
3.1. Security
4. Default VM setup
These instructions describe how we install/provision/configure each host by default.
4.1. Connections
service | port | type | direction |
ssh | 22 | tcp | incoming |
ssh | 22 | tcp | outgoing to gitlab.com |
collectd | 25826 | tcp | outgoing |
fluentd | 24224 | tcp | outgoing |
4.2. Centos / Scientific Linux
Some notes on administering Centos / Scientific linux hosts.
4.3. SLES 11
We are in the process of migration our SLES 11 machines to CentOS/Scientific Linux. We collect some notes on administering SLES hosts.
4.4. Ubuntu
We are in the process of migration our Ubuntu machines to CentOS/Scientific Linux.
5. Deploying and running services
Repositories:
- Deploy script: https://gitlab.com/CLARIN-ERIC/deploy-script
- Control script: https://gitlab.com/CLARIN-ERIC/control-script
5.1. Deploy a service
In the deploy users home directory (/home/deploy):
sh deploy.sh --name service-name --git git-repo-name --tag 1.0.0
Updates are performed by running the same command with a different tag and then using the control.sh script to restart the service.
5.2. Initialize a service
In the deploy users home directory (/home/deploy):
sh control.sh service-name init
Customize <service-name>/.env as needed.
5.3. Start the service
In the deploy users home directory (/home/deploy):
sh control.sh service-name start
Other commands available: stop, restart, backup, restore, ...
6. Infrastructure and service status information
A manually curated service status overview including planned maintenance is kept at clarin.eu/status.
Service availability statistics (sourced by StatusCake) are available at status.clarin.eu. Incidents are also posted automatically to the private sysalert channel on Slack.
Maintainers of services, in particular core services and A-services are requested to submit expected downtime information timely. For more information, see Service status guidelines.
7. Documents
- CLARIN Infrastructure Overview
- Docker Workflow and best practices
- Proposal: High Availability for the CLARIN infrastructure
- Sysops - infrastructure management
8. Services
9. Updates
(to be updated)
10. Metrics
10.1. Latency Checks
10.1.1. Latency dashboard
The latency check dashboard is available here: https://metrics.clarin.eu/d/000000019/latency
10.1.2. Managing latency checks
On transip-vps6 there is a collectd configuration in place that pings some of our VPSes. Ideally one for each provider / geographical area.
To edit the latency checks edit /etc/collectd.d/ping.conf:
LoadPlugin ping <Plugin ping> Host "rs238144.rs.hosteurope.de" Host "clarinvm.cesnet.cz" Host "idp1-clarin.esc.rzg.mpg.de" Host "clarin-vcr.ids-mannheim.de" Host "hetzner-vps1" </Plugin>
If a host is not properly reachable via a hostname, add a IP to hostname mapping in /etc/hosts. E.g.:
168.119.38.169 hetzner-vps1
11. Known issues
11.1. Docker
11.1.1. dial tcp: lookup index.docker.io: no such host
11.1.1.1. Error
dial tcp: lookup index.docker.io: no such host
11.1.1.2. Symptoms
While using Docker a user is unable to perform tasks such as pull new image or search for new images while the following error message appears:
# docker pull debian:8 Pulling repository debian FATA[0053] Get https://index.docker.io/v1/repositories/library/debian/images: dial tcp: lookup index.docker.io: no such host
11.1.1.3. Solution
No good solution available at this time.
11.1.1.4. References
- https://linuxconfig.org/docker-dial-tcp-lookup-index-docker-io-no-such-host-fix
- https://robinwinslow.uk/2016/06/23/fix-docker-networking-dns/
- https://stackoverflow.com/questions/29266560/docker-container-can-reach-dns-but-not-resolve-hosts
- https://github.com/moby/moby/issues/13381
12. GitLab
12.1. Managing a git repository on a server with a deploy key
- Enable deploy key in gitlab reository
- Goto Settings - Repository
- Expand "Deploy Keys"
- Enable the CLARIN keys (make sure to not use the public ones!)
- Configure a ssh connection for gitlab on the service
- Add the private part of the deploy key to /home/deploy/.ssh/id_rsa_gitlab_deploy
- Edit /home/deploy/.ssh/config
- Add:
#Deploys Host gitlab.com User git HostName gitlab.com IdentityFile ~/.ssh/id_rsa_gitlab_deploy
- Use the SSH location to clone the repository
- Example
git clone git@gitlab.com:CLARIN-ERIC/compose_transip_vps5.git