wiki:SystemAdministration

!!!!PLEASE DO NOT EDIT!!!!

THIS PAGE HAS BEEN MIGRATED TO THE Dev Google Drive

!!!!PLEASE DO NOT EDIT!!!!

CVEs

Tickets

Ticket Priority Summary Owner Created Modified
#1086 minor Problem with admin messages from lists.clarin.eu André Moreira 4 years ago 4 years ago
#1054 major Bad URL encoding in catalog.clarin.eu/vlo -> vlo.clarin.eu redirects André Moreira 6 years ago 6 years ago
#1010 minor Piwik overlay André Moreira 7 years ago 7 years ago
#1000 major Review server timezone André Moreira 7 years ago 7 years ago
#996 major catalog.clarin.eu SP: aa-statistics error message on login André Moreira 7 years ago 7 years ago
#981 major Add DKIM, DMARC DNS RRs for clarin.eu Sander Maijers 8 years ago 8 years ago
#887 major Configure local firewall for idp1-clarin and idp2-clarin Sander Maijers 8 years ago 8 years ago
#881 minor Migrate lists.clarin.eu Sander Maijers 8 years ago 8 years ago
#891 critical dmeventd for LVM hogs CPU on CentOS 7 Willem Elbers 8 years ago 8 years ago
#955 major Add Reverse DNS records for TransIP VMs Sander Maijers 8 years ago 8 years ago
#902 major Grant read access to httpd logs Sander Maijers 8 years ago 8 years ago
#885 major Django security update to 1.9.3 Sander Maijers 8 years ago 8 years ago
#884 critical Security-critical instructions to data centre support should have integrity Sander Maijers 8 years ago 8 years ago
#883 major Devise automatic tty I/O auditing setup & policy Sander Maijers 8 years ago 8 years ago
#865 major Make host pages adhere to host template in Trac Sander Maijers 8 years ago 8 years ago
#864 major Automated collection of running Docker containers Sander Maijers 8 years ago 8 years ago
#863 minor Automate collection of host info Sander Maijers 8 years ago 8 years ago

1. Hosts

1.1. Internally managed

1.1.1. Production (clarin.eu)

Canonical FQDN Aliases Services Ports Service Type IPv4-address OS (v)CPUs Memory
(GiB)
Storage
(GiB)
Hoster Type Responsible Support Collectd Fluentd Docker Compose
clarinvm.cesnet.cz catalog.clarin.eu
docker.clarin.eu
nexus.clarin.eu
office.clarin.eu
Netkernel
CLARIN EU trac
CLARIN NL trac
SVN
Metrics
Component Registry
Discovery service
PRODUCTION
PRODUCTION
PRODUCTION
PRODUCTION
PRODUCTION
PRODUCTION
PRODUCTION
78.128.216.72 CentOS
7.1.1503
8 32 500 CESNET sysops@clarin.eu cesnet-virtual@cesnet.cz 5.5 n/a 1.8.2 n/a
149-210-236-86.colo.transip.net clarineu-vps2 Reverse proxy 80, 443 PRODUCTION (Primary) 149.210.236.86
Priv. net.: 192.168.1.3
CentOS
7-5.1804.4.el7
2 4 150 TransIP VPS X4
@AMS0 (Amsterdam)
sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.0 td-agent
1.2.2
18.06.0 1.22.0
136-144-215-36.colo.transip.net clarineu-vps6
(clarineu-vps5) original
Reverse proxy 80, 443 PRODUCTION (Backup) 136.144.215.36
Priv. net.: 192.168.1.1
CentOS
7-5.1804.4.el7
2 4 150 TransIP VPS X4
@RTM0 (Delft)
sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.0 td-agent
1.2.2
18.06.0 1.22.0
149-210-250-181.colo.transip.net clarineu-vps9
www.clarin.eu
Main Website 44305 PRODUCTION (Primary) 149.210.250.181 CentOS
7-9.2009.1.el7
4 8 300 TransIP VPS X8
@AMS0 (Amsterdam)
sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.1 td-agent
1.11.5
20.10.7 1.29.2
136-144-221-254.colo.transip.net clarineu-vps8
www.clarin.eu
Main Website 44305 PRODUCTION (Backup) 136.144.221.254 CentOS
7-9.2009.1.el7
4 8 300 TransIP VPS X8
@RTM0 (Delft)
sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.1 td-agent
1.11.5
20.10.7 1.29.2
37.97.220.172.colo.transip.net clarineu-vps5
(clarineu-vps) original
Discovery service

Infra SPF MD pipelines

Infra static webserver

Unity IDM (2.8.x)
8444
44344, 44345
44343
PRODUCTION 2
PRODUCTION 1
PRODUCTION 1
PRODUCTION
37.97.220.172 CentOS
7-5.1804.4.el7
4 8 150
(300 but needs growfs)
TransIP VPS X8
@AMS0 (Amsterdam)
sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.0 td-agent
1.2.2
18.06.0 1.22.0
136-144-199-95.colo.transip.net clarineu-vps7
(clarineu-vps6) original
stats.clarin.eu
switchboard.clarin.eu
Matomo
Centre Registry
Switchboard
8082, 4425
44335
44399
PRODUCTION (Primary)
PRODUCTION (Primary)
PRODUCTION
136.144.199.95 CentOS
7-9.2009.1.el7
4 8 150
(300 available but FS not grown)
TransIP VPS X8
@RTM0 (Delft)
sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.1 td-agent
1.11.15
23.0.4 1.24.1
136-144-208-88.colo.transip.net clarineu-backups
(clarineu-vps7) original
BACKUPS PRODUCTION 136.144.208.88 CentOS 1 1 2TB TransIP VPS X1

@RTM0 (Delft)
sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.1 td-agent 1.3.3
CLARINEU-HAIP High available IP address 136.144.144.150 - - - - TransIP sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
- - - -
CLARINEU-HAIP-DEV High available IP address 136.144.144.52 - - - TransIP sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
- - - -
clarin-vcr.ids-mannheim.de collections.clarin.eu VCR 443 PRODUCTION 193.196.8.26 CentOS 4 8 100 IDS sysops@clarin.eu Oliver Schonefeld
CLARIN Slack
hetzner-vps3 vlo.clarin.eu VLO
Curation module/linkchecker
PRODUCTION 157.90.1.116 CentOS 7 24 128 1800GB Hetzner AX61-NVMe sysops@clarin.eu https://robot.your-server.de/server
hetzner-vps4 Infra SPF MD pipelines
Infra static webserver
Matomo
Centre Registry
8082, 44344, 44345
44343
44325
44335
PRODUCTION 2
PRODUCTION 2
PRODUCTION (Backup)
PRODUCTION (Backup)
95.216.225.96 CentOS 7 8 64 512GB Hetzner EX42-NVMe

(Finland)
sysops@clarin.eu https://robot.your-server.de/server 5.8.1 td-agent 1.11.5 20.10.12 1.29.2
transip-vps10 Unity IDM (3.x.x) PRODUCTION 37.97.132.227 AlmaLinux? 8 4 8 300GB TransIP VPS X8

@AMS0 (Amsterdam)
sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.9.0 td-agent 1.11.5 23.0.0 2.2.3

1.1.2. Beta / Development (clarin-dev.eu)

Canonical FQDN Aliases Services Ports Service Type IPv4-address OS (v)CPUs Memory
(GiB)
Storage
(GiB)
Hoster Type Responsible Support Collectd Fluentd Docker Compose
37-97-154-156.colo.transip.net clarineu-vps3
dev-www.clarin.eu
Main Website (dev primary)
idm
idm-delegation-pilot
4430, 4431
4432
2443, 1000
DEVELOPMENT
DEVELOPMENT
BETA
37.97.154.156
Priv. net. dev: 192.168.2.3
CentOS
7.9.2009
2 4 150 TransIP VPS X4
@AMS0 (Amsterdam)
sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.0 td-agent
1.2.6
20.10.12 2.2.3
37-97-184-230.colo.transip.net clarineu-vps4
legacy-d8b3-www.clarin-dev.eu
centres-staging.clarin.eu
alpha-d4-centres.clarin.eu
Main Website (legacy reference instance: Drupal 8 Bootstrap 3)
idm
Centre Registry

GitLab runners
44305

44335
44334
DEVELOPMENT
DEVELOPMENT
DEVELOPMENT
DEVELOPMENT
TOOL
37.97.184.230
Priv. net. dev: 192.168.2.4
AlmaLinux
8.5-4.el8
4 8 300 TransIP VPS X8
@AMS0 (Amsterdam)
sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.1 td-agent
1.11.5
20.10.7 1.29.2
193.144.46.251 eosc-cesga-vps1
legacy-d7b3-www.clarin-dev.eu
dev-www.clarin.eu
beta-switchboard.clarin.eu
nextcloud.clarin-dev.eu
Main Website (legacy reference instance: Drupal 7 Bootstrap 3)
Main Website (dev backup)
Switchboard beta
Nextcloud instance with direct SSH access on port 22422 (B2Drop Switchoboard plugin dev)
(OPENSTACK BASED - Non Standard MTU)

44305
44315
44399
44355
22422
193.144.46.251 CentOS
7.5.1804
12 24 800 CESGA sysops@clarin.eu https://fedcloud-osservices.egi.cesga.es/
(Login via EGI sso)

Ruben Diez: rdiez@cesga.es
5.8.1 td-agent
1.2.2
18.09.0 1.24.1
78.128.250.25 eosc-cesnet-vps1
logs.clarin.eu
Elastic search/Kibana
| Digital Object Gate
(OPENSTACK BASED - Non Standard MTU)

44332
78.128.250.25 12 64 1000 CESNET sysops@clarin.eu https://dashboard.cloud.muni.cz/auth/login/?next=/

(Select EGI checkin, then login via EGI sso)
hetzner-vps1 beta-vlo.clarin.eu
beta-collections.clarin.eu
-
fcs.clarin-dev.eu
-
-
VLO beta and curation beta
VCR Beta
Discovery service Alpha
FCS
Netkernel
Gitlab Runners
BETA
BETA
DEVELOPMENT
BETA
?
TOOL
168.119.38.169 CentOS 7 12 128 934GB Hetzner EX52-NVMe
(Germany)
sysops@clarin.eu https://robot.your-server.de/server - - - -
hetzner-vps2 alpha-vlo.clarin.eu
alpha-curation.clarin.eu
alpha-collections.clarin.eu
europeana-oai.clarin.eu
VLO alpha and curation alpha

VCR Alpha
Europeana OAI-PMH aggregator
DEVELOPMENT
DEVELOPMENT
136.243.133.121 CentOS 7 12 128 934GB Hetzner EX52-NVMe
(Germany)
sysops@clarin.eu https://robot.your-server.de/server
hetzner-vps5 DEVELOPMENT 65.108.201.6 AlmaLinux? 8 16 64 8000GB AX51 (Helsinki) sysops@clarin.eu https://robot.your-server.de/server

1.2. Externally managed, with central services

Canonical FQDN Aliases IPv4-address OS Docker (v)CPUs Memory
(GiB)
Storage
(GiB)
Hoster Responsible
vz07-clarin-list?.im.hum.uu.nl lists.clarin.eu
newlists.clarin.eu
131.211.143.192 Debian
6
n/a ? ? ? UU ictenmedia@uu.nl - Official (generic)
r.vanvalkenburg@uu.nl - Direct to René van Valkenburg
fsd-cloud22.zam.kfa-juelich.de monitoring.clarin.eu 134.94.199.42 Ubuntu 14.04.4 LTS n/a FZJ? CLARIN-support@fz-juelich.de
clarin.fz-juelich.de - 134.94.199.71 n/a FZJ? CLARIN-support@fz-juelich.de
clarin.ids-mannheim.de clarin.ids-mannheim.de 193.196.8.17 CentOS
7.4
n/a 4 16 64 IDS? Oliver Schonefeld
weblicht.sfs.uni-tuebingen.de weblicht.sfs.uni-tuebingen.de 130.183.206.38 Ubuntu
16.04
1.12.3 4 64 500 UTU? emanuel.dima@uni-tuebingen.de
spraakbanken.gu.se/ws/fcs/2.0/aggregator/ contentsearch.clarin.eu 130.241.42.13 Språkbanken leif-joran.olsson@svenska.gu.seadded aa

1.3. Decommissioned

Canonical FQDN Aliases IPv4-address OS Docker Hoster Decommisioning notes
dev-idp-clarin.esc.rzg.mpg.de dev-idp.clarin.eu
dev-sp.clarin.eu
130.183.206.39 Scientific Linux
7.5
18.09.0 MPCDF sysops@clarin.eu
alpha-vlo-clarin.esc.rzg.mpg.de 130.183.206.35 Scientific Linux
7.4
n/a MPCDF sysops@clarin.eu
beta-vlo-clarin.esc.rzg.mpg.de beta-vlo.clarin.eu 130.183.206.198 Scientific Linux
7.2
17.05.0-ce MPCDF sysops@clarin.eu
idp1-clarin.esc.rzg.mpg.de idm.clarin.eu 130.183.206.196 Scientific Linux
7.4
18.06.0 MPCDF sysops@clarin.eu
centres-clarin.esc.rzg.mpg.de staging-centres.clarin.eu 130.183.206.32 Scientific Linux
7.5
18.09.0 MPCDF sysops@clarin.eu
centres2-clarin.esc.rzg.mpg.de centres.clarin.eu 130.183.206.32 Scientific Linux
7.5
18.09.0 MPCDF sysops@clarin.eu
idp2-clarin.esc.rzg.mpg.de - 130.183.206.33 Scientific Linux
7.4
18.06.0 MPCDF sysops@clarin.eu
clarinvm.ics.muni.cz 147.251.9.199 CentOS
7.1.1503
?? CESNET sysops@clarin.eu
ems04.mpi.nl 192.87.79.165 Ubuntu
12.04.5 LTS
n/a MPI-PL sysops@clarin.eu
idp-clarin.esc.rzg.mpg.de - 130.183.206.37 SLES
11.3
n/a MPCDF sysops@clarin.eu
stoor146.meta.zcu.cz - 147.228.242.146 CentOS
7.1.1503
1.5.0 CESNET sysops@clarin.eu
catalog-clarin?.esc.rzg.mpg.de 192.87.79.171 SLES
11.2
n/a MPI-PL sysops@clarin.eu
im-linux-clarin-eu?.im.hum.uu.nl www.clarin.eu 131.211.143.212 Debian
8
n/a UU web team
Sander Maijers
ictenmedia@uu.nl
im-linux-dev-clarin-eu.hum.uu.nl - 131.211.143.192 Debian
8
n/a UU web team
Sander Maijers
ictenmedia@uu.nl
vz07-clarin-eu?.im.hum.uu.nl - 131.211.143.186 Debian
8
n/a UU web team
Sander Maijers
ictenmedia@uu.nl
lvps83-169-5-155.dedicated.hosteurope.de 83.169.5.155 CentOS n/a HostEurope Decommissioned per 31.05.2018
lvps92-51-161-129.dedicated.hosteurope.de vlo.clarin.eu 92.51.161.129 CentOS
7.1.1503
n/a HostEurope Decomissioned per 31-10-2018
cloud-90-147-170-203.cloud.ba.infn.it eosc-recas-vps1 90.147.170.203 CentOS RECAS Fatal Crash April 2021, never recommissioned
rs238144.rs.hosteurope.de vlo.clarin.eu 91.250.82.71 CentOS HostEurope Decomissioned July 2021
rs236235.rs.hosteurope.de alpha-vlo.clarin.eu
Virtual Collection Registry
Virtual Language Observatory
docker-runner-hosteurope-1
docker-runner-hosteurope-2
discovery
FCS beta
91.250.80.240 CentOS HostEurope Decomissioned July 2021
249811.rs.hosteurope.de Link Checker 5.35.250.44 CentOS HostEurope Decomissioned July 2021

2. DNS entries and TLS certificates

Hosted by TransIP

admins: Dieter Van Uytvanck, Andre Moreira, Willem Elbers

3. Getting access

Shell access to the CLARIN hosts is only possible via key-based SSH.

Contact sysops@clarin.eu to request access to a host. Make sure to include your public SSH key.

Instructions and guidelines on how to create your OpenSSH key pair can be found here.

3.1. Security

4. Default VM setup

These instructions describe how we install/provision/configure each host by default.

4.1. Connections

service port type direction
ssh 22 tcp incoming
ssh 22 tcp outgoing to gitlab.com
collectd 25826 tcp outgoing
fluentd 24224 tcp outgoing

4.2. Centos / Scientific Linux

Some notes on administering Centos / Scientific linux hosts.

4.3. SLES 11

We are in the process of migration our SLES 11 machines to CentOS/Scientific Linux. We collect some notes on administering SLES hosts.

4.4. Ubuntu

We are in the process of migration our Ubuntu machines to CentOS/Scientific Linux.

5. Deploying and running services

Repositories:

5.1. Deploy a service

In the deploy users home directory (/home/deploy):

sh deploy.sh --name service-name --git git-repo-name --tag 1.0.0

Updates are performed by running the same command with a different tag and then using the control.sh script to restart the service.

5.2. Initialize a service

In the deploy users home directory (/home/deploy):

sh control.sh service-name init

Customize <service-name>/.env as needed.

5.3. Start the service

In the deploy users home directory (/home/deploy):

sh control.sh service-name start

Other commands available: stop, restart, backup, restore, ...

6. Infrastructure and service status information

A manually curated service status overview including planned maintenance is kept at clarin.eu/status.

Service availability statistics (sourced by StatusCake) are available at status.clarin.eu. Incidents are also posted automatically to the private sysalert channel on Slack.

Maintainers of services, in particular core services and A-services are requested to submit expected downtime information timely. For more information, see Service status guidelines.

7. Documents

8. Services

9. Updates

9.1. Schedule

The schedule is kept in this Google Doc. Don't forget to check the last field once you've done the updates.

9.2. Workflow

  1. Install security updates for beta servers on the monday. Reboot if neccessary, no announcement required
  2. Install security updates for production servers on Tuesday. If a reboot is neccessary make a proper announcement for the Thursday.

In exceptional cases spread the update over two weeks. Half of the servers in week 1, the other half on week 2.These cases include:

  • exceptionally many updates
  • updates to the kernel (might affect the docker daemon)
  • updates to the docker daemon

9.3. Usefull commands

Task CentOS (yum) AlmaLinux (dnf)
List all available yum updateinfo list available dnf list available
List available security updates yum updateinfo list security dnf updateinfo list --security
dnf updateinfo list --security --sec-severity [Severity]
Install all available secutiry updates yum update --security dnf upgrade --security
Install specific CVE yum update --cve <CVE> dnf upgrade --cve CVE-xxxx-xxxx,CVE-yyyy-yyyy
Check if a reboot is required needs-restarting -r dnf needs-restarting -r

Check for kernel and docker updates, pay special attention and exclude in case of doubt.

10. Metrics

10.1. Latency Checks

10.1.1. Latency dashboard

The latency check dashboard is available here: https://metrics.clarin.eu/d/000000019/latency

10.1.2. Managing latency checks

On transip-vps6 there is a collectd configuration in place that pings some of our VPSes. Ideally one for each provider / geographical area.

To edit the latency checks edit /etc/collectd.d/ping.conf:

LoadPlugin ping
<Plugin ping>
        Host "rs238144.rs.hosteurope.de"
 	Host "clarinvm.cesnet.cz"
	Host "idp1-clarin.esc.rzg.mpg.de"
	Host "clarin-vcr.ids-mannheim.de"
	Host "hetzner-vps1"
</Plugin>

If a host is not properly reachable via a hostname, add a IP to hostname mapping in /etc/hosts. E.g.:

168.119.38.169	hetzner-vps1

11. Known issues

11.1. Docker

11.1.1. dial tcp: lookup index.docker.io: no such host

11.1.1.1. Error

dial tcp: lookup index.docker.io: no such host

11.1.1.2. Symptoms

While using Docker a user is unable to perform tasks such as pull new image or search for new images while the following error message appears:

# docker pull debian:8
Pulling repository debian
FATA[0053] Get https://index.docker.io/v1/repositories/library/debian/images: dial tcp: lookup index.docker.io: no such host

11.1.1.3. Solution

No good solution available at this time.

11.1.1.4. References

12. GitLab

12.1. Managing a git repository on a server with a deploy key

  1. Enable deploy key in gitlab reository
  • Goto Settings - Repository
  • Expand "Deploy Keys"
  • Enable the CLARIN keys (make sure to not use the public ones!)
  1. Configure a ssh connection for gitlab on the service
  • Add the private part of the deploy key to /home/deploy/.ssh/id_rsa_gitlab_deploy
  • Edit /home/deploy/.ssh/config
  • Add:
    #Deploys
    Host gitlab.com
        User git
        HostName gitlab.com
        IdentityFile ~/.ssh/id_rsa_gitlab_deploy
    
  1. Use the SSH location to clone the repository
  • Example
    git clone git@gitlab.com:CLARIN-ERIC/compose_transip_vps5.git
    
Last modified 8 months ago Last modified on 09/07/23 09:29:42