wiki:SystemAdministration

Version 271 (modified by Twan Goosen, 6 years ago) (diff)

--

Tickets

Ticket Priority Summary Owner Created Modified
#1086 minor Problem with admin messages from lists.clarin.eu André Moreira 4 years ago 4 years ago
#1054 major Bad URL encoding in catalog.clarin.eu/vlo -> vlo.clarin.eu redirects André Moreira 6 years ago 6 years ago
#1010 minor Piwik overlay André Moreira 7 years ago 7 years ago
#1000 major Review server timezone André Moreira 7 years ago 7 years ago
#996 major catalog.clarin.eu SP: aa-statistics error message on login André Moreira 7 years ago 7 years ago
#981 major Add DKIM, DMARC DNS RRs for clarin.eu Sander Maijers 8 years ago 8 years ago
#887 major Configure local firewall for idp1-clarin and idp2-clarin Sander Maijers 8 years ago 8 years ago
#881 minor Migrate lists.clarin.eu Sander Maijers 8 years ago 8 years ago
#891 critical dmeventd for LVM hogs CPU on CentOS 7 Willem Elbers 8 years ago 8 years ago
#955 major Add Reverse DNS records for TransIP VMs Sander Maijers 8 years ago 8 years ago
#902 major Grant read access to httpd logs Sander Maijers 8 years ago 8 years ago
#885 major Django security update to 1.9.3 Sander Maijers 8 years ago 8 years ago
#884 critical Security-critical instructions to data centre support should have integrity Sander Maijers 8 years ago 8 years ago
#883 major Devise automatic tty I/O auditing setup & policy Sander Maijers 8 years ago 8 years ago
#865 major Make host pages adhere to host template in Trac Sander Maijers 8 years ago 8 years ago
#864 major Automated collection of running Docker containers Sander Maijers 8 years ago 8 years ago
#863 minor Automate collection of host info Sander Maijers 8 years ago 8 years ago

1. Hosts

1.1. Internally managed

Canonical FQDN Aliases Services IPv4-address OS (v)CPUs Memory
(GiB)
Storage
(GiB)
Hoster Type Responsible Support Collectd Fluentd Docker Compose
lvps92-51-161-129.dedicated.hosteurope.de vlo.clarin.eu 92.51.161.129 CentOS
7.1.1503
8 32 500 HostEurope Virtual Server Linux Unlimited High I/O 8.0 sysops@clarin.eu support@hosteurope.de 5.5 n/a n/a n/a
rs236235.rs.hosteurope.de alpha-vlo.clarin.eu Netkernel
Virtual Collection Registry
Virtual Language Observatory
PRODUCTION
BETA
ALPHA
91.250.80.240 CentOS
7.1.1503
4 18 750 HostEurope Root Server M sysops@clarin.eu support@hosteurope.de 5.7.1 td-agent
1.0.2
17.05.0-ce 1.17.0
lvps83-169-5-155.dedicated.hosteurope.de Vlo benchmark BENCHMARKING 83.169.5.155 CentOS HostEurope Virtual Server Linux Enterprise 8.2 sysops@clarin.eu support@hosteurope.de
rs238144.rs.hosteurope.de 91.250.82.71 CentOS 8 40 800 HostEurope Root Server XL sysops@clarin.eu support@hosteurope.de
clarinvm.ics.muni.cz catalog.clarin.eu
docker.clarin.eu
nexus.clarin.eu
office.clarin.eu
CLARIN EU trac
CLARIN NL trac
SVN
Shibboleth SP (catalog)
-> Component Registry
PRODUCTION
PRODUCTION
PRODUCTION
PRODUCTION
147.251.9.199 CentOS
7.1.1503
8 32 500 CESNET sysops@clarin.eu cloud@metacentrum.cz 5.5 n/a 1.8.2 n/a
idp1-clarin.esc.rzg.mpg.de aai1.clarin.eu Unity IDM
Discovery service
PRODUCTION
PRODUCTION 1
130.183.206.196 Scientific Linux
7.4
2 4 20 MPCDF sysops@clarin.eu clarin-support@rzg.mpg.de 5.8.0 n/a 1.9.1 1.11.2
idp2-clarin.esc.rzg.mpg.de aai2.clarin.eu
beta-stats.clarin.eu
Infra SPF MD pipelines
Infra static webserver
Piwik beta
PRODUCTION 2
PRODUCTION 2
BETA
130.183.206.33 Scientific Linux
7.4
2 4 20 MPCDF sysops@clarin.eu clarin-support@rzg.mpg.de 5.8.0 n/a 17.05.0-ce 1.17.1
dev-idp-clarin.esc.rzg.mpg.de dev-idp.clarin.eu
dev-sp.clarin.eu
DEVELOPMENT
DEVELOPMENT
130.183.206.39 Scientific Linux
7.4
2 4 20 MPCDF sysops@clarin.eu clarin-support@rzg.mpg.de 5.8.0 td-agent
0.12.40
17.05.0-ce 1.8.1
centres-clarin.esc.rzg.mpg.de centres.clarin.eu Centre Registry PRODUCTION 130.183.206.32 SLES
11.4
1 1 17 MPCDF sysops@clarin.eu clarin-support@rzg.mpg.de 5.5 n/a n/a n/a
centres2-clarin.esc.rzg.mpg.de staging-centres.clarin.eu Centre Registry STAGING 130.183.206.40 Scientific Linux
7.4
1 1 2 MPCDF sysops@clarin.eu clarin-support@rzg.mpg.de 5.8.0 td-agent
1.0.2
17.05.0-ce 1.18.0
beta-vlo-clarin.esc.rzg.mpg.de beta-vlo.clarin.eu VLO BETA 130.183.206.198 Scientific Linux
7.2
8 16 1000 MPCDF sysops@clarin.eu clarin-support@rzg.mpg.de 5.5 n/a 17.05.0-ce 1.17.0
alpha-vlo-clarin.esc.rzg.mpg.de VLO ALPHA (semi-decomissioned) 130.183.206.35 Scientific Linux
7.3
4 15 125 MPCDF Twan Goosen clarin-support@rzg.mpg.de 5.4.2 n/a n/a n/a
149-210-236-86.colo.transip.net transip-vps2 Reverse proxy PRODUCTION (Primary) 149.210.236.86
192.168.1.2
CentOS
7.4.1708
2 4 150 TransIP sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.0 td-agent
1.0.2
17.05.0-ce 1.17.0
37-97-154-156.colo.transip.net transip-vps3
dev-www.clarin.eu
stats.clarin.eu
Main Website (dev)
Piwik
DEVELOPMENT
PRODUCTION
37.97.154.156
192.168.1.2
CentOS
7.4.1708
2 4 150 TransIP sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.0 td-agent
1.0.2
17.05.0-ce 1.16.1
37-97-184-230.colo.transip.net transip-vps4
www.clarin.eu
Main Website PRODUCTION 37.97.184.230
192.168.1.1
CentOS
7.4.1708
2 4 150 TransIP sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.0 td-agent
1.0.2
17.05.0-ce 1.17.1
37.97.220.172.colo.transip.net transip-vps
transip-vps5
Reverse proxy
Discovery service
Infra SPF MD pipelines
Infra static webserver
Gitlab runner
PRODUCTION (Secondary)
PRODUCTION 2
PRODUCTION 1
PRODUCTION 1
PRODUCTION
37.97.220.172
192.168.1.1
CentOS
7.4.1708
2 4 ? TransIP sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
5.8.0 td-agent
1.0.2
17.05.0-ce 1.17.1
CLARINEU-HAIP High available IP address 136.144.144.150 - - - - TransIP sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
- - - -
CLARINEU-HAIP-DEV High available IP address 136.144.144.52 - - - TransIP sysops@clarin.eu CP, 2nd best: support@transip.nl
Status: TransNOC
- - - -

1.2. Externally managed, with central services

Canonical FQDN Aliases IPv4-address OS Docker (v)CPUs Memory
(GiB)
Storage
(GiB)
Hoster Responsible
vz07-clarin-list?.im.hum.uu.nl lists.clarin.eu
newlists.clarin.eu
131.211.143.192 Debian
6
n/a ? ? ? UU ictenmedia@uu.nl
fsd-cloud22.zam.kfa-juelich.de monitoring.clarin.eu 134.94.199.42 Ubuntu 14.04.4 LTS n/a FZJ? CLARIN-support@fz-juelich.de
clarin.fz-juelich.de - 134.94.199.71 n/a FZJ? CLARIN-support@fz-juelich.de
clarin.ids-mannheim.de clarin.ids-mannheim.de 193.196.8.17 CentOS
7.4
n/a 4 16 64 IDS? Oliver Schonefeld
weblicht.sfs.uni-tuebingen.de weblicht.sfs.uni-tuebingen.de 130.183.206.38 Scientific Linux
6.7
n/a 4 64 500 UTU? emanuel.dima@uni-tuebingen.de

1.3. Decommissioned

Canonical FQDN Aliases IPv4-address OS Docker Hoster Responsible
ems04.mpi.nl 192.87.79.165 Ubuntu
12.04.5 LTS
n/a MPI-PL sysops@clarin.eu
idp-clarin.esc.rzg.mpg.de - 130.183.206.37 SLES
11.3
n/a MPCDF sysops@clarin.eu
stoor146.meta.zcu.cz - 147.228.242.146 CentOS
7.1.1503
1.5.0 CESNET sysops@clarin.eu
catalog-clarin?.esc.rzg.mpg.de 192.87.79.171 SLES
11.2
n/a MPI-PL sysops@clarin.eu
im-linux-clarin-eu?.im.hum.uu.nl www.clarin.eu 131.211.143.212 Debian
8
n/a UU web team
Sander Maijers
ictenmedia@uu.nl
im-linux-dev-clarin-eu.hum.uu.nl - 131.211.143.192 Debian
8
n/a UU web team
Sander Maijers
ictenmedia@uu.nl
vz07-clarin-eu?.im.hum.uu.nl - 131.211.143.186 Debian
8
n/a UU web team
Sander Maijers
ictenmedia@uu.nl

2. DNS entries and TLS certificates

Hosted by TransIP

admins: Dieter Van Uytvanck, Andre Moreira, Willem Elbers

3. Getting access

Shell access to the CLARIN hosts is only possible via key-based SSH.

Contact sysops@clarin.eu to request access to a host. Make sure to include your public SSH key.

Instructions and guidelines on how to create your OpenSSH key pair can be found here.

3.1. Security

4. Default VM setup

These instructions describe how we install/provision/configure each host by default.

4.1. Centos / Scientific Linux

Some notes on administering Centos / Scientific linux hosts.

4.2. SLES 11

We are in the process of migration our SLES 11 machines to CentOS/Scientific Linux. We collect some notes on administering SLES hosts.

4.3. Ubuntu

We are in the process of migration our Ubuntu machines to CentOS/Scientific Linux.

5. Infrastructure and service status information

A manually curated service status overview including planned maintenance is kept at clarin.eu/status.

Service availability statistics (sourced by StatusCake) are available at status.clarin.eu. Incidents are also posted automatically to the private sysalert channel on Slack.

Maintainers of services, in particular core services and A-services are requested to submit expected downtime information timely. For more information, see Service status guidelines.

6. Documents

7. Services

8. Updates

Bi-weekly on Tuesdays.

Canonical FQDN OS Updates Comments
Kernel Packages Docker
37.97.220.172.colo.transip.net CentOS
7
12-01-2018 12-01-2018 12-01-2018 Kernel: 3.10.0-693.11.6.el7.x86_64
dev-idp-clarin.esc.rzg.mpg.de Scientific Linux
7.4
12-01-2018 12-01-2018 12-01-2018 Kernel: 3.10.0-693.11.6.el7.x86_64
centres2-clarin.esc.rzg.mpg.de Scientific Linux
7.4
12-01-2018 25-01-2018 12-01-2018 Kernel: 3.10.0-693.11.6.el7.x86_64
beta-vlo-clarin.esc.rzg.mpg.de Scientific Linux
7.2
12-01-2018 12-01-2018 10-11-2017 Kernel: 3.10.0-693.11.6.el7.x86_64
lvps92-51-161-129.dedicated.hosteurope.de CentOS
7.1.1503
12-01-2018 12-01-2018 N/A Kernel: 3.10.0-042stab127.2 (OpenVZ)
rs236235.rs.hosteurope.de CentOS
7.4.1708
12-01-2018 12-01-2018 12-01-2018 Kernel: 3.10.0-693.11.6.el7.x86_64
clarinvm.ics.muni.cz CentOS
7.1.1503
12-01-2018 12-01-2018 PENDING Kernel: 3.10.0-229.el7.x86_64 <-- check with CESSNET
idp2-clarin.esc.rzg.mpg.de Scientific Linux
7.2
12-01-2018 12-01-2018 05-10-2017 Kernel: 3.10.0-693.11.6.el7.x86_64
idp1-clarin.esc.rzg.mpg.de Scientific Linux
7.2
12-01-2018 12-01-2018 PENDING Kernel: 3.10.0-693.11.6.el7.x86_64
149-210-236-86.colo.transip.net CentOS
7.1.1503
12-01-2018 12-01-2018 03-11-2017 Kernel: 3.10.0-693.11.6.el7.x86_64
37-97-154-156.colo.transip.net CentOS
7
12-01-2018 12-01-2018 14-11-2017 kernel: 3.10.0-693.11.6.el7.x86_64
37-97-184-230.colo.transip.net CentOS
7
12-01-2018 12-01-2018 14-11-2017 kernel: 3.10.0-693.11.6.el7.x86_64
centres-clarin.esc.rzg.mpg.de SLES
11.3
12-01-2018 12-01-2018 N/A Kernel: 3.0.101-108.21.1.x86_64
alpha-vlo-clarin.esc.rzg.mpg.de Scientific Linux
7.3
12-01-2018 12-01-2018 N/A Kernel: 3.10.0-693.11.6.el7.x86_64
ems04.mpi.nl Ubuntu
12.04.5 LTS
SKIPPED SKIPPED SKIPPED Planned to be decommissioned asap

9. Known issues

9.1. Docker

9.1.1. dial tcp: lookup index.docker.io: no such host

9.1.1.1. Error

dial tcp: lookup index.docker.io: no such host

9.1.1.2. Symptoms

While using Docker a user is unable to perform tasks such as pull new image or search for new images while the following error message appears:

# docker pull debian:8
Pulling repository debian
FATA[0053] Get https://index.docker.io/v1/repositories/library/debian/images: dial tcp: lookup index.docker.io: no such host

9.1.1.3. Solution

No good solution available at this time.

9.1.1.4. References

10. GitLab

10.1. Managing a git repository on a server with a deploy key

  1. Enable deploy key in gitlab reository
  • Goto Settings - Repository
  • Expand "Deploy Keys"
  • Enable the CLARIN keys (make sure to not use the public ones!)
  1. Configure a ssh connection for gitlab on the service
  • Add the private part of the deploy key to /home/deploy/.ssh/id_rsa_gitlab_deploy
  • Edit /home/deploy/.ssh/config
  • Add:
    #Deploys
    Host gitlab.com
        User git
        HostName gitlab.com
        IdentityFile ~/.ssh/id_rsa_gitlab_deploy
    
  1. Use the SSH location to clone the repository
  • Example
    git clone git@gitlab.com:CLARIN-ERIC/compose_transip_vps5.git