Default CentOS 7 host

IUS packages overview: ​https://dl.iuscommunity.org/pub/ius/stable/CentOS/7/x86_64/repoview/

1. Disable this

wpa_supplicant: we do not need wireless networking tools.

lvm2 and lvm2-libs: requires daemons, sometimes buggy ones such as dmeventd. Do not install without reason. At some data centers, e.g. MPCDF, LVM is mandatory.

tuned: undesirable?

NetworkManager: only desirable for GUI interaction, overwrites resolv.conf.

2. Run provisioning script

##Persist journalctl logs over reboot
## https://unix.stackexchange.com/questions/159221/how-do-i-display-log-messages-from-previous-boots-under-centos-7
mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal
systemctl restart systemd-journald

## Set hostname to hostname shown in hoster's control panel
## Use host's local name only, without the domain part in order for postfix mail system to work (see postfix configuration bellow)
sudo hostnamectl set-hostname 'clarineric-vps4'

## Install default packages
sudo yum update &&
## Add the official Docker yum repository to get access to the latest Docker versions
sudo tee '/etc/yum.repos.d/docker.repo' <<'EOF'
[dockerrepo]
name=Docker Repository
baseurl=https://download.docker.com/linux/centos/$releasever/x86_64/stable/
enabled=1
gpgcheck=1
gpgkey=https://download.docker.com/linux/centos/gpg
EOF
## Add the Fedora Extra Packages for Enterprise Linux (EPEL) yum repository
## http://www.rackspace.com/knowledge_center/article/install-epel-and-additional-repositories-on-centos-and-red-hat
sudo yum makecache fast

sudo yum install 'epel-release' 'git' 'docker-ce' 'unzip' 'mailx' 'tmux' 'tree' 'htop' 'rsync' 'collectd' &&
curl -L https://github.com/docker/compose/releases/download/1.22.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose &&
chmod +x /usr/local/bin/docker-compose &&
ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

mkdir /etc/docker/

### Configure collectd
# See section 2.1 on this page

### Configure Td-agent (fluentd)
# See section 3 on this page

### Configure docker daemon
# See section 2.3 on this page

## Create sysop users
sudo useradd 'andmor' &&
sudo usermod -aG 'wheel' 'andmor' &&
sudo passwd -d andmor &&
sudo passwd -e andmor
sudo useradd 'dietuyt' &&
sudo usermod -aG 'wheel' 'dietuyt' && 
sudo passwd -d dietuyt &&
sudo passwd -e dietuyt
sudo useradd 'wilelb' &&
sudo usermod -aG 'wheel' 'wilelb' &&
sudo passwd -d wilelb &&
sudo passwd -e wilelb

## Disable unneeded/interfering services.
sudo systemctl stop NetworkManager.service wpa_supplicant.service
sudo systemctl disable NetworkManager.service wpa_supplicant.service

sudo -e '/etc/ssh/sshd_config' &&
## Configure sshd in '/etc/ssh/sshd_config'
### No root login
# PermitRootLogin no
### No username/password login
# PasswordAuthentication no
# Restart sshd: service sshd restart

## Enable services
systemctl enable rsyslog
systemctl enable systemd-journald
systemctl enable collectd
systemctl enable docker

## Start services
systemctl start rsyslog.service
systemctl start systemd-journald.service
systemctl start collectd.service
systemctl start docker.service

sysctl net.bridge.bridge-nf-call-iptables=1 &&
sysctl net.bridge.bridge-nf-call-ip6tables=1

2.1. "deploy" and "setup" users' configuration -> needs manual copy of private deploy key for gitlab/github to /tmp before running

#!/bin/bash

## Make git only usable for git group
groupadd 'git'
chown root:git /bin/git
chmod 750 /bin/git

## Create deploy user and add to the the 'docker' group
sudo useradd 'deploy'
usermod -aG 'docker' 'deploy'

## Create setup user and add it to the 'deploy', 'git' and 'docker' groups:
useradd 'setup'
usermod -aG 'deploy' 'setup'
usermod -aG 'git' 'setup'
usermod -aG 'docker' 'setup'

## Configure 'setup' user gitlab access
mkdir -p /home/setup/.ssh
sudo tee '/home/setup/.ssh/config' <<'EOF'
#Deploys
Host gitlab.com
    User git
    HostName gitlab.com
    IdentityFile ~/.ssh/id_rsa_gitlab_deploy
EOF

## <1> <Manually copy the private deploy key to /tmp/id_rsa_gitlab_deploy>
mkdir -p /home/setup/.ssh/
mv /tmp/id_rsa_gitlab_deploy /home/setup/.ssh/

chown -R setup:setup /home/setup/.ssh
chmod -R 600 /home/setup/.ssh/id_rsa_gitlab_deploy
chmod go+rx /home/deploy

## Add sudo rights to deploy and setup users
echo 'Cmnd_Alias GIT_TAGSONLY_COMMANDS = /bin/git init *, /bin/git remote add origin *, /bin/git reflog expire --expire=all --all, /bin/git tag -l, /bin/git tag -d *, /bin/git fetch origin --depth\=1 +refs/tags/*\:refs/tags/*, /bin/git reset --hard tags/*, /bin/git ls-remote --tags *, /bin/git describe --tags, /bin/git gc --prune=all, /bin/git add ., /bin/git commit -m *, /bin/git rev-parse --abbrev-ref HEAD, /bin/git branch *, /bin/git branch -D *.backup*, /bin/git checkout *, /bin/git status, /bin/git status *, /bin/git diff, /bin/git diff *'  | sudo EDITOR='tee -a' visudo
echo 'Cmnd_Alias CONTAINER_CONTROL = /home/deploy/**/control.sh'  | sudo EDITOR='tee -a' visudo

echo 'deploy ALL= NOPASSWD: CONTAINER_CONTROL'  | sudo EDITOR='tee -a' visudo
echo 'deploy ALL=(setup) NOPASSWD: GIT_TAGSONLY_COMMANDS'  | sudo EDITOR='tee -a' visudo

echo 'setup ALL= NOPASSWD: CONTAINER_CONTROL'  | sudo EDITOR='tee -a' visudo

## Add alias to deploy user shell
echo 'alias git="sudo -u setup git"' | tee -a /home/deploy/.bashrc &> /dev/null

## Configure git on 'setup' user account
sudo -i -u setup git config --global user.email "sysops@clarin.eu"
sudo -i -u setup git config --global user.name "deploy" 
 
## Setup the deploy.sh script on the 'deploy' user home
sudo -u deploy bash -i -c "
(cd ~
mkdir deploy-assets
cd deploy-assets
git init deploy-script
cd deploy-script
git remote add origin git@gitlab.com:CLARIN-ERIC/deploy-script.git
git fetch origin --depth=1 +refs/tags/1.0.0-alpha8:refs/tags/1.0.0-alpha8
git reset --hard tags/1.0.0-alpha8
git gc --prune=all
cd ../..
ln -s deploy-assets/deploy-script/deploy.sh deploy.sh)"

2.2. Postfix configuration for host email (script) -> run as it is

#!/bin/bash

DOMAIN=clarin.eu
ROOT_EMAIL=sysops@${DOMAIN}
SMTP_ADDR=vps.transip.email:587
SMTP_USER=< check: Transip control panel->BladeVPS->Manage your mail service->Username & Password. >
SMTP_PWD=< check: Transip control panel->BladeVPS->Manage your mail service->Username & Password. >


yum install cyrus-sasl cyrus-sasl-plain sendmail

postconf -e "inet_protocols=ipv4"
postconf -e "mydomain=${DOMAIN}"
postconf -e "myhostname=`hostname | sed -E "s/\./-/g"`"
postconf -e "myorigin=\$myhostname"
postconf -e "mydestination=\$myhostname,localhost.\$mydomain,localhost"
postconf -e "virtual_alias_maps=regexp:/etc/postfix/virtual"
postconf -e "sender_canonical_maps=regexp:/etc/postfix/canonical"
postconf -e "smtp_header_checks=regexp:/etc/postfix/header_checks"
postconf -e "mynetworks_style=host"
postconf -e "remote_header_rewrite_domain=${DOMAIN}"
postconf -e "append_dot_mydomain=no"
postconf -e "relayhost=vps.transip.email:587"
postconf -e "smtp_sasl_auth_enable=yes"
postconf -e "smtp_sasl_security_options=noanonymous"
postconf -e "smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd"
postconf -e "smtp_use_tls=yes"
postconf -e "smtp_tls_security_level=encrypt"
postconf -e "smtp_tls_note_starttls_offer=yes"
printf '%s\n' "/^(.+)@(.+)/ ${ROOT_EMAIL}" | tee -a /etc/postfix/virtual &> /dev/null
printf '%s\n' "/^(.+)@([a-zA-Z0-9\-]+)$/ vps-\${1}_at_\${2}@${DOMAIN}" | tee -a /etc/postfix/canonical &> /dev/null
printf '%s\n' "/^From:[[:space:]]+vps\-(.+)_at_(.+)@(.+)/ REPLACE From: \"VPS host mail [\${1}@\${2}]\" <vps-\${1}_at_\${2}@\${3}>" | tee -a /etc/postfix/header_checks &> /dev/null
printf '%s\n' "${SMTP_ADDR} ${SMTP_USER}:${SMTP_PWD}" | tee -a /etc/postfix/sasl_passwd &> /dev/null
postmap /etc/postfix/virtual
postmap /etc/postfix/canonical
postmap /etc/postfix/header_checks
postmap /etc/postfix/sasl_passwd
chown root:root /etc/postfix/sasl_passwd /etc/postfix/sasl_passwd.db
chmod 0600 /etc/postfix/sasl_passwd /etc/postfix/sasl_passwd.db
chown -Rv postfix:postfix /var/lib/postfix

sudo service postfix restart

## Test the configuration
printf '%s\n' 'test' | mail -s "test message from $(hostname)" root

2.3. Collectd configuration

Create or edit /etc/collectd.d/metrics.clarin.eu.conf with the following content:

LoadPlugin cpu
LoadPlugin df
<Plugin df>
       MountPoint "/"
       IgnoreSelected false
       ReportByDevice false
       ReportReserved false
       ReportInodes true
       ValuesAbsolute true
       ValuesPercentage true
</Plugin>
LoadPlugin disk
# <Plugin disk>
# </Plugin>
LoadPlugin interface
<Plugin interface>
       Interface "eth0"
       IgnoreSelected false
</Plugin>
LoadPlugin load
LoadPlugin memory
LoadPlugin network
<Plugin network>
    Server "metrics.clarin.eu" "25826"
</Plugin>
LoadPlugin processes
LoadPlugin users

2.4. Docker daemon configuration

Create or edit /etc/docker/daemon.json with the following content:

{
  "log-driver": "fluentd",
  "log-opts": {
    "fluentd-address": "localhost:24224",
    "tag": "docker.{{.Name}}"
  }
}

2.4.1. Docker daemon storage driver options

For docker versions 18+ on CentOS 7 no extra configuration is needed. The system will use the overlay2 driver by default.

More information can be found ​here.

3. Td-agent

### Td-agent installation ###

Reference: ​https://docs.fluentd.org/v1.0/articles/install-by-rpm

Run as it is:

#!/bin/bash

TD_AGENT_VERSION="3.2.0-0.el7"
MULTI_FORMAT_PARSER_VERSION="1.0.0"
SECURE_FORWARD_PARSER_VERSION="0.4.5"
 
echo "=============================="
echo " td-agent Installation Script "
echo "=============================="
echo "This script requires superuser access to install rpm packages."
echo "You will be prompted for your password by sudo."

# clear any previous sudo permission
sudo -k

# run inside sudo
sudo sh <<SCRIPT

  # add GPG key
  rpm --import https://packages.treasuredata.com/GPG-KEY-td-agent

  # add treasure data repository to yum
  cat >/etc/yum.repos.d/td.repo <<'EOF';
[treasuredata]
name=TreasureData
baseurl=http://packages.treasuredata.com/3/redhat/\$releasever/\$basearch
gpgcheck=1
gpgkey=https://packages.treasuredata.com/GPG-KEY-td-agent
EOF

  # update your sources
  yum check-update

  # install the toolbelt
  yes | yum install -y td-agent-${TD_AGENT_VERSION}

SCRIPT

# message
echo ""
echo "Installation completed. Happy Logging!"
echo ""
echo "NOTE: In case you need any of these:"
echo "  1) security tested binary with a clear life cycle management"
echo "  2) advanced monitoring and management"
echo "  3) support SLA"
echo "Please check Fluentd Enterprise (https://www.treasuredata.com/fluentd/)."

/opt/td-agent/embedded/bin/gem install fluent-plugin-multi-format-parser -v ${MULTI_FORMAT_PARSER_VERSION}
/opt/td-agent/embedded/bin/gem install fluent-plugin-secure-forward -v ${SECURE_FORWARD_PARSER_VERSION}
mkdir /var/log/fluent
mkdir -p /home/td-agent/certstore
#put ca_cert.pem in /home/td-agent/certstore/ca_cert.pem
chown -R td-agent /var/log/fluent/
chown -R td-agent:td-agent /home/td-agent
systemctl enable td-agent
systemctl start td-agent.service

3.1. Td-agent Configuration

Create or edit /etc/td-agent/td-agent.conf with the following content (adapt the variable `self_hostname`):

<system>
  log_level warn
</system>

<source>
    @type forward
    port 24224
    bind 127.0.0.1
</source>

<filter **>
  @type record_transformer
  <record>
    clarin_host ${hostname}
  </record>
</filter>

<filter docker.**>
    @type parser
    key_name log
    reserve_data true
    remove_key_name_field true
    <parse>
        @type multi_format
        #Match default fluentd messages
        <pattern>
            format regexp
            expression /^(?<log_time>[^ ]+ [^ ]+ [^ ]+) (?<log_source>[^:]+): \{\\*"message\\*":\\*"(?<message>.*?)\\*"\}$/
        </pattern>
        #Match fluentd JSON messages
        <pattern>
            format regexp
             expression /^(?<log_time>[^ ]+ [^ ]+ [^ ]+) (?<log_source>[^:]+): (?<message>\{.*\})$/
        </pattern>
        #Match plain stdout messages
        <pattern>
            format regexp
            expression /^(?<message>.*)$/
        </pattern>
    </parse>
</filter>

<filter docker.**>
    @type parser
    key_name message
    reserve_data true
    remove_key_name_field true
    <parse>
      @type multi_format
      <pattern>
        format json
      </pattern>
      <pattern>
        format none
      </pattern>
    </parse>
</filter>

<match docker.**>
  @type rewrite_tag_filter
  hostname_command "hostname -s"
  <rule>
    key container_id
    pattern ^(.*)$
    tag ${hostname}.${tag}
  </rule>
</match>

<match fluent.**>
  @type rewrite_tag_filter
  hostname_command "hostname -s"
  <rule>
    key message
    pattern ^(.*)$
    tag ${hostname}.${tag}
  </rule>
</match>

<match **>
    @type copy
    <store>
        @type secure_forward
        self_hostname transip-vps5
        shared_key    mxXWooqM2NWBoFbkpmALz8Aw

        secure yes
        ca_cert_path /home/td-agent/certstore/ca_cert.pem

        <server>
            host 91.250.80.240
            port 24284
        </server>
        flush_interval 30s
    </store>
    <store>
        @type file
        path /var/log/fluent/aggregation
        symlink_path /var/log/fluent/aggregation.current
        time_slice_format %Y%m%d
        time_slice_wait 10m
        time_format %Y%m%dT%H%M%S%z
        time_key sent_at
        compress gzip
        buffer_type file
        buffer_path /var/log/fluent/aggregation.*
        append true
        flush_interval 30s
    </store>
</match>

Restart td-agent service

systemctl restart td-agent.service

4. Install collectd

5. Setting multiple IPv4 addresses

In /etc/sysconfig/network-scripts/ifcfg-eth0:

# Generated by dracut initrd
NAME="eth0"
HWADDR="52:54:00:b1:a5:b9"
ONBOOT=yes
NETBOOT=yes
UUID="f6b8e07f-d5c0-4729-bfc2-5f90cbfa06dd"
IPV6INIT=no
IPV6_AUTOCONF=no
BOOTPROTO=static
TYPE=Ethernet
IPADDR0=37.97.154.156
IPADDR1=37.97.157.49
IPADDR2=37.97.157.50
NETMASK0=255.255.255.0
NETMASK1=255.255.255.0
NETMASK2=255.255.255.0
GATEWAY=37.97.154.1
GATEWAY0=37.97.154.1
GATEWAY1=37.97.157.1
GATEWAY2=37.97.157.1

1. SELinux

Check status:

sestatus

Check policy of a file:

ls -Z /etc/collectd.conf

Check policy of a process:

ps -Z docker

Set user, role and type on a file:

sudo chcon -v --user=system_u --role=object_r --type=etc_t /etc/collectd.conf

Restore policy on a file:

restorecon -v /etc/collectd.conf