Changes between Version 12 and Version 13 of SystemAdministration/Hosts/ems04.mpi.nl

Timestamp:

09/05/16 10:43:50 (8 years ago)

Author:

Sander Maijers

Comment:

contents: change backup policy to reflect reality since a few months

SystemAdministration/Hosts/ems04.mpi.nl

@@ -86 +86 @@
 ## Backup policy ##
 
-An internal backup program, nbackup, backs up a lot of directories (daily, rotated weekly), esp. under `/srv/`. For more details, see scheduled job named 'cronjob 6: nbackup' in the relevant section. The nightly backups are stored in `/srv/backup/a/` and automatically retrieved by a cronjob running as the `corpman` user from the host `lux08` at MPI-PL. The cronjob status is e-mailed to [[mailto:"Sander Maijers" <Sander.Maijers@mpi.nl>]] (because MPI-PL forbids communication with Google e-mail addresses, and sander@clarin.eu and by extension sysops@clarin.eu exposes them to Google).
-Such a status e-mail looks like this:
-{{{
- Monday, February 22, 2016 7:32 AM
-retrieve_backups.retrieve_backups(backup_user_private_key_file_path = '/lat/tools/scripts/nbackup/ssh_keys/bekkup@ems04.mpi.nl__2012-12-11.rsa.priv',
-                                  host_keys_file_path               = '/lat/tools/scripts/nbackup/ssh_keys/known_hosts',
-                                  host_to_back_up                   = 'ems04.mpi.nl',
-                                  backup_Unix_user                  = 'bekkup',
-                                  local_backups_directory_path      = '/data/corpora/MPI_workspace/tla/backups/ems04.mpi.nl/',
-                                  backups_subdirectory_path         = 'a/',
-                                  remote_delete_after_retrieval     = True,
-                                  local_backup_file_permissions     = 0640)
-
-Retrieving backup file 'ems04.mpi.nl___2016-02-22T03:30Z.pax.lz.enc' ... 
-
-rotate_backups.rotate_backups(backups_directory_path    = '/data/corpora/MPI_workspace/tla/backups/ems04.mpi.nl/',
-                              host_to_back_up           = 'ems04.mpi.nl',
-                              number_of_backups_to_keep = 7)
-
-Keeping backup at '/data/corpora/MPI_workspace/tla/backups/ems04.mpi.nl/ems04.mpi.nl___2016-02-22T03:30Z.pax.lz.enc' ...
-Keeping backup at '/data/corpora/MPI_workspace/tla/backups/ems04.mpi.nl/ems04.mpi.nl___2016-02-21T03:30Z.pax.lz.enc' ...
-Keeping backup at '/data/corpora/MPI_workspace/tla/backups/ems04.mpi.nl/ems04.mpi.nl___2016-02-20T03:30Z.pax.lz.enc' ...
-Keeping backup at '/data/corpora/MPI_workspace/tla/backups/ems04.mpi.nl/ems04.mpi.nl___2016-02-19T03:30Z.pax.lz.enc' ...
-Keeping backup at '/data/corpora/MPI_workspace/tla/backups/ems04.mpi.nl/ems04.mpi.nl___2016-02-18T03:30Z.pax.lz.enc' ...
-Keeping backup at '/data/corpora/MPI_workspace/tla/backups/ems04.mpi.nl/ems04.mpi.nl___2016-02-17T03:30Z.pax.lz.enc' ...
-Keeping backup at '/data/corpora/MPI_workspace/tla/backups/ems04.mpi.nl/ems04.mpi.nl___2016-02-16T03:30Z.pax.lz.enc' ...
-Removing backup at '/data/corpora/MPI_workspace/tla/backups/ems04.mpi.nl/ems04.mpi.nl___2016-02-15T03:30Z.pax.lz.enc' ...
-}}}
+An internal backup program, nbackup, backs up a lot of directories (daily, rotated weekly), esp. under `/srv/`. 
+For more details, see scheduled job named 'cronjob 6: nbackup' in the relevant section. 
+The nightly backups are stored in `/srv/backup/a/`.
+'''They must be manually moved from this directory to [[clarinvm.ics.muni.cz]].'''
 
 # Required connectivity (firewall) #
 
 ## Current status ##
+
 ### External firewalls ###
+
 None.
+
 ### Internal firewall ###
 
-{{{
-ACCEPT net fw tcp 80
-ACCEPT net fw tcp 443
-ACCEPT net fw tcp 22
-
-## LDAP access
-ACCEPT net:131.211.143.186 fw tcp 10389 # (OLD) UU ICT & Media: www.clarin.eu, tst.clarin.eu
-ACCEPT net:131.211.143.212 fw tcp 10389 # UU ICT & Media: www.clarin.eu
-# ACCEPT net:131.211.143. fw tcp 10389 # UU ICT & Media: www-staging.clarin.eu
-ACCEPT net:131.211.143.211 fw tcp 10389 # UU ICT & Media: www-dev.clarin.eu
-ACCEPT net:147.228.242.146 fw tcp 10389 # stoor146, nexus.clarin.eu
-ACCEPT net:147.251.9.199 fw tcp 10389 # clarinvm for Sonatype Nexus
-# ACCEPT net:195.169.216.170 fw tcp 10389 # Workstation home sanmai, temporary
-
-ACCEPT net:172.16.16.67 fw tcp 10389 # lux17.mpi.nl
-ACCEPT net:134.94.32.21 fw tcp 10389 # Owncloud, Benedikt Von St. Vith
-ACCEPT net:134.94.199.71 fw tcp 10389 # Owncloud 2 (new server), Benedikt Von St. Vith
-ACCEPT net:172.16.17.200 fw tcp 10389 # tlatest08.mpi.nl
-
-## LDAP <-> catalog CLARIN IdP
-ACCEPT net:192.87.79.171 fw tcp 10389
-
-
-ACCEPT net:130.183.206.196 fw tcp 10389
-## idp2-clarin.esc.rzg.mpg.de
-ACCEPT net:130.183.206.33 fw tcp 10389
-
-## LDAP <-> dev-idp.clarin.eu
-ACCEPT net:130.183.206.39 fw tcp 10389
-
-ACCEPT net fw icmp
-}}}
+See `/etc/shorewall/rules`.
 
 # Scheduled jobs #
 
-## As superuser ##
-{{{
-#!sh
-
-SHELL=/bin/sh
-PATH=/usr/local/sbin/:/usr/local/bin/:/sbin:/bin/:/usr/sbin/:/usr/bin/:/root/bin/:/srv/apps/installations/xmlsectool-1.2.0/
-MAILTO='sysops@clarin.eu'
-
-from_email='cronjobber_ems04@ems04'
-#sanmai_email='sander@clarin.eu'
-sysops_email='sysops@clarin.eu'
-cronjobber='/srv/apps/installations/cronjobber'
-checker='/srv/scripts/exec/cronjobber_checker.sh'
-cronjobs_state_directory_path='/srv/cronjobs_state/'
-nbackup_file_path='nbackup_wrapper.sh'
-spf_directory_path='/srv/Python/venvs/2014-11-20_SPF/'
-
-*/15 *  * * * "${cronjobber}" -n 'cronjob 1: cron.php refresh for Drupal 6' -x "${checker}" -a 7d -s "${cronjobs_state_directory_path}/1/" -c 'curl --verbose -f --compressed "https://user.clarin.eu/cron.php"' -F "${from_email}" -T "$MAILT
-O"
-
-# */25 *  * * * "${cronjobber}" -n 'cronjob 2: Publish CMD files to infra.clarin.eu' -x "${checker}" -a 7d -s "${cronjobs_state_directory_path}/2/" -c '/srv/scripts/exec/update-cmd.sh' -T "${dietuyt_email}" -F "${from_email}" -T "$MAILTO"
-
-0 */56  * * * "${cronjobber}" -n 'cronjob 3: /srv/www/drupal6/ and /srv/www/infra.clarin.eu/content/ (Apache httpd) file permissions' -x "${checker}" -a 7d -s "${cronjobs_state_directory_path}/3/" -c '/srv/scripts/exec/Apache_pub_file_per
-missions.sh' -F "${from_email}" -T "$MAILTO"
-
-*/15 *  * * * "${cronjobber}" -n 'cronjob 4: user.clarin.eu Drupal 6 <-> LDAP sync.' -x "${checker}" -a 4d -s "${cronjobs_state_directory_path}/4/" -c '/srv/scripts/exec/Drupal_6_to_LDIF.r' -F "${from_email}" -T sysops@clarin.eu
-
-## 27-2-2016: Disabled backup for emergency disk space issues on /srv/. The /srv/ backup backed up the MySQL db as well now, resulting in too large backup files (2.2 GiB).
-## 29-2-2016: Re-enabled after adding new virtual disk with uuid ec75d326-28d9-492c-af34-e5bab3a929bd and adding /srv/backup -> /mnt/backup.
-30   4  * * * "${cronjobber}" -n 'cronjob 6: nbackup' -x "${checker}" -a 7d -s "${cronjobs_state_directory_path}/6/" -c "${nbackup_file_path} back_up -host_dumps_directory /srv/dumps_for_backup/ -SVN_repositories_base_directory /srv/subversion/ -Trac_installations_base_directory /srv/trac/ /srv/backup/a/ /etc/ /srv/LDAP/ /root/ /srv/scripts/ /srv/www/ /var/log/apache2/ /srv/cronjobs_state/ /home/ /var/lib/postgresql/ /srv/Python/ /srv/apps/" -F "${from_email}" -T sysops@clarin.eu
-
-26 7-19/1 * * * "${cronjobber}" -n 'cronjob 11: SPF - pyFF' -x "${checker}" -a 7d -s "${cronjobs_state_directory_path}/11/" -c '(cd "/srv/Python/venvs/2014-11-20_SPF/etc/pyff_config/" && . ./control.sh && pyff_fetch_md ; pyff_activate &&
-pyff_run job_a && pyff_run job_b && pyff_run job_c && pyff_run job_d && pyff_sign && pyff_verify_signatures && pyff_publish)' -T "$MAILTO"
-
-0 0     * * * "${cronjobber}" -n 'cronjob 13: Piwik - process Apache httpd logs of yesterday (infra.clarin.eu:443)' -x "${checker}" -a 7d -s "${cronjobs_state_directory_path}/13/" -c '/srv/www/piwik/misc/log-analytics/import_logs.py --log-format-name=ncsa_extended --log-hostname=infra.clarin.eu --url=https://stats.clarin.eu/ --idsite=4 $(date --date=yesterday +/var/log/apache2/infra.clarin.eu:443-access@\%Y-\%m-\%d.log)' -T sander@clarin.eu
-
-0 0    * * * "${cronjobber}" -n 'cronjob 14: Piwik - process Apache httpd logs of yesterday (infra.clarin.eu:80)' -x "${checker}" -a 7d -s "${cronjobs_state_directory_path}/14/" -c '/srv/www/piwik/misc/log-analytics/import_logs.py --log-format-name=ncsa_extended --log-hostname=infra.clarin.eu --url=https://stats.clarin.eu/ --idsite=4 $(date --date=yesterday +/var/log/apache2/infra.clarin.eu:80-access@\%Y-\%m-\%d.log)' -T sander@clarin.eu
-
-44 0     * * * "${cronjobber}" -n 'cronjob 15: Compress old daily timestamped logs in /var/log/apache2/' -x "${checker}" -a 7d -s "${cronjobs_state_directory_path}/15/" -c 'find /var/log/apache2/ -type f -name "*.log" -mtime +2 -exec gzip -9 "{}" \;' -T "$MAILTO"
-
-0 */1    * * * "${cronjobber}" -n 'cronjob 16: Remove old /tmp/tmp.* and /tmp/opendj-checkinstance-*.log files and /tmp/tmp*/ directories' -x "${checker}" -a 7d -s "${cronjobs_state_directory_path}/16/" -c 'find "/tmp/" -mindepth 1 -maxdepth 1 -mtime 1 \( -type f -name "tmp.*" -o -type f -name "opendj-checkinstance-*.log" -o -type d -name "tmp*" \) -exec rm -rf "{}" \+' -T "$MAILTO"
-}}}
-
-## As `www-data` ##
-{{{
-#!sh
-
-SHELL=/bin/sh
-PATH='/usr/local/sbin/:/usr/local/bin/:/sbin/:/bin/:/usr/sbin/:/usr/bin/'
-MAILTO='sysops@clarin.eu'
-
-from_email='cronjobber_ems04@ems04'
-sanmai_email='sander@clarin.eu'
-sysops_email='sysops@clarin.eu'
-cronjobber='/srv/apps/installations/cronjobber'
-checker='/srv/scripts/exec/cronjobber_checker.sh'
-cronjobs_state_directory_path='/srv/cronjobs_state/'
-nbackup_file_path='/root/bin/nbackup_wrapper.sh'
-spf_directory_path='/srv/Python/venvs/2014-11-20_SPF/'
-
-5 *     * * * "${cronjobber}" -n 'cronjob 12: Piwik archiving' -x "${checker}" -a 7d -s "${cronjobs_state_directory_path}/12/" -c '/usr/bin/php5 /srv/www/piwik/console core:archive --url=https://stats.clarin.eu/' -T sander@clarin.eu
-
-30 7-19/1  * * * "${cronjobber}" -n 'cronjob 17: SPF - Retrieve, normalize and split out SAML metadata about SPs as registered with identity federations according to the Centre Registry, and compare metadata about each SPF SP with the per
-tinent control version. ' -x "${checker}" -a 7d -s "${cronjobs_state_directory_path}/17/" -c '(cd /srv/scripts/exec/SPF_SAML_metadata_processor/ && python3 -Wall SPF_SAML_metadata_processor.py download_all_saml_metadata_from_identity_fede
-rations,split_identity_federation_saml_metadata_batches_and_diff_entities /srv/www/infra.clarin.eu/aai/sps_at_identity_federations/)' -T sander@clarin.eu
-
-*/25 *  * * * "${cronjobber}" -n 'cronjob 2: Publish CMD files to infra.clarin.eu' -x "${checker}" -a 7d -s "${cronjobs_state_directory_path}/2/" -c '/srv/scripts/exec/update-cmd.sh' -T 'sysops@clarin.eu'
-}}}
-
+There are numerous important crontobs running under uids for `root` and `www-data`.
+A utility called `cronjobber` manages this and stores cron job output/state under `/srv/cronjobs_state/`.
+Issue e.g. `sudo -u www-data crontab -u` to view the current status.