CMDI 1.2 Rollout

To take place on 7 and 8 July 2016.

Also see the previously published ​rollout plan (which is slightly outdated in the details by now).

Rough schedule

• Thursday
  • ~10:30~12:00 Component Registry preparation + beta
  • ~13:00~15:00 Component Registry production
  • ~15:00~17:00 VLO beta
• Friday
  • ~9:00~12:00 VLO production
  • ~13:00~17:00 New harvest, testing and announcements

Detailed release workflow

Steps in order of execution

Preparation

Preparation steps can take place a couple of days before the deployment

• [x] Twan: Announce downtime on 7/8 July
• [x] Twan: Prepare component conversion, i.e. fix/remove problematic components and profiles
• [x] Willem/Menzo: Fix specifications with implicit cardinalities
• [x] Menzo: Make deployment packages for the harvester (ultimately 6 July)
  • ​oai-harvest-manager-0.5.a06cc3.tar.gz (direct link)
  • ​oaiharvest.sh (updated handling of log files, commented out Meertens sync as Meertens is (for now) part of the main CLARIN harvest)
  • ​oaiharvest-env.sh (environment vars)
  • cron job for Meertens harvest can be disabled (for now)
• [x] Menzo: Run 'Others' harvest on alpha, package results for deployment
  • ​others.tar.bz2 (direct link; to replace the dirs in /srv/vlo-data/others/)
  • ​others-sil.tar.bz2 (direct link; to be added to /srv/vlo-data/others/)
• [x] Menzo: Run 'Clarin' harvest on alpha, package results for deployment
  • ​clarin.tar.bz2 (direct link; to replace the dirs in /srv/vlo-data/clarin/)
• [x] Twan: Make deployment packages for the VLO (ultimately 6 July)
  • ​vlo-4.0.0-beta2-Distribution.tar.gz (direct link)
  • ​vlo-4.0.0-Distribution.tar.gz (direct link)

Toolkit

One day before the deployment

• [x] Menzo: Change ​the default Component Registry location from the alpha server to the production server
• [x] Menzo: Tag/release toolkit on GitHub, make development branch (master should be stable)
  • ​https://github.com/clarin-eric/cmdi-toolkit/releases/tag/cmdi-1.2.0
• [x] Menzo: Check if the toolkit is available and up-to-date at ​https://infra.clarin.eu/CMDI/1.x
• [x] Twan: Make a runnable package with ready to use configuration file for the conversion tool
  • ​component-updater-2.1.0.zip (direct link)
• [x] Twan: Make a deployment package for the Component Registry
  • ​component-registry-rest-2.1.0-beta-mpi-deploy.tar.gz (direct link - beta)
  • ​component-registry-rest-2.1.0-mpi-deploy.tar.gz (direct link)

Component Registry

Preparation and beta deployment

• [x] Willem: Put the production Component Registry in read-only mode
• [x] Willem: Send a dump of the production database to Menzo: beta-vlo-clarin:/home/menwin/compreg.pre.cmdi1.2.rollout.sql.bz2
• [x] Menzo: Check for newly defined implicit cardinalties and fix if needed (Menzo can generate a fix.sql file on basis of the dump)
  • ​compreg.pre.cmdi1.2.rollout.fix.sql (direct link)
  • [x] Willem: Apply the database fix script (if needed) in production
• [x] Willem: Make a new dump of the production database and transfer this to the Component Registry beta host (dev-sp.clarin.eu) + send a copy to Twan
• [x] Twan: Run a 'dry' component conversion on a dump of the Component Registry database to see if there are no new problematic cases
• [x] Willem: Replace the database on the Component Registry beta host with the fresh dump
• [x] Willem: Run the upgrade tool (see below for details) on beta and make sure that it terminates without any errors
  • in case of errors, most likely one or more components need to be fixed
• [x] Willem: Deploy and start the Component Registry version with support for CMDI 1.2 on beta

Production update and deployment:

• [x] Willem: Stop the production Component Registry
• Convert of all component and profile definitions within the production Component Registry with the component upgrade tool:
  • [x] Willem: Run the upgrade tool (see below for details) and make sure that it terminates without any errors
    • in case of errors, most likely one or more components need to be fixed
  • [x] Willem: Make a backup of the freshly converted database
• [x] Willem: Deploy and start the Component Registry version with support for CMDI 1.2 on production

VLO and harvester

On beta:

• [x] Willem: Deploy the VLO on beta
• [x] Willem: Put the prepared harvest results on beta
  • [x] Willem: (re)move /srv/vlo-data/clarin/*
  • [x] Willem: cd /srv/vlo-data/clarin/; tar xfj clarin.tar.bz2
  • [x] Willem: cd /srv/vlo-data/clarin/results; mv cmdi cmdi-1_1; mv cmdi-1_2 cmdi
  • [x] Willem: (re)move /srv/vlo-data/others/*
  • [x] Willem: cd /srv/vlo-data/others/; tar xfj others.tar.bz2
  • [x] Willem: cd /srv/vlo-data/others/; tar xfj others-sil.tar.bz2 (should merge with previously created dirs)
  • [x] Willem: cd /srv/vlo-data/others/results; mv cmdi cmdi-1_1; mv cmdi-1_2 cmdi
• [x] Willem: Run an import on beta

On production:

• [x] Willem: Transfer the Solr index from beta and the prepared harvest results to production in order to 'bootstrap' the VLO
  • [x] Willem: cd /srv/vlo-data/work/vu/results; mv cmdi cmdi-1_1; mv cmdi-1_2 cmdi
• [x] Willem: Deploy the OAI harvester with support for CMDI 1.2, configure harvest result to CMDI 1.2
• [x] Willem: Deploy the VLO version with support for CMDI 1.2
• [x] Willem: Disable the Meertens harvest cron job (for now it's included again in the main CLARIN harvest)

Testing

• [x] Willem: Start a VLO import manually
• [x] Willem: Run a harvest with conversion to CMDI 1.2 (manually or depend on cronjob)
• [x] Willem: Run a new VLO import (manually or depend on cronjob)

Next day:

• [x] Menzo: Validate harvest results (CMDI 1.1 and CMDI 1.2)
  • ​catalog-cmdi-1_1-validation-20160705.log (direct link; CMDI 1.1 validation of the 4 July CLARIN harvest against the CR before the update)
    validation result: 18.27% failure rate (files: 760387 total, 621496 passed, 138891 failed; 12.03 GB total, 1410 files/second, 22.85 MB/second)
  • ​alpha-cmdi-1_1-validation-20160705.log (direct link; CMDI 1.1 validation of the 4 July CLARIN harvest against the CR on the alpha server)
    validation result: 18.27% failure rate (files: 760387 total, 621496 passed, 138891 failed; 12.03 GB total, 1206 files/second, 19.55 MB/second)
  • ​catalog-cmdi-1_1-validation-20160706.log (direct link; CMDI 1.1 validation of the 4 July CLARIN harvest against the CR after the update)
    validation result: 18.27% failure rate (files: 760387 total, 621496 passed, 138891 failed; 12.03 GB total, 812 files/second, 13.16 MB/second)
  • ​analysis of the differences between 1.1 and 1.2 validation
• [x] Twan: Validate import results

Announcements and documentation

• [x] Make a draft version of the specification available as PDF
  • ​CMDI-1_2-specification-draft-20160706.pdf
• [x] Make an update version of the executive summary available
• [x] Update ​CMDI and ​CMDI 1.2 pages of the clarin.eu website
• [x] Make a news item/blog post about CMDI 1.2 pointing to sources of information and laying out the roadmap for further CMDI 1.2 infrastructure integration

Component conversion

A utility called the ​ComponentUpdater applies the upgrade stylesheet to all components in the Component Registry database and validates the output. If all components transform and validate successfully, it commits the changes.

A prepared (compiled and pre-configured) package will be made available to run on production at the time of rollout of CMDI 1.2.

Problematic components

The components (and profiles) listed by id below become invalid (with respect to schema or schematron rules) after conversion. They need to be adapted or removed before the migration.

Status: all changes applied to production database on 2016/07/13. Original components attached as retrieved on that day before applying changes.

Components that need to be fixed

• [x] clarin.eu:cr1:c_1280305685305
  • minimal cardinality > maximum cardinality
• [x] clarin.eu:cr1:c_1311927752346
  • duplicate (empty) enumeration items

Components that can be removed

These are all either unused components clearly created for test purposes or have been deleted by the user.

• [x] clarin.eu:cr1:p_1337778924929
• [x] clarin.eu:cr1:p_1288172614022
• [x] clarin.eu:cr1:p_1288172614024
• [x] clarin.eu:cr1:p_1290431694477
• [x] clarin.eu:cr1:p_1274880881900
• [x] clarin.eu:cr1:p_1317737532981
• [x] clarin.eu:cr1:c_1311927752348