CMDI 1.2 Rollout

Planned to take place on 7 July 2016.

Also see the previously published ​rollout plan (which is slightly outdated in the details by now).

Detailed release workflow

Steps in order of execution

Preparation

Preparation steps can take place a couple of days before the deployment

• [] Twan: Announce downtime on 7/8 July
• [x] Twan: Prepare component conversion, i.e. fix/remove problematic components and profiles
• [x] Willem/Menzo: Fix specifications with implicit cardinalities
• [] Twan: Make a deployment package for the VLO (ultimately 6 July)
• [] Twan: Make a deployment package for the Component Registry (ultimately 6 July)
• [] Twan: Make a runnable package with ready to use configuration file for the conversion tool
• [] Menzo: Make a deployment package for the harvester (ultimately 6 July)

• [] Menzo: Run 'Others' harvest on alpha, package results for deployment
• [] Menzo: Run 'Clarin' harvest on alpha, package results for deployment

Toolkit

• [] Menzo: Change ​the default Component Registry location from the alpha server to the production server
• [] Menzo: Tag/release toolkit on GitHub, make development branch (master should be stable)
• [] Menzo: Check if the toolkit is available and up-to-date at ​https://infra.clarin.eu/CMDI/1.x

Component Registry

• [] Willem: Put the Component Registry in read-only mode
• [] Willem/Menzo: Check for newly defined implicit cardinalties and fix if needed (Menzo can generate a fix.sql file on basis of a db dump)
• [] Twan: Run a 'dry' component conversion on a dump of the Component Registry database to see if there are no new problematic cases
• [] Willem: Stop the Component Registry
• Convert of all component and profile definitions within the Component Registry with the component upgrade tool:
  • [] Willem: Backup the Component Registry database
  • [] Willem: Run the upgrade tool (see below for details) and make sure that it terminates without any errors
    • in case of errors, most likely one or more components need to be fixed
  • [] Willem: Make a backup of the freshly converted database
• [] Willem: Deploy and start the Component Registry version with support for CMDI 1.2

VLO and harvester

On beta:

• [] Willem: Deploy the harvester and VLO on beta
• [] Willem: Put the prepared harvest results on beta
• [] Willem: Run an import on beta

On production:

• [] Menzo: Validate existing harvest result (with CMDI 1.1) as a reference. Store validation results.
• [] Willem: Transfer the harvest results and Solr index from beta to production to 'bootstrap' the VLO
• [] Willem: Deploy the OAI harvester with support for CMDI 1.2, configure harvest result to CMDI 1.2
• [] Willem: Deploy the VLO version with support for CMDI 1.2

Testing

• [] Menzo: Validate existing harvest result (with CMDI 1.1) to see that CMDI 1.1 records still validate (the same way). Compare to pre-deployment validation results.
• [] Willem: Run a harvest with conversion to CMDI 1.2 (manually or depend on cronjob)
• [] Willem: Run a VLO import (manually or depend on cronjob)

Next day:

• [] Menzo: Validate harvest results (CMDI 1.1 and CMDI 1.2)
• [] Twan: Validate import results

Announcements and documentation

• [] Make a draft version of the specification available as PDF
• [] Make an update version of the executive summary available
• [] Update ​CMDI and ​CMDI 1.2 pages of the clarin.eu website
• [] Make a news item/blog post about CMDI 1.2 pointing to sources of information and laying out the roadmap for further CMDI 1.2 infrastructure integration

Component conversion

A utility called the ​ComponentUpdater applies the upgrade stylesheet to all components in the Component Registry database and validates the output. If all components transform and validate successfully, it commits the changes.

A prepared (compiled and pre-configured) package will be made available to run on production at the time of rollout of CMDI 1.2.

Problematic components

The components (and profiles) listed by id below become invalid (with respect to schema or schematron rules) after conversion. They need to be adapted or removed before the migration.

Status: all changes applied to production database on 2016/07/13. Original components attached as retrieved on that day before applying changes.

Components that need to be fixed

• [x] clarin.eu:cr1:c_1280305685305
  • minimal cardinality > maximum cardinality
• [x] clarin.eu:cr1:c_1311927752346
  • duplicate (empty) enumeration items

Components that can be removed

These are all either unused components clearly created for test purposes or have been deleted by the user.

• [x] clarin.eu:cr1:p_1337778924929
• [x] clarin.eu:cr1:p_1288172614022
• [x] clarin.eu:cr1:p_1288172614024
• [x] clarin.eu:cr1:p_1290431694477
• [x] clarin.eu:cr1:p_1274880881900
• [x] clarin.eu:cr1:p_1317737532981
• [x] clarin.eu:cr1:c_1311927752348