CMDI 1.2 Rollout

Planned to take place on 7 July 2016.

Also see the previously published ​rollout plan (which is slightly outdated in the details by now).

Detailed release workflow

Steps in order of execution

Preparation

Preparation steps can take place a couple of days before the deployment

• [x] Twan: Prepare component conversion, i.e. fix/remove problematic components and profiles
• [x] Willem/Menzo: Fix specifications with implicit cardinalities
• [] Twan: Make a deployment package for the VLO
• [] Twan: Make a deployment package for the Component Registry
• [] Twan: Make a runnable package with ready to use configuration file for the conversion tool
• [] Menzo: Make a deployment package for the harvester

VLO and harvester

In theory, these steps take place some days before the deployment if the schedule allows for it (it depends on an admin being available)

At least one day before:

• [] Willem: Deploy the harvester and VLO on beta
• [] Willem: Run a full harvest and import on beta

On the day itself:

• [] Menzo: Validate existing harvest result (with CMDI 1.1) as a reference. Store validation results.
• [] Willem: Transfer the harvest results and Solr index from beta to production to 'bootstrap' the VLO
• [] Willem: Deploy the OAI harvester with support for CMDI 1.2, configure harvest result to CMDI 1.2
  • Configure to use alpha Component Registry until deployment of production Component Registry
• [] Willem: Deploy the VLO version with support for CMDI 1.2
  • Configure to use alpha Component Registry until deployment of production Component Registry

Toolkit

• [] Menzo: Change ​the default Component Registry location from the alpha server to the production server
• [] Menzo: Tag/release toolkit on GitHub, make development branch (master should be stable)
• [] Menzo: Check if the toolkit is available and up-to-date at ​https://infra.clarin.eu/CMDI/1.x

Component Registry

• [] Willem: Put the Component Registry in read-only mode
• [] Willem/Menzo: Check for newly defined implicit cardinalties and fix if needed (Menzo can generate a fix.sql file on basis of a db dump)
• [] Twan: Run a 'dry' component conversion on a dump of the Component Registry database to see if there are no new problematic cases
• [] Willem: Stop the Component Registry
• Convert of all component and profile definitions within the Component Registry with the component upgrade tool:
  • [] Willem: Backup the Component Registry database
  • [] Willem: Run the upgrade tool (see below for details) and make sure that it terminates without any errors
    • in case of errors, most likely one or more components need to be fixed
  • [] Willem: Make a backup of the freshly converted database
• [] Willem: Deploy and start the Component Registry version with support for CMDI 1.2
• [] Willem: Reconfigure VLO and harvester to use production Component Registry

Testing

• [] Menzo: Validate existing harvest result (with CMDI 1.1) to see that CMDI 1.1 records still validate (the same way). Compare to pre-deployment validation results.
• [] Willem: Run a harvest with conversion to CMDI 1.2
• [] Menzo: Validate harvest results (CMDI 1.1 and CMDI 1.2)
• [] Willem: Run a VLO import
• [] Twan: Validate import results

Component conversion

A utility called the ​ComponentUpdater applies the upgrade stylesheet to all components in the Component Registry database and validates the output. If all components transform and validate successfully, it commits the changes.

A prepared (compiled and pre-configured) package will be made available to run on production at the time of rollout of CMDI 1.2.

Problematic components

The components (and profiles) listed by id below become invalid (with respect to schema or schematron rules) after conversion. They need to be adapted or removed before the migration.

Status: all changes applied to production database on 2016/07/13. Original components attached as retrieved on that day before applying changes.

Components that need to be fixed

• [x] clarin.eu:cr1:c_1280305685305
  • minimal cardinality > maximum cardinality
• [x] clarin.eu:cr1:c_1311927752346
  • duplicate (empty) enumeration items

Components that can be removed

These are all either unused components clearly created for test purposes or have been deleted by the user.

• [x] clarin.eu:cr1:p_1337778924929
• [x] clarin.eu:cr1:p_1288172614022
• [x] clarin.eu:cr1:p_1288172614024
• [x] clarin.eu:cr1:p_1290431694477
• [x] clarin.eu:cr1:p_1274880881900
• [x] clarin.eu:cr1:p_1317737532981
• [x] clarin.eu:cr1:c_1311927752348