wiki:CMDI 1.2/Rollout

Version 73 (modified by Twan Goosen, 8 years ago) (diff)

checked previously unchecked boxes

CMDI 1.2 Rollout

  1. CMDI 1.2 Rollout
    1. Rough schedule
    2. Detailed release workflow
      1. Preparation
        1. Toolkit
      2. Component Registry
      3. VLO and harvester
      4. Testing
      5. Announcements and documentation
    3. Component conversion
      1. Problematic components
        1. Components that need to be fixed
        2. Components that can be removed

To take place on 7 and 8 July 2016.

Also see the previously published ​rollout plan (which is slightly outdated in the details by now).

Rough schedule

  • Thursday
    • ~10:30~12:00 Component Registry preparation + beta
    • ~13:00~15:00 Component Registry production
    • ~15:00~17:00 VLO beta
  • Friday
    • ~9:00~12:00 VLO production
    • ~13:00~17:00 New harvest, testing and announcements

Detailed release workflow

Steps in order of execution

Preparation

Preparation steps can take place a couple of days before the deployment

Toolkit

One day before the deployment

Component Registry

Preparation and beta deployment

  • [x] Willem: Put the production Component Registry in read-only mode
  • [x] Willem: Send a dump of the production database to Menzo: beta-vlo-clarin:/home/menwin/compreg.pre.cmdi1.2.rollout.sql.bz2
  • [x] Menzo: Check for newly defined implicit cardinalties and fix if needed (Menzo can generate a fix.sql file on basis of the dump)
  • [x] Willem: Make a new dump of the production database and transfer this to the Component Registry beta host (dev-sp.clarin.eu) + send a copy to Twan
  • [x] Twan: Run a 'dry' component conversion on a dump of the Component Registry database to see if there are no new problematic cases
  • [x] Willem: Replace the database on the Component Registry beta host with the fresh dump
  • [x] Willem: Run the upgrade tool (see below for details) on beta and make sure that it terminates without any errors
    • in case of errors, most likely one or more components need to be fixed
  • [x] Willem: Deploy and start the Component Registry version with support for CMDI 1.2 on beta

Production update and deployment:

  • [x] Willem: Stop the production Component Registry
  • Convert of all component and profile definitions within the production Component Registry with the component upgrade tool:
    • [x] Willem: Run the upgrade tool (see below for details) and make sure that it terminates without any errors
      • in case of errors, most likely one or more components need to be fixed
    • [x] Willem: Make a backup of the freshly converted database
  • [x] Willem: Deploy and start the Component Registry version with support for CMDI 1.2 on production

VLO and harvester

On beta:

  • [x] Willem: Deploy the VLO on beta
  • [x] Willem: Put the prepared harvest results on beta
    • [x] Willem: (re)move /srv/vlo-data/clarin/*
    • [x] Willem: cd /srv/vlo-data/clarin/; tar xfj clarin.tar.bz2
    • [x] Willem: cd /srv/vlo-data/clarin/results; mv cmdi cmdi-1_1; mv cmdi-1_2 cmdi
    • [x] Willem: (re)move /srv/vlo-data/others/*
    • [x] Willem: cd /srv/vlo-data/others/; tar xfj others.tar.bz2
    • [x] Willem: cd /srv/vlo-data/others/; tar xfj others-sil.tar.bz2 (should merge with previously created dirs)
    • [x] Willem: cd /srv/vlo-data/others/results; mv cmdi cmdi-1_1; mv cmdi-1_2 cmdi
  • [x] Willem: Run an import on beta

On production:

  • [x] Willem: Transfer the Solr index from beta and the prepared harvest results to production in order to 'bootstrap' the VLO
    • [x] Willem: cd /srv/vlo-data/work/vu/results; mv cmdi cmdi-1_1; mv cmdi-1_2 cmdi
  • [x] Willem: Deploy the OAI harvester with support for CMDI 1.2, configure harvest result to CMDI 1.2
  • [x] Willem: Deploy the VLO version with support for CMDI 1.2
  • [x] Willem: Disable the Meertens harvest cron job (for now it's included again in the main CLARIN harvest)

Testing

  • [x] Willem: Start a VLO import manually
  • [x] Willem: Run a harvest with conversion to CMDI 1.2 (manually or depend on cronjob)
  • [x] Willem: Run a new VLO import (manually or depend on cronjob)

Next day:

  • [x] Menzo: Validate harvest results (CMDI 1.1 and CMDI 1.2)
  • [x] Twan: Validate import results

Announcements and documentation

Component conversion

A utility called the ​ComponentUpdater applies the upgrade stylesheet to all components in the Component Registry database and validates the output. If all components transform and validate successfully, it commits the changes.

A prepared (compiled and pre-configured) package will be made available to run on production at the time of rollout of CMDI 1.2.

Problematic components

The components (and profiles) listed by id below become invalid (with respect to schema or schematron rules) after conversion. They need to be adapted or removed before the migration.

Status: all changes applied to production database on 2016/07/13. Original components attached as retrieved on that day before applying changes.

Components that need to be fixed

  • [x] clarin.eu:cr1:c_1280305685305
    • minimal cardinality > maximum cardinality
  • [x] clarin.eu:cr1:c_1311927752346
    • duplicate (empty) enumeration items

Components that can be removed

These are all either unused components clearly created for test purposes or have been deleted by the user.

  • [x] clarin.eu:cr1:p_1337778924929
  • [x] clarin.eu:cr1:p_1288172614022
  • [x] clarin.eu:cr1:p_1288172614024
  • [x] clarin.eu:cr1:p_1290431694477
  • [x] clarin.eu:cr1:p_1274880881900
  • [x] clarin.eu:cr1:p_1317737532981
  • [x] clarin.eu:cr1:c_1311927752348

Attachments (3)

Download all attachments as: .zip