wiki:Curation Module

Responsible for this page: ​Davor Ostojić.
Last content check: 03-11-2015
Status: design

Purpose

The purpose of this page is to collect relevant information about Curation Module project.

Project: Curation Module

The goal of this project is to implement software component for curation and quality assessment which can be integrated in the CLARINs VLO workflow. Project is initialized by Metadata Curation Task Force. Specification for the Curation Module is based on the Metadata Quality Assessement Service proposal. Curation Module will be able to validate and normalize single MD records, repositories and profiles, to assess their quality and to produce reports with different information for different actors in VLO workflow. For implementation this project will use some of the existing CLARIN components.

Contents

  1. Project: Curation Module
    1. People
    2. Getting code
    3. Usage
    4. System Requirements
    5. Dependencies
    6. Requirements
    7. Building and Deploying
    8. Design
    9. Tickets
    10. Status, Planning and Roadmap
    11. Resources
    12. History
    13. Meetings

People

Getting code

github: ​https://github.com/clarin-eric/clarin-curation-module

Usage

Web Application

url: ​https://clarin.oeaw.ac.at/curate/

REST API

instance:

profile:

CLI

To run it from the command line: java -cp curate.jar:path_to_maven_dependecies/* eu.clarin.cmdi.curation.main.Main

Parameters:

  • -config <file>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;a path to the configuration file, internal configuration file is used by default. This parameter is optional

input type:

  • -p to curate a profile
  • -i to curate an instance
  • -c to curate a collection

resource:

  • -path <path> - Space separated paths to file or folder to be curated
  • -url <url> - Space separated urls to profile or instance to be curated
  • -id <profilesId> - Space separated CLARIN profile IDs in format: clarin.eu:cr1:p_xxx

Allowed combinations are:

  • -p -path / url / id
  • -i -path / url
  • -c -path

System Requirements

Requirements for the project are based on Metadata Quality Assessement Service requirements

Identified Use Cases

Use Case 1 – Metadata Creator checks the validity of newly created record

  • Title: Check validity of metadata record
  • Actor: MD Creator
  • Level: User Goal
  • Main Success Scenario:
    1. User copies MD record into the web form and starts validation by clicking "Validate" button
    2. Module does schema validation, link checks, vocabulary check, facet coverage assessment
    3. User gets the report with status, eventual errors and assessment
    4. User gets instructions how to improve MD record (recommended profile, recommended values)

Use Case 2 – MD Modeler checks the quality of profiles

  • Title: Check quality of profile/schema
  • Actor: MD Modeler
  • Level: User Goal
  • Main Success Scenario:
    1. CMDI Editor runs the curation module and passes as argument profile or schema
    2. Module does link checks and facet coverage assessment
    3. User gets the report on links availability and facet coverage

Use Case 3 – Repository Admin checks quality of metadata in his repository

  • Title: Check overall quality of metadata in repository
  • Actor: Repository Admin
  • Level: User Goal
  • Main Success Scenario:
    1. Admin runs module from command line and passes as argument location containing MD records
    2. Module does quality assessment of the records
    3. Admin gets summarized report on overall quality of MD records in his repository

Use Case 4 – Curation Module in VLO workflow

  • Title: Use Case 4 – Curation Module in VLO workflow
  • Actor: VLO workflow
  • Level: Summary
  • Main Success Scenario:
    1. Curation Module is called before vlo-importer component with location where MD records are stored as argument
    2. Module does validation and normalization and generates different kinds of reports and normalized MD record
    3. VLO importer uses normalized records in post-processing phase and imports them into SOLR
    4. After importing script emails reports to VLO admin, MD Curators and data providers

Requirements

  • Curation module will be integrated in VLO workflow. It has to provide VLO with normalized MD records for further (post-) processing and ingestion in SOLR. Since there are already huge number of records (currently 800K) and this number grows every week, curation module has to be performable.
  • Ability to provide validation and / or assessment service for metadata creators (on instance level).
  • Curation module has to provide service for CMDI profile/schema assessment for for MD modeler.
  • Curation module must be able to work with single record and with collections in batch mode.
  • Local and remote invocation.
  • Implementation of curation workflow includes following steps: schema validation, URL inspection, value validation against vocabularies and normalization. (See Cmdi/QualityCriteria for full list of checks)
  • Feedback about errors, per record or per collection.
  • Feedback about quality: score, facet coverage.
  • Provision of instructions on MD improvement.
  • Time dimension, keep track of quality in time for analysis.

Dependencies

For implementation following projects will be used:

Requirements

  • java 8
  • maven 3
  • tomcat 7+

Building and Deploying

To build the application 

   mvn clean install

copy war file from curation-module-web into tomcats webapps folder

Design

Component Diagram:

Tickets

#676 - Create a metadata curation module

Status, Planning and Roadmap

Status: implementation

Resources

History

Meetings

​gdoc used for minutes and notes

telco 08/04/2016​

telco 28/04/2016​

Last modified 7 years ago Last modified on 05/09/17 14:35:06

Attachments (3)

Download all attachments as: .zip