wiki:ComponentRegistryAndEditor

Project: Component Registry

The Component Registry stores CMDI metadata components and profiles and offers a REST service to list, retrieve, store and modify them. The front end is a separate web application that acts as a client of the Component Registry REST service and displays available components and profiles in the registry in addition to offering an editing interface for the creation, modification and publication of components and profiles.

The Component Registry has means to keep itself coherent and consistent: check all offered components and keep an administration of the added and removed components. All components get a unique identifier wherewith other applications can identify it to the registry.

Components and profiles are specified and stored in a format specified by the ​CMD component schema. The registry offers profile schema's in the XSD format, which get generated from the profile specifications on the fly using the ​comp2schema XSLT.

Documentation for the new Javascript based front end can be found at its ​GitHub wiki pages.

Subpages

Contents

  1. Project: Component Registry
    1. Subpages
    2. Contents
    3. People
    4. Getting code
    5. System Requirements
    6. Overview
      1. Development and deployment process
    7. Milestones and Planning
    8. Building and Deploying
      1. Overview
      2. Building
      3. Deployment
      4. Dependencies
    9. Management
      1. Administration interface
      2. Managing accounts
      3. Managing user teams
    10. Design
    11. Tickets
    12. History
      1. Historical pages

People

  • Project owner: Twan Goosen

People who have worked on the Component Registry and Editor in the past:

  • Patrick Duin (original developer)
  • Jean-Charles Ferrières (developer back end)
  • George Georgovassilis (developer back end)
  • Olha Shkaravska (developer front/back)
  • Mitchell Seaton (React front end prototype)

Getting code

The source is available from the two CLARIN GitHub repositories: ​back end (REST) and ​front end.

System Requirements

To serve:

  • Docker

OR

  • Java 1.7
  • Tomcat 7
  • PostgreSQL server

To use the front-end:

  • Web browser with Javascript support
    • Tested with Firefox 70.0, Google Chrome 78.0, Safari 9.0

To build:

  • Docker

OR

  • Java 11
  • Maven 3.6+

Overview

Front end project          Back end project
       ^ |                     | ^
       | +-----*GitHub*--------+ |
       |        /\  |            |
       |        \/  | (Deploy    |
       |   TravisCI-+->package)  |
       |            |            |
Front end docker    +---->Back end docker
         |     ^           ^       |
         |     |           |       |
         |     Compose project     |
         |         |       +-DB    |
         |         |       +-Proxy |
         |         *               |
         +-------*GitLab*----------+
                  |    |            
          GitLab CI -> Docker Registry
                              ^         
                              |         
      Deployed project----Upstream server---Deploy & control tools         
          (compose)           ^         
                              |         
                          Central proxy         
                              ^         
                              |         
                            Client

Development and deployment process

  • Feature implementations and fixes are assigned to milestones (​back end, ​front end); one milestone per (minor) version
  • Active development happens in the development branch and/or specific feature/issue branches in the GitHub repositores (​back end and ​front end)
  • As soon as the items on the version milestone have been reached, a release branch is created (e.g. 2.4.0) and minor version number in trunk is bumped
  • The branched version gets deployed to a testing server and is tested according to a test plan.
  • Once testing has completed successfully, the new version gets deployed to the production server (catalog.clarin.eu, see SystemAdministration for info and contacts).
  • The deployed version gets tagged and the master branch is forwarded
  • Development for the next minor version continues in a development branch ahead of master
  • Maintenance releases may be done to the current production branch (leading to e.g. 2.4.1)

Deployment is carried out on basis of Docker images derived from the back end and front end project. See Deployment section below for details.

Milestones and Planning

See ​back end and ​front end milestones on GitHub.

Building and Deploying

Overview

CLARIN infrastructure work flow: 

Front end source ------------------> Docker image
                                            |
                                            |
                                            +--> Docker compose project
                                            |          ^        ^
                                            |          |        |
Back end source --> Travis build --> Docker image     Database   Proxy

Building

There are two sets of separate projects with different build procedures:

  1. REST back end
    • Maven based build of REST service
      • No strict need to build this locally, as the ​Travis project builds and also takes care of deploying tagged commits to GitHub
      • The sources come with a build.sh script that will build the entire project in a Docker container with the right versions of Java and Maven installed, so no need to have those installed locally
    • Docker image ​docker-component-registry-rest which makes an image based on a compiled REST project (typically from a Travis tag build deployed to GitHub)
  2. React front end
    • npm/webpack based build of React.js front end
      • No strict need to build this locally, as the build is also carried out as part of the Docker image build (see below)
    • Docker image ​docker-component-registry-front-end which is based on a multistage build that acts on the sources of the front end project

Note that the front end and back are built completely independently. In previous versions the back end would include a compiled front end but this is no longer the case. Both have to be deployed separately, and the front end must be configured to find the REST service of the back end at a suitable location. The best way to achieve this is using the dedicated Docker Compose project (see Deployment section).

Deployment

The front end and back end need to be deployed and configured separately.

Deployment normally should happen using the Docker Compose project ​compose_compreg, which can be configured for a specific environment using environment variables (.env file). See the Compose project's documentation for details on the configuration.

Currently Deployed Locations

Dependencies

Internal dependency: the REST service depends on the front end at build time. The compiled front end page is included in the WAR package through the maven-dependency-plugin.

External dependencies: see REST service dependencies.

The Component Registry as a whole also depends on the ​CMDI toolkit, specifically the general component schema and the comp2schema XSLT.

Management

Administration interface

An administration interface that allows for inspection and modification of all public and private components is available at the path /admin.

E.g. for production: ​http://catalog.clarin.eu/ds/ComponentRegistry/admin].

Which accounts have administration rights is configured in the context parameter eu.clarin.cmdi.componentregistry.adminUsers, or in case of a Docker deployment via an environment variable (see ​compose project).

Managing accounts

As of version 2.2.2 of the Component Registry, user accounts can be managed through the administration interface. To edit the properties of a user (typically the , go to ​Manage accounts, then select the user from the drop down in the Manage existing account section and click Select. Be really careful, especially with the principal name as this is used to connect a user in the registry to its federated identity!

Transferring a user's data to another identity

Go to the Manage accounts page in the administration interface (see above).

  • See if the new identity exists in the Manage existing account dropdown
    • If so, select it and remote it.
    • If it cannot be removed because there already is data, change the principal name to something that does not exist
  • Find the old identity in the dropdown and select it
    • Change the principal name to the new principal name
    • Notice that the user ID does not change, hence all data is now linked to the new identity

Managing user teams

As of version 1.14.0 of the Component Registry, team (aka 'group') functionality is available. Teams can be managed, either through a JMX interface as described in ComponentRegistryAndEditor/ManagingGroups or through the Admin interface (select 'Manage teams' at ​http://catalog.clarin.eu/ds/ComponentRegistry/admin).

Creating a team

In the 'New team' form, fill in the team name, select an owner and press Create team.

Adding a user to a team

In the 'Teams' subsection, select the target team from the dropdown and press Select.

Then in the options for that team, select the target user from the dropdown list of all users. If the user is not in the list, ask the user to log in to the Component Registry, go to ​settings and submit their display name. After this the user should appear in the list with the provided display name. Alternatively you can create the user account yourself via the *Manage accounts* section of the admin interface iff you know the exact string (eppn or eptid) by which the user will be identified.

After selecting the target user, press Add member after which the user's name should appear in the members listing.

Removing a user from a team

Select the team from the dropdown, press Select, then simply click the red cross next to the user's name and/or identifier.

Design

For design details of the modules of this project, see their respective pages:

There is a description of the high level architecture (picture below).

High level overview of Component Registry

Tickets

List of all open tickets for this project:

Ticket Summary Priority Owner Milestone
#130 Importing component/profile should strip out components with componentId attributes. major Twan Goosen
#155 error message when not enough attributes to login major Twan Goosen
#164 Copy data category properties into component/element major Twan Goosen
#186 Transfer ownership of components major Dieter Van Uytvanck
#379 Throw exceptions from update method of ComponentRegistry interface major Twan Goosen
#428 Examine why ConcurrentRestServiceTest does not fail major Twan Goosen
#735 Data category should be searchable in the CCR major menzo.windhouwer@di.huc.knaw.nl
#147 Add a parameter to link to the components/profiles tab minor Twan Goosen
#221 add a favicon for the component registry minor Twan Goosen

History

The Component Registry has been developed as part of the CLARIN project.

People who have worked on this project include (in chronological order) Patrick Duin (original author), Twan Goosen, Jean-Charles Ferrières, Olha Shkaravska and George Georgovassilis

The original implementation of the backend (REST service) was using file system storage for the users and their registries. As of version 1.8 this was replaced by a database based implementation. The code still reflects the original, more hierarchical storage model (e.g. a registry per user even though all components are stored in the same table).

For a detailed version history, see the ​changelog.

Historical pages

Last modified 4 years ago Last modified on 11/15/19 09:56:59