wiki:InfrastructureOverview

Table of Contents

  1. 1. Technology
    1. 1.1. Gateway applications
      1. 1.1.1. Virtual Language Observatory
      2. 1.1.2. Federated Content Search engine
      3. 1.1.3. Virtual Collection Registry
    2. 1.2. Infrastructure applications
      1. 1.2.1. Centre Registry
      2. 1.2.2. Metadata
        1. 1.2.2.1. Metadata Harvester
        2. 1.2.2.2. Component Registry
        3. 1.2.2.3. Concept Registry
        4. 1.2.2.4. Curation module
      3. 1.2.3. Federated Identity
        1. 1.2.3.1. SAML metadata aggregation: PyFF
        2. 1.2.3.2. CLARIN Identity Provider
        3. 1.2.3.3. Discovery Service
        4. 1.2.3.4. Authorisation Service
      4. 1.2.4. Matomo (formerly Piwik)
      5. 1.2.5. Validators
        1. 1.2.5.1. OAI-PMH (metadata) validator
        2. 1.2.5.2. CMDI (metadata) validator
        3. 1.2.5.3. SRU-CQL (FCS) validator
      6. 1.2.6. Language Resource Switchboard
    3. 1.3. Software development
      1. 1.3.1. Subversion repository
      2. 1.3.2. Trac
      3. 1.3.3. Nexus maven repository
      4. 1.3.4. Docker repository
      5. 1.3.5. Third-party infrastructure (hosted in the cloud)
        1. 1.3.5.1. GitHub
        2. 1.3.5.2. Travis
    4. 1.4. System management
      1. 1.4.1. Icinga
      2. 1.4.2. Grafana
      3. 1.4.3. Kibana
    5. 1.5. Office systems
      1. 1.5.1. Document (CE) archive
      2. 1.5.2. People and Organisation database
    6. 1.6. Communication systems
      1. 1.6.1. Website
      2. 1.6.2. Mailing Lists
      3. 1.6.3. MailChimp
      4. 1.6.4. Slack
      5. 1.6.5. Social media
        1. 1.6.5.1. Facebook
        2. 1.6.5.2. Twitter
        3. 1.6.5.3. LinkedIn
        4. 1.6.5.4. YouTube
    7. 1.7. Third-party infrastructure applications & services
      1. 1.7.1. B2DROP [EUDAT]
      2. 1.7.2. B2SHARE [EUDAT]
      3. 1.7.3. B2SAFE [EUDAT]
      4. 1.7.4. B2ACCESS [EUDAT]
      5. 1.7.5. Potential A-services

1. Technology

1.1. Gateway applications

1.1.1. Virtual Language Observatory

Metadata exploration web application, with faceted browser.

Own software. Stable.

Public info: ​http://www.clarin.eu/vlo Technical info: https://trac.clarin.eu/wiki/CmdiVirtualLanguageObservatory

1.1.2. Federated Content Search engine

Search engine to interact with the federated content search endpoints.

Own software. Stable.

Public info: ​https://www.clarin.eu/content/content-search Technical info: https://trac.clarin.eu/wiki/FCS%20Aggregator

1.1.3. Virtual Collection Registry

Web application to manage and publish digital bookmark sets (virtual collections).

Own software. Stable.

Public info: ​http://www.clarin.eu/content/virtual-collections Technical info: https://trac.clarin.eu/wiki/VirtualCollectionRegistry

1.2. Infrastructure applications

1.2.1. Centre Registry

Database with information about the CLARIN centres.

Own software. Stable.

Public info: ​https://www.clarin.eu/clarin-eric-datatables/centres Technical info: ​https://centres.clarin.eu/about

1.2.2. Metadata

1.2.2.1. Metadata Harvester

Application to harvest the metadata descriptions from centres.

Own software. Stable.

Public info: ​http://clarin.eu/harvester Technical info: https://trac.clarin.eu/wiki/OAIHarvester

1.2.2.2. Component Registry

Application to manage and create CLARIN-compliant metadata profiles.

Own software. Stable.

Public info: ​http://clarin.eu/componentregistry Technical info: https://trac.clarin.eu/wiki/ComponentRegistryAndEditor

1.2.2.3. Concept Registry

Application to manage and create CLARIN-compliant concept definitions

Existing software (OpenSKOS) from CatchPLUS, with some modifications. Stable.

Public info: ​http://www.clarin.eu/ccr Technical info: ​http://openskos.org/

1.2.2.4. Curation module

Application to detect and correct errors in CLARIN-compliant metadata descriptions.

Experimental, under development by OEAW

Technical info: ​https://github.com/acdh-oeaw/clarin-curation-module

1.2.3. Federated Identity

1.2.3.1. SAML metadata aggregation: PyFF

Application to merge technical information from national identity federations into 1 file.

Existing software from NORDUnet. Stable.

Technical info: ​https://github.com/leifj/pyFF

1.2.3.2. CLARIN Identity Provider

Application for the registration of accounts for those who do not have an academic Identity Provider.

Our instance of (​Unity IDM), which is also used by EUDAT for ​B2ACCESS. See ServiceProviderFederation/CLARIN IdP.

Public info: ​https://user.clarin.eu/ Technical info: ​http://www.clarin.eu/content/clarin-identity-provider

1.2.3.3. Discovery Service

Application to select an identity provider upon federated login.

Self-developed lightweight solution. Stable.

Technical info: ​http://www.clarin.eu/content/clarin-discovery-service

1.2.3.4. Authorisation Service

Middleware that allows a web application or service to perform user delegation: act on behalf of an authenticated user (e.g. writing to her/his protected workspace).

Experimental

Will be provided by ​Unity IDM

Technical info: OAuthDelegation, ​https://www.clarin.eu/content/user-delegation

1.2.4. Matomo (formerly Piwik)

Application that keeps usage statistics of various web applications and sites.

Third-party software (self hosted). Stable.

Technical info: https://trac.clarin.eu/wiki/AccessStatistics

1.2.5. Validators

1.2.5.1. OAI-PMH (metadata) validator

Application to check the validity of an OAI-PMH endpoint for metadata harvesting.

Third-party software (self hosted). Stable.

Public info: ​http://clarin.eu/oaivalidator Technical info: ​http://dl.cs.uct.ac.za/projects/re/

1.2.5.2. CMDI (metadata) validator

Command-line application to check the validity of large sets of CMDI files.

Own software. Stable.

Technical info: https://trac.clarin.eu/browser/CMDIValidator

1.2.5.3. SRU-CQL (FCS) validator

Application to check the validity of an SRU-CQL endpoint for content search. Own software.

Stable Public info: ​http://clarin.eu/oaivalidator Technical info: ​http://dl.cs.uct.ac.za/projects/re/

1.2.6. Language Resource Switchboard

Application to suggest web applications that can process a specific file type.

Experimental, under development by EKUT

Technical info: https://trac.clarin.eu/wiki/LanguageResourceSwitchboard

1.3. Software development

1.3.1. Subversion repository

Source code repository for self-developed software. Will be gradually migrated to GitHub.

Third-party software (self hosted)

Technical info: ​http://www.clarin.eu/svn

1.3.2. Trac

Issue tracker for software development and wiki. The issue tracker will be gradually migrated to GitHub, while the wiki will remain for internal information exchange.

Third-party software (self hosted)

Technical info: ​http://www.clarin.eu/svn

1.3.3. Nexus maven repository

Repository to share compiled Java packages (jars) and Maven files (poms).

Third-party software (self hosted).

Technical info: https://trac.clarin.eu/wiki/Nexus

1.3.4. Docker repository

Third-party software (self hosted). Stable.

Technical info: https://trac.clarin.eu/wiki/ServerConfig/docker

1.3.5. Third-party infrastructure (hosted in the cloud)

1.3.5.1. GitHub

Issue tracking and code repository.

Technical info: https://trac.clarin.eu/wiki/GitHub

1.3.5.2. Travis

Continuous Integration platform, tightly coupled with GitHub.

Technical info: https://trac.clarin.eu/wiki/GitHub#Travis

1.4. System management

1.4.1. Icinga

Monitoring system.

Third-party software (hosted at computing centre).

Technical info: SystemAdministration/Monitoring/Icinga

1.4.2. Grafana

Computing Resource Dashboard.

Third-party software (self hosted).

Technical info: SystemAdministration/Monitoring/collectd

1.4.3. Kibana

Log querying and visualisation (on top of ​Elasticsearch, with input from ​Fluentd)

Third-party software (self hosted).

Technical info: SystemAdministration/Monitoring/Kibana

1.5. Office systems

1.5.1. Document (CE) archive

Simple web application for the CLARIN document archive.

Own software. Stable.

Technical info: ​https://github.com/dietervu/ce-archive (private)

1.5.2. People and Organisation database

Lightweight Drupal-based CRM solution (​RedHen) to manage the CLARIN contacts.

Third-party software. Stable (self hosted).

1.6. Communication systems

1.6.1. Website

Drupal-based content management system.

Third-party software, with a few own additions. Stable.

Public info: ​http://www.clarin.eu/ Technical info: https://trac.clarin.eu/wiki/Webdev

1.6.2. Mailing Lists

Mailman system.

Third-party software, hosted at Utrecht University.

Public info: ​http://www.clarin.eu/content/mailing-lists Technical info: https://trac.clarin.eu/wiki/lists.clarin.eu

1.6.3. MailChimp

Newsflash distribution system.

Third-party software, hosted in the cloud.

Public info: ​http://www.clarin.eu/content/newsflash

1.6.4. Slack

Chat platform for developers. Experimental.

Third-party software, hosted in the cloud.

Public info: ​https://clarineric.slack.com Technical info: https://trac.clarin.eu/wiki/Slack

1.6.5. Social media

1.6.5.1. Facebook

​https://www.facebook.com/ClarinEric/

1.6.5.2. Twitter

​https://twitter.com/CLARINERIC

1.6.5.3. LinkedIn

​https://www.linkedin.com/company/9411460

1.6.5.4. YouTube

​https://www.youtube.com/channel/UCJPks1mzisqsS4NrBFKIWag

1.7. Third-party infrastructure applications & services

1.7.1. B2DROP [EUDAT]

Cloud-storage platform based on ownCloud, similar to DropBox. Also referred to (within CLARIN) as “workspaces”.

Public info: ​https://b2drop.eudat.eu/ Technical info: https://trac.clarin.eu/wiki/Workspaces

1.7.2. B2SHARE [EUDAT]

Simple data upload platform, “scientific youtube”, based on CERN’s Invenio repository.

Public info: ​https://b2share.eudat.eu/

1.7.3. B2SAFE [EUDAT]

Data replication service based on ​iRODS.

Public info: ​https://eudat.eu/services/b2safe Technical info: ​http://www.clarin.eu/event/2015/clarin-b2safe-workshop

1.7.4. B2ACCESS [EUDAT]

Authentication module (proxy Service Provider) for EUDAT services.

Public info: ​https://b2access.eudat.eu Technical info: ​https://eudat.eu/services/b2access

1.7.5. Potential A-services

To be extended with services provided by CLARIN centres; a first candidate list is available at ​http://www.clarin.eu/node/4002

Last modified 5 years ago Last modified on 01/02/19 09:46:13