Changeset 2696 for SMC4LRT/chapters/Infrastructure.tex

Timestamp:

03/13/13 18:06:34 (11 years ago)

Author:

vronk

Message:

added notes from Menzo about DCR CLAVAS interaction

File:

• SMC4LRT/chapters/Infrastructure.tex (modified) (6 diffs)

SMC4LRT/chapters/Infrastructure.tex

@@ -1 @@
-\chapter{Underlying infrastructure}\label{ch:components}
-
-As stated before, the proposed module is part of CMDI and depends on multiple modules of the infrastructure. Before we describe the interaction itself in chapter \ref{method}, we introduce in short these modules and the data they provide:
+\chapter{Underlying infrastructure}
+\label{ch:components}
+
+
+\section{CLARIN / CMDI}
+
+CLARIN - Common Language Resource and Technology Infrastructure - constituted by over 180 members from round 38 countries. The mission of this project is to
+
+\begin{quotation}
+    create a research infrastructure that makes language resources and technologies (LRT) available to scholars of all disciplines, especially SSH large-scale pan-European collaborative effort to create, coordinate and make language resources and technology available and readily useable.
+\end{quotation}
+
+The infrastructure foresees a federated network of centers (with federated identity management) but mainly providing resources and services in an agreed upon / coherent / uniform / consistent /standardized manner. The foundation for this goal shall be the Common or Component Metadata infrastructure, a model that caters for flexible metadata profiles, allowing to accommodate existing schemas.
+
+
+
+As stated before, the SMC is part of CMDI and depends on multiple modules of the infrastructure. Before we describe the interaction itself in chapter \ref{method}, we introduce in short these modules and the data they provide:
 
 \begin{itemize}
@@ -20 +34 @@
 \end{figure*}
 
-\subsection{CMDI - Production side}
+\subsection{CMDI - DCR/CR/RR}
+\label{dcr}
 
 The \emph{Data Category Registry} (DCR) is a central registry that enables the community to collectively define and maintain a set of relevant linguistic data categories. The resulting commonly agreed controlled vocabulary is the cornerstone for grounding the semantic interpretation within the CMD framework. 
@@ -26 +41 @@
 %Next to a web interface for users to browse and manage the data categories, DCR provides a REST-style webservice allowing applications to access the information (provided in Data Category Interchange Format - DCIF). The data categories are assigned a persistent identifier, making them globally and permanently referenceable.
 
-The \emph{Component Metadata Framework} (CMD) is built on top of the DCR and complements it. While the DCR defines the atomic concepts, within CMD the metadata schemas can be constructed out of reusable components - collections of metadata fields. The components can contain other components, and they can be reused in multiple profiles as long as each field "refers via a PID to exactly one data category in the ISO DCR, thus indicating unambiguously how the content of the field in a metadata description should be interpreted" \cite{Broeder+2010}. This allows to trivially infer equivalencies between metadata fields in different CMD-based schemas. While the primary registry used in CMD is the ISOcat DCR, other authoritative sources for data categories ("trusted registries") are accepted, especially Dublin Core Metadata Initiative \cite{DCMI:2005}.
+The \emph{Component Metadata Framework} (CMD) is built on top of the DCR and complements it. While the DCR defines the atomic concepts, within CMD the metadata schemas can be constructed out of reusable components - collections of metadata fields. The components can contain other components, and they can be reused in multiple profiles as long as each field ``refers via a PID to exactly one data category in the ISO DCR, thus indicating unambiguously how the content of the field in a metadata description should be interpreted'' \cite{Broeder+2010}. This allows to trivially infer equivalencies between metadata fields in different CMD-based schemas. While the primary registry used in CMD is the ISOcat DCR, other authoritative sources for data categories (``trusted registries'') are accepted, especially Dublin Core Metadata Initiative \cite{DCMI:2005}.
 % \emph{Component Registry} implements the Component Data Model and allows to define, maintain and publish CMD-components and -profiles.
 
@@ -44 +59 @@
 !Cf. Erhard Hinrichs 2009
 
-And a last relevant intiative to mention is that of a \texttt{Vocabulary Alignment Service} being developed and run within the Dutch program CATCH\footnote{\textit{Continuous Access To Cultural Heritage} - \url{http://www.catchplus.nl/en/}}, which serves as a neutral manager and provider of controlled vocabularies. There are plans to reuse or enhance this service for the needs of the CLARIN project.
-
 \noindent
 All these components are running services, that this work shall directly build upon.
@@ -53 +66 @@
 
 Consequently, the infrastructure also foresees a dedicated module, \emph{Semantic Mapping}, that exploits this novel mechanism to deliver correspondences between different metadata schemas. The details of its functioning and its interaction with the aforementioned modules is described in the following chapter \ref{method}.
+
+\subsection{Vocabulary Service / Reference Data Registry}
+
+\subsubsection{Motivation \& related activities in the community}
+The urgent need for reliable community-shared registry services for concepts, controlled vocabularies and reference data for both the LRT and Digital Humanities community has been discussed on many occasions in various contexts. Applications and tasks requiring or profiting from this kind of service comprise Data-Enrichment / Annotation, Metadata Generation, Curation, Data Analysis, etc. As there is a substantial overlap in the vocabularies relevant for the various communities and even more so a high potential for reusability on the technical level, there is a strong case for tight cooperation between different initiatives. 
+
+In the context of the CLARIN initiative, one activity to tackle this issue -- mainly driven by CLARIN-NL -- is the project/taskforce \emph{CLAVAS - Vocabulary Alignment Service for CLARIN} where the plan is to reuse and enhance for CLARIN needs a SKOS-based  vocabulary repository and editor OpenSKOS\furl{http://openskos.org}, developed and run within the dutch program CATCHplus\footnote{\textit{Continuous Access To Cultural Heritage} - \url{http://www.catchplus.nl/en/}}. See below for a more detailed description of this system. As of spring 2013, the Standing Committee on CLARIN Technical Centers (SCCTC) adopted the issue of Controlled Vocabularies and Concept Registries as one of the infrastructural (A-center) services to be dealt with.
+
+In parallel, within the sister ESFRI project DARIAH a taskforce with the same goal has been set up : \emph{Service for Reference Data and Controlled Vocabularies}. This taskforce was introduced at the 2nd VCC Meeting in Vienna in November 2012. It is conceived as a collaborative endeavor between VCC1/Task 5: Data federation and interoperability and VCC3/Task3: Reference Data Registries (and external partners). The main goal is to \emph{establish a service providing controlled vocabularies and reference data} for the DARIAH (and CLARIN) community.
+
+Regarding the responsibilities of the DARIAH working groups: 
+VCC3/Task 3 identifies and recommends vocabularies relevant for the community. VCC1/Task 5 provides basic/generic services relevant for whole community. Especially, the Schema Registry, that allows to express mappings between different schemas seems to be one starting point. In accordance with the VCC1 strategy, concentrate on pulling together (pooling) existing resources and only implement necessary ``glue'' to put the pieces together (data conversion, service-wrappers...)
+
+Thus there is a momentum and a high potential for a collaborative approach in at least these two big initiatives CLARIN and DARIAH, that serve a very wide-spread and diverse community.
+
+\subsubsection{Abstract service description}
+As to the service itself it is primarily meant to serve other applications, rather than being used directly by end users, but a basic user interface is still necessary for administration etc.  By using global semantic identifiers instead of strings, such a service enables the harmonization of metadata descriptions and annotations and is an indispensable step towards semantic data and \xne{LOD}. 
+Besides providing vocabularies, the service should also hold and expose equivalencies (and other relationships) between concepts from different vocabularies (concept schemes). These relationships come primarily from existing mappings, but can (and hopefully will) be subsequently generated (manually) for specific subsets on demand in a community process. An example for equivalencies from Wikipedia\footnote{\href{http://de.wikipedia.org/wiki/Johann_Wolfgang_von_Goethe}{page for J. W. Goethe}}:
+\begin{verbatim}
+GND: 118540238 | LCCN: n79003362 | NDL: 00441109 | VIAF: 24602065 | Wikipedia-Personensuche
+\end{verbatim}
+
+\subsubsection{Vocabulary Service - CLAVAS}
+As described in previous section (\ref{dcr}), a solid pilar for defining and maintaining data categories is the ISOcat data category registry. However, while ISOcat has been in productive use for some time, it is – by design – not usable for all kinds of reference data. In general, it suits well for defining concepts/data categories (with closed or open concept domains), but its complex data model and standardization workflow does not lend itself well to maintain “semi-closed” concept domains, controlled vocabularies, like lists of entities (e.g. organizations or authors). In such cases, the concept domain is not closed (new entities need to be added), but it is also not open (not any string is a valid entity). Besides, the domain may be very large (millions of entities) and has to be presumed changing (especially new entities being added).
+
+This shortcoming leads to a need for an additional registry/repository service for this kind of data (controlled vocabularies). Within the CLARIN project mainly the abovementioned taskforce \emph{CLAVAS} is concerned with this challenge.
+The foundation is the vocabulary repository and editor OpenSKOS\furl{http://openskos.org}.
+
+This repository can serve as a project independent manager and provider of controlled vocabularies.
+One important feature of the OpenSKOS system is its distributed nature. It allows individual instances to synchronize the maintained vocabularies among each other via OAI-PMH protocol. This caters for a reliable redundant system, as multiple instances would provide identical synchronized data, while the primary responsibility for individual vocabularies could lie with different instances/organizations based on their specialization, field of expertise.
+
+Currently, the Meertens Institute\furl{http://meertens.knaw.nl/} of the Dutch Royal Academy of Sciences (KNAW), as well as Netherlands Institute for Sound and Vision\furl{http://www.beeldengeluid.nl/} are running an instance of OpenSKOS.
+As the work on this vocabulary repository started in the context of a cultural heritage program, originally it served vocabularies not directly relevant for the LRT-community \emph{GTAA - Gemeenschappelijke Thesaurus Audiovisuele Archieven} or \emph{AAT - Art \& Architecture Thesaurus}\furl{http://openskos.org/api/collections}. As part of the process of adaptation to the needs of CLARIN and LRT-community data categories from \xne{ISOcat} have been converted into SKOS-format and ingested into the system.
+\xne{CLARIN Centre Vienna} is also running a prototypical instance of the OpenSKOS system with ISOcat data. 
+
+A plan has been developed/adopted to support further vocabularies relevant for the community.
+Following are those to be handled in short-term, in order of urgency/relevance/prirority: 
+\begin{itemize}
+\item the list of language codes\todo{url: ISO-639}
+\item country codes
+\item organization names for the domain of language resources
+\end{itemize}
+
+See \ref{refdata} for a more complete list of required reference data together with candidate existing vocabularies
+and \ref{dcr-skos} for discussion on mapping the information about data categories from ISOcat to \xne{SKOS}.
+
+\subsection{Interaction between DCR, VAS and client applications}
+
+
+In my view you do that in ISOcat by binding the constrained DC to the
+CLAVAS vocabulary, e.g., the constrained domain of /language ID/ (DC-2482)
+could look as follows:
+
+I think is no need to express the relationship between this constrained DC
+and the vocabulary in CLAVAS itself. Many DCs (or any other application
+using CLAVAS) can refer to the same CLAVAS vocabulary.
+
+
+See above for my reasoning. I don't think this information needs to be in
+CLAVAS. 
+I do think that ISOcat, CLAVAS, RELcat, an actual language
+resource all provide a part of the semantic network. 
+
+And if you can express these all in RDF, which we can for almost all of them (maybe
+except the actual language resource ... unless it has a schema adorned
+with ISOcat DC references ... < insert a SCHEMAcat plug ;-) >, but for
+metadata we have that in the CMDI profiles ...) you could load all the
+relevant parts in a triple store and do your SPARQL/reasoning on it. Well
+that's where I'm ultimately heading with all these registries related to
+semantic interoperability ... I hope ;-)
+
+
+Maybe I should add to this that I clearly see ISOcat as an user of CLAVAS,
+i.e., for constrained DCs. 
+
+However, ISOcat as a provider of vocabularies
+is less clear to me. Many of the value domains are small and CLAVAS is
+overkill. 
+
+Where the value domains are big (ISO 639-3) or can only be
+partially enumerated (organization names) ISOcat can't/shouldn't contain
+the value domains but just refer to CLAVAS, i.e., ISOcat wouldn't be a
+provider. 
+Still there are some closed DCs which might be good vocabulary
+providers, e.g., /linguistic subject/ (DC-2527/), and still also need to
+stay in ISOcat. I think at some point we should create a smaller set of
+metadata DCs to be harvested by CLAVAS. Hennie and I discussed this also
+somewhere last year ... I'll be a the Meertens on Thursday, maybe we can
+talk it over once more.
+
+
+>>
+
+I guess the discussion is about two different things:
+- how to specify that the range of some metadata property consists of Concepts from a specific ConceptScheme
+-> this can not be done in SKOS, but external schema definitions could refer to the URI of some (CLAVAS/OpenSKOS) ConceptScheme
+- how to specifiy relations between Concepts that are in different ConceptSchemes
+-> this can be done in SKOS using skos: exactMatch, closeMatch, broaderMatch, narrowerMatch, relatedMatch. OpenSKOS supports adding and searching these properties already, and the OpenSKOS editor also already has support for it.
+
+> - define them in a new clavas namespace and add the properties as a specialization to OpenSKOS, you consider them part of the vocabulary definition then
+> --> is a bit against the OpenSKOS 'philosophy' that OpenSKOS is a platform for SKOS, by definition.
+> - add them to your metadata schema or profile, your consider them as constraints on vocabulary usage for a given metadata field
+> --> this would be my preference
+> - add them to a definition in ISOcat, and let your metadata schema refer to ISOcat instead of OpenSKOS. ISOcat extends the OpenSKOS definition then.
+> --> leads to mixing of ISOcat and OpenSKOS, in semantic and technical ways. Not my preference.
+
+In what I propose ISOcat constrained DCs can refer to a CLAVAS vocabulary as a way to constrain (we stretch this a bit if a vocabulary is 'open', e.g., like organization names where it provides the preferred spelling of known organizations but you still have to be able to add new organization names). In ISOcat such constraints have the same status as, for example, the data type, which is that ISOcat just provides hints it has no way to enforce this. Look at CMDI where the CMDI elements refer to a ISOcat DC via a concept link but they may have a completely different data type. In an ideal world the Component Editor would take over the data type and the CLAVAS vocabulary from the linked DC specification. This way the reference to the CLAVAS vocabulary ends up in the CMD component/profile specification and the derived XSD, and can be used by tools that support CLAVAS, e.g., Arbil (well its in the planning).
+
+So although ISOcat refers to CLAVAS as a hint, the metadata schema is the final one that has the real CLAVAS vocabulary reference, i.e., no reference to CLAVAS via ISOcat. Hennie, I think that still meets your preference and prevents unwanted mixing.
+
 
 
@@ -72 +195 @@
 and \emph{Metadata Service} that provides search access to this body of data. As such, Metadata Service is the primary application to use Semantic Mapping, to optionally expand user queries before issuing a search in the Metadata Repository. \cite{Durco2011}
 
+
+\section{Content Repositories}
+Metadata is only one aspect of the availability of resources. It is the first step to announce and describe the resources. However it is of little value, if the resources themselves are not equally well accessible. Thus another pillar of the CLARIN infrastructure are Content Repositories - centres to ensure availability of resources.
+
+The requirements for these repositories: PIDs, CMD, OAI-PMH
+\todo{cite: center-B paper}
+
+\section{Distrbuted system - federated search}
+
+Metadata -> harvesting via OAI-PMH
+but Content search has to be really distributed.
+
+?
+\begin{description}
+\item[Z39.50/SRU/SRW/CQL] LoC
+\item[OAI-PMH]
+\end{description}