source: SMC4LRT/chapters/Data.tex @ 3671

\chapter{Analysis of the data landscape}
\label{ch:data}
This section gives an overview of existing standards and formats for metadata in the field of Language Resources and Technology together with a description of their characteristics and their respective usage in the initiatives and data collections. Special attention is paid to the Component Metadata Framework representing the base data model for the infrastructure this work is part of.


\section{Component Metadata Framework}
\label{def:CMD}

The \emph{Component Metadata Framework} (CMD) is the data model of the CLARIN Component Metadata Infrastructure. (See \ref{def:CMDI} for information about the infrastructure. The XML-schema defining CMD -- the \xne{general-component-schema} -- is featured in appendix \ref{lst:cmd-schema}.)
CMD is used to define the so-called \var{profiles} being constructed out of reusable \var{components} -- collections of metadata fields. The components can contain other components and they can be reused in multiple profiles. Profile itself is just a special kind of a component (a sub class), with some additional administrative information.
The actual core provision for semantic interoperability is the requirement, that each CMD element (i.e. metadata field) refers ``via a PID to exactly one data category\footnote{persistently referenceable concept definition} (cf. \ref{def:DCR}), thus 
indicating unambiguously how the content of the field in a metadata description should be interpreted'' \cite{Broeder+2010}. 

%This approach of integrating prerequisites for semantic interoperability directly into the process of metadata creation is fundamentally different from the traditional methods of schema matching that try to establish pairwise alignments between already existing schemas -- be it algorithm-based or by means of explicit manually defined crosswalks\cite{Shvaiko2005}. 

While the primary registry for data categories used in CMD is the \xne{ISOcat} Data Category Registry (cf. \ref{def:DCR}), other authoritative sources are accepted (so-called ``trusted registries''), especially the set of terms maintained by the Dublin Core Metadata Initiative \cite{DCMI:2005}.

Once the profiles are defined they are transformed into a XML-Schema, that prescribes the structure of the instance records.
The generated schema also conveys as annotation the information about the referenced data categories.


\subsection{CMD Profiles }
In the CR 124\footnote{All numbers are as of 2013-06 if not stated otherwise} public Profiles and 696 Components are defined. Table \ref{table:dev_profiles} shows the development of the CR and DCR population over time.

Next to the `native' CMD profiles a number of profiles have been created that implement existing metadata formats, like OLAC/DCMI-terms, TEI Header or the META-SHARE schema. The resulting profiles proof the flexibility/expressi\-vi\-ty of the CMD metamodel. The individual profiles differ also very much in their structure -- next to flat profiles with just one level of components or elements with 5 to 20 fields (\textit{dublincore}, \textit{collection}, the set of \textit{Bamdes}-profiles) there are complex profiles with up to 10 levels (\textit{ExperimentProfile}, profiles for describing Web Services ) and a few hundred elements. The biggest single profile is currently the remodelled maximum schema from the META-SHARE project \cite{Gavrilidou2012meta} for describing corpora, with 419 components and 1587 elements 
(when expanded\footnote{The reusability of components results in an element expansion, i.e., elements of a component (e.g. \textit{Contact}) included by three other components (\textit{Project}, \textit{Institution}, \textit{Access}) will appear three times in the instantiated record.}).


\begin{table}
\caption{The development of defined profiles and DCs over time}
\label{table:dev_profiles}
%  \begin{tabular}{ l | r | r | r | r }
  \begin{tabular}{ l  r  r  r  r }

    \hline
date     & 2011-01 & 2012-06 & 2013-01 & 2013-06  \\
    \hline
Profiles & 40 & 53 & 87 & 124 \\
Distinct Components & 164 & 298 & 542 & 828 \\
Expanded Components & 1055 & 1536 & 2904 & 5757 \\
Distinct Elements & 511 & 893 & 1505 & 2399 \\
Expanded Elements & 1971 & 3030 & 5754 & 13232 \\
Distinct data categories & 203 & 266 & 436 & 499 \\
Data categories in the Metadata profile & 277 & 712 & 774 & 791 \\
Ratio of elements without DCs & 24,7\% & 17,6\% & 21,5\% & 26,5\% \\
Components with DCs & 28 & 67 & 115 & 140 \\

    \hline
  \end{tabular}
\end{table}


\subsection{Instance Data}

%\todoin{ add historical perspective on data - list overall}

The main CLARIN OAI-PMH harvester\footnote{\url{http://catalog.clarin.eu/oai-harvester/}}
collects records from 69 providers on daily basis. The complete dataset amounts to 540.065 records.
16 of the providers offer CMDI records, the other 53 provide OLAC/DC records\label{info:olac-records}, that are being converted into the corresponding CMD profile after harvesting. Next to these 81.226 original OLAC records, there a few providers offering their OLAC or DCMI-terms records already converted into CMDI, thus all in all OLAC, DCMI-terms records amount to 139.152.
On the other hand, some of the comparatively few providers of `native' CMD records expose multiple profiles (e.g. Meertens Institute uses 12 different profiles.) So we encounter both situations: one profile being used by many providers and one provider using many profiles.


\begin{table}
\caption{Top 20 CMD profiles, with the respective number of records}
\label{tab:cmd-profiles}
\begin{center}
  \begin{tabular}{ r l }
    \hline
\# records & profile \\
    \hline
155.403 & Song \\
138.257 & Session \\
92.996 & OLAC-DcmiTerms \\
46.156 & DcmiTerms \\
28.448 & SongScan \\
21.256 & SourceScan \\
19.059 & LiteraryCorpusProfile \\
16519 & Source \\
13626 & imdi-corpus \\
10610 & media-session-profile \\
7961 & SongAudio \\     
7557 & SymbolicMusicNotation \\
4485 & LCC DataProviderProfile \\
4485 & SourceProfile \\
4417 & Text \\
1982 & Soundbites-recording \\
1530 & Performer \\
1475 & ArthurianFiction \\
939 & LrtInventoryResource \\
873 & teiHeader \\
    \hline
  \end{tabular}
\end{center}
\end{table}

\begin{table}
\caption{Top 20 CMD collections, with the respective number of records}
\begin{center}
  \begin{tabular}{ r l }
    \hline
\# records & colleciton \\
    \hline
243.129 & Meertens collection: Liederenbank \\
46.658 & DK-CLARIN Repository \\
46.156 & Nederlands Instituut voor Beeld en Geluid Academia collectie \\
29.266 & childes \\
24.583 & DoBeS archive \\
23.185 & Language and Cognition \\
14.593 & talkbank \\
14.363 & Acquisition \\
14.320 & Institut fÃŒr Deutsche Sprache, CLARIN-D Zentrum, Mannheim \\
12.893 & MPI CGN \\
10.628 & Bavarian Archive for Speech Signals (BAS) \\
7.964 & Pacific And Regional Archive for Digital Sources in Endangered Cultures\\
7.348 & WALS RefDB \\
5.689 & Lund Corpora \\
4.640 & Oxford Text Archive \\
4.492 & Leipzig Corpora Collection \\
3.539 & Institut fÃŒr Deutsche Sprache, CLARIN-D Zentrum, Mannheim \\
3.280 & A Digital Archive of Research Papers in Computational Linguistics \\
3.147 & CLARIN NL \\
3.081 & MPI fÃŒr Bildungsforschung \\    
\hline
  \end{tabular}
\end{center}
\end{table}

We can also observe a large disparity on the amount of records between individual providers and profiles. Almost half of all records is provided by the Meertens Institute (\textit{Liederenbank} and \textit{Soundbites} collections), another 25\% by MPI for Psycholinguistics (\textit{corpus} + \textit{Session} records from the \textit{The Language Archive}). On the other hand there are 25 profiles that have less than 10 instances. This can be owing both to the state of the respective project (resources and records still being prepared) and the modelled granularity level (collection vs. individual resource).



\section{Other Metadata Formats and Collections }


Riley and Becker \cite{Riley2010seeing} put the overwhelming amount of existing metadata standards into a systematic comprehensive overview analyzing the use of standards from four aspects: community, domain, function, and purpose. Despite its aspiration on comprehensiveness it leaves out some of the formats relevant in the context of this work: IMDI, EDM, ESE, TEI?

The CLARIN deliverable \textit{Interoperability and Standards} \cite{CLARIN_D5.C-3} provides overview of standards, vocabularies and other normative/standardization work in the field of Language Resources and Technology.


\subsection{Dublin Core metadata terms + OLAC}
Since 1995
Maintained Dublin Core Metadata Initiative
DC, OLAC

"Dublin" refers to Dublin, Ohio, USA where the work originated during the 1995 invitational OCLC/NCSA Metadata Workshop,[8] hosted by the Online Computer Library Center (OCLC), a library consortium based in Dublin, and the National Center for Supercomputing Applications (NCSA).

comes in two version: 15 core elements  and 55 qualified terms ? 

\begin{quotation}
Early Dublin Core workshops popularized the idea of "core metadata" for simple and generic resource descriptions. The fifteen-element "Dublin Core" achieved wide dissemination as part of the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) and has been ratified as IETF RFC 5013, ANSI/NISO Standard Z39.85-2007, and ISO Standard 15836:2009.
\end{quotation}



Given its simplicity it is used as the common denominator in many applications, among others it is the base format in the OAI-PMH protocol.

It is required/expected as the base 
openarchives register: \url{http://www.openarchives.org/Register/BrowseSites}
 2006 OAI-repositories

DublinCore Resource Types\furl{http://dublincore.org/documents/resource-typelist/}

DublinCore to RDF mapping\furl{http://dublincore.org/documents/dcq-rdf-xml/}

\label{def:OLAC}

\xne{OLAC Metadata}\furl{http://www.language-archives.org/}format\cite{Bird2001},OLAC \cite{Simons2003OLAC} is a more specialized version of the \xne{Dublin Core metadata terms}, adapted to the needs of the linguistic community:

\begin{quotation}
 Uniform description across archives is ensured by limiting the values of certain metadata elements to the use of terms from agreed-upon controlled vocabularies. [\dots] OLAC adds encoding schemes that are designed specifically for describing language resources, such as subject language and linguistic data type. 
\end{quotation}

The \xne{OLAC Metadata} is the set of metadata elements archives participating in have agreed to use for describing language resources.

\todoin{check http://www.language-archives.org/OLAC/metadata.html}

 OLAC Archives contain over 100,000 records, covering resources in half of the world's living languages. More statistics on coverage.
http://www.language-archives.org/

Most of the OLAC records are integrated into CMDI (cf. \ref{tab:cmd-profiles}, \ref{reports:OLAC})


\subsection{TEI / teiHeader}
\label{def:tei}

\begin{quotation}
The Text Encoding Initiative (TEI) is a consortium which collectively develops and maintains a standard for the representation of texts in digital form. 
\end{quotation}

\url{http://www.tei-c.org/}

TEI is a de-facto standard for encoding any kind of digital textual resources being developed by a large community since 1994. It defines a set of elements to annotate individual aspects of the text being encoded. For the purposes of text description, metadata encoding the complex top-level element \code{teiHeader} is foreseen. TEI is not prescriptive, but rather descriptive, it does not provide just one fixed schema, but allows for a certain flexibility wrt to elements used and inner structure, allowing to generate custom schemas adopted to projects' needs.

Thus there is also not just one fixed \xne{teiHeader}. 

 TEI/teiHeader/ODD,



\subsection{ISLE/IMDI}

IMDI = ISLE Metadata  
http://www.mpi.nl/imdi/

The ISLE Meta Data Initiative (IMDI) is a proposed metadata standard to describe multi-media and multi-modal language resources. The standard provides interoperability for browsable and searchable corpus structures and resource descriptions with help of specific tools. 

Predecessor of CMDI

\subsection{MODS/METS}

Metadata Encoding and Transmission Standard - an XML schema for encoding descriptive, administrative, and structural metadata regarding objects within a digital library

Metadata Object Description Schema - is a schema for a bibliographic element set that may be used for a variety of purposes, and particularly for library applications.

\subsection{ESE, Europeana Data Model - EDM}

ESE Europeana Semantic Elements-

EDM\furl{http://europeana.ontotext.com/resource/edm/hasType?role=all} \cite{doerr2010europeana}


he Linked Data approach will play a major role in the European Digital Library (
http://europeana.eu
)
and solutions that can handle data expressed in the newly created, RDF-based
Europeana Data Model
(EDM)
are currently being investigated. This report summarizes the results of a study we performed on existing
RDF stores, in the context of Europeana and encompasses the following contributions


data.europeana.eu: The Europeana Linked Open Data Pilot\cite{haslhofer2011data}

\subsection{META-SHARE}
\label{def:META-SHARE}
Within the project META-SHARE format

META-SHARE created a new metadata model \cite{Gavrilidou2012meta}. Although inspired by the Component Metadata, META-SHARE metadata imposes a single large schema for all resource types with a minimal core subset of obligatory metadata elements and with many optional components. 
%In cooperation between metadata teams from CLARIN and META-SHARE 

The original META-SHARE schema actually accomodates four models for different resource types. Consequently, the model has been expressed as 4 CMD profiles each for a distinct resource type however all four sharing most of the components, as can be seen in figure \ref{fig:resource_info_5}. The biggest single profile is currently the remodelled maximum schema from the META-SHARE project for describing corpora, with 117 distinct components and 337 elements. When expanded, this translates to 419 components and 1587 elements. However, many of the components and elements are optional (and conditional), thus a specific instance will never use all the possible elements.

MetaShare ontology\furl{http://metashare.ilsp.gr/portal/knowledgebase/TheMetaShareOntology}


\subsection{Other}

OAI-ORE - is this a schema?



\section{Ontologies, Controlled Vocabularies, Reference Data, Authority Files}
\label{refdata}

Based on popular demand, the work on reference data for the SSH-community should cover at least the following dimensions (with tentative denominations of corresponding existing vocabularies):

\begin{itemize}
\item Data Categories / Concepts - ISOcat
\item Languages - ISO-639
\item Countries - country codes
\item Persons - GND, VIAF
\item Organizations - GND, VIAF
\item Schlagwörter/Subjects - GND, LCSH
\item Resource Typology - 
\end{itemize}

AAT - international Architecture and Arts Thesaurus
GND - Gemeinsame Norm Datei (GND ontology\furl{http://d-nb.info/standards/elementset/gnd}
GTAA - Gemeenschappelijke Thesaurus Audiovisuele Archieven (Common Thesaurus [for] Audiovisual Archives)
VIAF - Virtual International Authority File


Other related relevant activities and initiatives

A broader collection of related initiatives can be found at the German National Library website:
\furl{http://www.dnb.de/DE/Standardisierung/LinksAFS/linksafs_node.html}
FRBR - Functional Requirements for Bibliographic Records  2002 \cite{FRBR1998}

RDA - Resource Description and Access
http://metadaten-twr.org/ - Technology Watch Report: Standards in Metadata and Interoperability (last entry from 2011)
At MPDL, within the escidoc publication platform there seems to be (work  on) a service (since 2009 !) for controlled vocabularies: \furl{http://colab.mpdl.mpg.de/mediawiki/Control_of_Named_Entities}
Entity Authority Tool Set - a web application for recording, editing, using and displaying authority information about entities – developed at the New Zealand Electronic Text Centre (NZETC).
http://eats.readthedocs.org/en/latest/


\subsection{ISOcat - Data Category Registry}

ISO12620

\subsection{Classification Schemes, Taxonomies }
LCSH, DDC


\subsection{Other controlled Vocabularies}

Language codes ISO-639-1

\subsection{Domain Ontologies, Vocabularies}
Organization-Lists
LT-World !?


\subsubsection{LT-World}
Regarding existing domain-specific semantic resources \texttt{LT-World}\footnote{\url{http://www.lt-world.org/}},  the ontology-based portal covering primarily Language Technology being developed at DFKI\footnote{\textit{Deutsches Forschungszentrum fÃŒr KÃŒnstliche Intelligenz} - \url{http://www.dfki.de}},  is a prominent resource providing information about the entities (Institutions, Persons, Projects, Tools, etc.) in this field of study. \cite{Joerg2010}



\section{LRT Metadata Catalogs/Collections}
\label{sec:lrt-md-catalogs}
\todoin{Overview of catalogs, name, since, \#providers, \#resources}

\todoin{[DFKI/LT-World]  - collection or ontology}

\subsection{CMDI}
collections, profiles/Terms, ResourceTypes!

\subsection{OLAC}

\subsection{LAT, TLA}
Language Archiving Technology, now The Language Archive - provided by Max Planck Insitute for Psycholinguistics \footnote{\url{http://www.mpi.nl/research/research-projects/language-archiving-technology}}

\subsection{META-NET}



\begin{quotation}
META-SHARE is an open, integrated, secure and interoperable sharing and exchange facility for LRs (datasets and tools) for the Human Language Technologies domain and other applicative domains where language plays a critical role.

META-SHARE is implemented in the framework of the META-NET Network of Excellence. It is designed as a network of distributed repositories of LRs, including language data and basic language processing tools (e.g., morphological analysers, PoS taggers, speech recognisers, etc.).

\end{quotation}

The distributed networks of repositories consists of a number of member repositories, that offer their own subset of resource.

A few\footnote{7 as of 2013-07} of the members repositories play the role of managing nodes providing ``a core set of services critical to the whole of the META-SHARE network''\cite{Piperidis2012meta}, especially collecting the resource descriptions from other members and exposing the aggregated information to the users.
The whole network offers approximately 2.000 resources (the numbers differ even across individual managing nodes).


MetaShare ontology\furl{http://metashare.ilsp.gr/portal/knowledgebase/TheMetaShareOntology}



\subsection{ELRA}

European Language Resources Association

\furl{http://elra.info}


ELRA’s missions are to promote language resources for the Human Language Technology (HLT) sector, and to evaluate language engineering technologies. To achieve these two major missions, we offer a range of services, listed below and described in the "Services around Language Resources" section:


http://www.elda.org/
Evaluations and Language resources Distribution Agency

ELDA - Evaluations and Language resources Distribution Agency – is ELRA’s operational body, set up to identify, classify, collect, validate and produce the language resources which may be needed by the HLT – Human Language Technology – community. Besides, ELDA is involved in HLT evaluation campaigns.

ELDA handles the practical and legal issues related to the distribution of language resources, provides legal advice in the field of HLT, and drafts and concludes distribution agreements on behalf of ELRA.

ELRA Catalog 

http://catalog.elra.info/


Universal Catalog+
 Universal Catalogue is a repository comprising information regarding Language Resources (LRs) identified all over the world.


\subsection{Other}


\begin{description}
\item[LDC]  Linguistic Data Consortium\furl{http://www.ldc.upenn.edu/}
\item[OTA LR] Archiving Service provided by Oxford Text Archive \url{http://ota.oucs.ox.ac.uk/}
\end{description}

\section{Other Metadata Catalogs/Collections}
\label{sec:other-md-catalogs}

\subsection{(Digital) Libraries}


General (Libraries, Federations):

\begin{description}
\item[OCLC] \url{http://www.oclc.org}
    world's biggest Library Federation
\item[LoC] Library of Congress \url{http://www.loc.gov}
\item[EU-Lib] European Library \url{http://www.theeuropeanlibrary.org/portal/organisation/handbook/accessing-collections\_ en.htm}
\item[europeana] virtual European library - cross-domain portal \url{http://www.europeana.eu/portal/}
\end{description}




\section{Summary}

In this chapter, we gave an overview of the existing formats and datasets in the broad context of Language Resources and Technology