source: SMC4LRT/chapters/Data.tex @ 3665

\chapter{Analysis of the data landscape}
\label{ch:data}
This section gives an overview of existing standards and formats for metadata and content annotations in the field of Language Resources and Technology together with a description of their characteristics and their respective usage in the projects and initiatives.


\section{Metadata Formats}


\subsection{Component Metadata Framework}
\label{def:CMD}

The \emph{Component Metadata Framework} (CMD) is the data model of the CLARIN Component Metadata Infrastructure. (See \ref{def:CMDI} for information about the infrastructure. The XML-schema defining CMD -- the \xne{general-component-schema} -- is featured in appendix \ref{lst:cmd-schema}.)
CMD is used to define the so-called \var{profiles} being constructed out of reusable \var{components} -- collections of metadata fields. The components can contain other components and they can be reused in multiple profiles. Profile itself is just a special kind of a component (a sub class), with some additional administrative information.
The actual core provision for semantic interoperability is the requirement, that each CMD element (i.e. metadata field) refers ``via a PID to exactly one data category\footnote{persistently referenceable concept definition} (cf. \ref{def:DCR}), thus 
indicating unambiguously how the content of the field in a metadata description should be interpreted'' \cite{Broeder+2010}. 

This approach of integrating prerequisites for semantic interoperability directly into the process of metadata creation is fundamentally different from the traditional methods of schema matching that try to establish pairwise alignments between already existing schemas -- be it algorithm-based or by means of explicit manually defined crosswalks\cite{Shvaiko2005}. 

While the primary registry for data categories used in CMD is the \xne{ISOcat} Data Category Registry (cf. \ref{def:DCR}), other authoritative sources are accepted (so-called ``trusted registries''), especially the set of terms maintained by the Dublin Core Metadata Initiative \cite{DCMI:2005}.

Once the profiles are defined they are transformed into a XML-Schema, that prescribes the structure of the instance records.
The generated schema also conveys as annotation the information about the referenced data categories.


\subsubsection{CMD Profiles }
In the CR 124\footnote{All numbers are as of 2013-06 if not stated otherwise} public Profiles and 696 Components are defined. Table \ref{table:dev_profiles} shows the development of the CR and DCR population over time.

Next to the `native' CMD profiles a number of profiles have been created that implement existing metadata formats, like OLAC/DCMI-terms, TEI Header or the META-SHARE schema. The resulting profiles proof the flexibility/expressi\-vi\-ty of the CMD metamodel. The individual profiles differ also very much in their structure -- next to flat profiles with just one level of components or elements with 5 to 20 fields (\textit{dublincore}, \textit{collection}, the set of \textit{Bamdes}-profiles) there are complex profiles with up to 10 levels (\textit{ExperimentProfile}, profiles for describing Web Services ) and a few hundred elements. The biggest single profile is currently the remodelled maximum schema from the META-SHARE project \cite{Gavrilidou2012meta} for describing corpora, with 419 components and 1587 elements 
(when expanded\footnote{The reusability of components results in an element expansion, i.e., elements of a component (e.g. \textit{Contact}) included by three other components (\textit{Project}, \textit{Institution}, \textit{Access}) will appear three times in the instantiated record.}).


\begin{table}
\caption{The development of defined profiles and DCs over time}
\label{table:dev_profiles}
%  \begin{tabular}{ l | r | r | r | r }
  \begin{tabular}{ l  r  r  r  r }

    \hline
date     & 2011-01 & 2012-06 & 2013-01 & 2013-06  \\
    \hline
Profiles & 40 & 53 & 87 & 124 \\
Distinct Components & 164 & 298 & 542 & 828 \\
Expanded Components & 1055 & 1536 & 2904 & 5757 \\
Distinct Elements & 511 & 893 & 1505 & 2399 \\
Expanded Elements & 1971 & 3030 & 5754 & 13232 \\
Distinct data categories & 203 & 266 & 436 & 499 \\
Data categories in the Metadata profile & 277 & 712 & 774 & 791 \\
Ratio of elements without DCs & 24,7\% & 17,6\% & 21,5\% & 26,5\% \\
Components with DCs & 28 & 67 & 115 & 140 \\

    \hline
  \end{tabular}
\end{table}


\subsubsection{Instance Data}


%\todoin{ add historical perspective on data - list overall}

The main CLARIN OAI-PMH harvester\footnote{\url{http://catalog.clarin.eu/oai-harvester/}}
collects records from 69 providers on daily basis. The complete dataset amounts to 540.065 records.
16 of the providers offer CMDI records, the other 53 provide OLAC/DC records\label{info:olac-records}, that are being converted into the corresponding CMD profile after harvesting. Next to these 81.226 original OLAC records, there a few providers offering their OLAC or DCMI-terms records already converted into CMDI, thus all in all OLAC, DCMI-terms records amount to 139.152.
On the other hand, some of the comparatively few providers of `native' CMD records expose multiple profiles (e.g. Meertens Institute uses 12 different profiles.) So we encounter both situations: one profile being used by many providers and one provider using many profiles.


\begin{table}
\caption{Top 20 profiles, with the respective number of records}
\begin{center}
  \begin{tabular}{ r l }
    \hline
\# records & profile \\
    \hline
155.403 & Song \\
138.257 & Session \\
92.996 & OLAC-DcmiTerms \\
46.156 & DcmiTerms \\
28.448 & SongScan \\
21.256 & SourceScan \\
19.059 & LiteraryCorpusProfile \\
16519 & Source \\
13626 & imdi-corpus \\
10610 & media-session-profile \\
7961 & SongAudio \\     
7557 & SymbolicMusicNotation \\
4485 & LCC DataProviderProfile \\
4485 & SourceProfile \\
4417 & Text \\
1982 & Soundbites-recording \\
1530 & Performer \\
1475 & ArthurianFiction \\
939 & LrtInventoryResource \\
873 & teiHeader \\
    \hline
  \end{tabular}
\end{center}
\end{table}

\begin{table}
\caption{Top 20 collections, with the respective number of records}
\begin{center}
  \begin{tabular}{ r l }
    \hline
\# records & colleciton \\
    \hline
243.129 & Meertens collection: Liederenbank \\
46.658 & DK-CLARIN Repository \\
46.156 & Nederlands Instituut voor Beeld en Geluid Academia collectie \\
29.266 & childes \\
24.583 & DoBeS archive \\
23.185 & Language and Cognition \\
14.593 & talkbank \\
14.363 & Acquisition \\
14.320 & Institut fÃŒr Deutsche Sprache, CLARIN-D Zentrum, Mannheim \\
12.893 & MPI CGN \\
10.628 & Bavarian Archive for Speech Signals (BAS) \\
7.964 & Pacific And Regional Archive for Digital Sources in Endangered Cultures\\
7.348 & WALS RefDB \\
5.689 & Lund Corpora \\
4.640 & Oxford Text Archive \\
4.492 & Leipzig Corpora Collection \\
3.539 & Institut fÃŒr Deutsche Sprache, CLARIN-D Zentrum, Mannheim \\
3.280 & A Digital Archive of Research Papers in Computational Linguistics \\
3.147 & CLARIN NL \\
3.081 & MPI fÃŒr Bildungsforschung \\    
\hline
  \end{tabular}
\end{center}
\end{table}

We can also observe a large disparity on the amount of records between individual providers and profiles. Almost half of all records is provided by the Meertens Institute (\textit{Liederenbank} and \textit{Soundbites} collections), another 25\% by MPI for Psycholinguistics (\textit{corpus} + \textit{Session} records from the \textit{The Language Archive}). On the other hand there are 25 profiles that have less than 10 instances. This can be owing both to the state of the respective project (resources and records still being prepared) and the modelled granularity level (collection vs. individual resource).


\subsection{Dublin Core + OLAC}

DC, OLAC

openarchives register: \url{http://www.openarchives.org/Register/BrowseSites}
 2006 OAI-repositories

DublinCore Resource Types\furl{http://dublincore.org/documents/resource-typelist/}

DublinCore to RDF mapping\furl{http://dublincore.org/documents/dcq-rdf-xml/}

\label{def:OLAC}
A more specific version of the dublincore terms, adapted to the needs of the linguistic community is the 
OLAC\furl{http://www.language-archives.org/}format\cite{Bird2001}

OLAC \cite{Simons2003OLAC}.

\todoin{check http://www.language-archives.org/OLAC/metadata.html}

\begin{quotation}
The OLAC metadata set is the set of metadata elements that participating archives have agreed to use for describing language resources. Uniform description across archives is ensured by limiting the values of certain metadata elements to the use of terms from agreed-upon controlled vocabularies. The OLAC metadata set is equally applicable whether the resources are available online or not. The metadata set consists of the fifteen elements of the Dublin Core Metadata Set, plus the refinements and encoding schemes of the DCMI Metadata Terms—a widely accepted standard for describing resources of all types. To this general standard, OLAC adds encoding schemes that are designed specifically for describing language resources, such as subject language and linguistic data type. The OLAC Metadata Usage Guidelines describe (with examples) all the elements, refinements, and encoding schemes that may be used in OLAC metadata descriptions. The OLAC Metadata standard defines the XML format that is used for the interchange of metadata descriptions among participating archives. 
\end{quotation}




\subsection{TEI / teiHeader}
\label{tei}

 TEI/teiHeader/ODD,


\subsection{ISLE/IMDI}
 
\subsection{MODS/METS}

\subsection{Europeana Data Model - EDM}

\subsection{META-SHARE}
META-SHARE is another multinational project aiming to build an infrastructure for language resource\cite{Piperidis2012meta}, however focusing more on Human Language Technologies domain.\furl{http://meta-share.eu}

\begin{quotation}
META-NET is designing and implementing META-SHARE, a sustainable network of repositories of language data, tools and related web services documented with high-quality metadata, aggregated in central inventories allowing for uniform search and access to resources. Data and tools can be both open and with restricted access rights, free and for-a-fee. META-SHARE targets existing but also new and emerging language data, tools and systems required for building and evaluating new technologies, products and services.
\end{quotation}

\begin{quotation}
META-SHARE is an open, integrated, secure and interoperable sharing and exchange facility for LRs (datasets and tools) for the Human Language Technologies domain and other applicative domains where language plays a critical role.

META-SHARE is implemented in the framework of the META-NET Network of Excellence. It is designed as a network of distributed repositories of LRs, including language data and basic language processing tools (e.g., morphological analysers, PoS taggers, speech recognisers, etc.).

\end{quotation}

The distributed networks of repositories consists of a number of member repositories, that offer their own subset of resource.

A few\footnote{7 as of 2013-07} of the members repositories play the role of managing nodes providing ``a core set of services critical to the whole of the META-SHARE network''\cite{Piperidis2012meta}, especially collecting the resource descriptions from other members and exposing the aggregated information to the users.
The whole network offers approximately 2.000 resources (the numbers differ even across individual managing nodes).


MetaShare ontology\furl{http://metashare.ilsp.gr/portal/knowledgebase/TheMetaShareOntology}


\subsection{Other}

OAI-ORE - is this a schema?



\section{Content/Annotation Formats}

CHILDES, TEI, EAF!
(CES/XCES)
Open Annotation Collaboration (OAC)\footnote{\url{http://openannotation.org/}}

[LAF] Linguistic Annotation Framework



\section{Ontologies, Controlled Vocabularies, Reference Data, Authority Files}
\label{refdata}

Based on popular demand, the work on reference data for the SSH-community should cover at least the following dimensions (with tentative denominations of corresponding existing vocabularies):

\begin{itemize}
\item Data Categories / Concepts - ISOcat
\item Languages - ISO-639
\item Countries - country codes
\item Persons - GND, VIAF
\item Organizations - GND, VIAF
\item Schlagwörter/Subjects - GND, LCSH
\item Resource Typology - 
\end{itemize}

AAT - international Architecture and Arts Thesaurus
GND - Gemeinsame Norm Datei (GND ontology\furl{http://d-nb.info/standards/elementset/gnd}
GTAA - Gemeenschappelijke Thesaurus Audiovisuele Archieven (Common Thesaurus [for] Audiovisual Archives)
VIAF - Virtual International Authority File


Other related relevant activities and initiatives

A broader collection of related initiatives can be found at the German National Library website:
\furl{http://www.dnb.de/DE/Standardisierung/LinksAFS/linksafs_node.html}
FRBR - Functional Requirements for Bibliographic Records 
RDA - Resource Description and Access
http://metadaten-twr.org/ - Technology Watch Report: Standards in Metadata and Interoperability (last entry from 2011)
At MPDL, within the escidoc publication platform there seems to be (work  on) a service (since 2009 !) for controlled vocabularies: \furl{http://colab.mpdl.mpg.de/mediawiki/Control_of_Named_Entities}
Entity Authority Tool Set - a web application for recording, editing, using and displaying authority information about entities – developed at the New Zealand Electronic Text Centre (NZETC).
http://eats.readthedocs.org/en/latest/


\subsection{ISOcat - Data Category Registry}

ISO12620

\subsection{Classification Schemes, Taxonomies }
LCSH, DDC


\subsection{Other controlled Vocabularies}
Tagsets: STTS
Language codes ISO-639-1

\subsection{Domain Ontologies, Vocabularies}
Organization-Lists
LT-World !?



\section{LRT Metadata Catalogs/Collections}
\label{sec:lrt-md-catalogs}
\todoin{Overview of catalogs, name, since, \#providers, \#resources}

\todoin{[DFKI/LT-World]  - collection or ontology}

\subsection{CMDI}
collections, profiles/Terms, ResourceTypes!

\subsection{OLAC}

\subsection{LAT, TLA}
Language Archiving Technology, now The Language Archive - provided by Max Planck Insitute for Psycholinguistics \footnote{\url{http://www.mpi.nl/research/research-projects/language-archiving-technology}}

\subsection{META-NET}


\subsection{ELRA}

\subsection{Other}


\begin{description}
\item[LDC]  Linguistic Data Consortium 
\item[OTA LR] Archiving Service provided by Oxford Text Archive \url{http://ota.oucs.ox.ac.uk/}
\end{description}

\section{Other Metadata Catalogs/Collections}
\label{sec:other-md-catalogs}

\subsection{(Digital) Libraries}


General (Libraries, Federations):

\begin{description}
\item[OCLC] \url{http://www.oclc.org}
    world's biggest Library Federation
\item[LoC] Library of Congress \url{http://www.loc.gov}
\item[EU-Lib] European Library \url{http://www.theeuropeanlibrary.org/portal/organisation/handbook/accessing-collections\_ en.htm}
\item[europeana] virtual European library - cross-domain portal \url{http://www.europeana.eu/portal/}
\end{description}




\section{Summary}

In this chapter, we gave an overview of the existing formats and dataset in the broad context of Language Resources and Technology