source: SMC4LRT/chapters/Data.tex @ 3776

\chapter{Analysis of the data landscape}
\label{ch:data}
This section gives an overview of existing standards and formats for metadata in the field of Language Resources and Technology together with a description of their characteristics and their respective usage in the initiatives and data collections. Special attention is paid to the Component Metadata Framework representing the base data model for the infrastructure this work is part of.


\section{Component Metadata Framework}
\label{def:CMD}

The \emph{Component Metadata Framework} (CMD) is the data model of the CLARIN Component Metadata Infrastructure. (See \ref{def:CMDI} for information about the infrastructure. The XML-schema defining CMD -- the \xne{general-component-schema} -- is featured in appendix \ref{lst:cmd-schema}.)
CMD is used to define the so-called \var{profiles} being constructed out of reusable \var{components} -- collections of metadata fields. The components can contain other components and they can be reused in multiple profiles. Profile itself is just a special kind of a component (a sub class), with some additional administrative information.
The actual core provision for semantic interoperability is the requirement, that each CMD element (i.e. metadata field) refers ``via a PID to exactly one data category(cf. \ref{def:DCR}\footnote{in short: persistently referencable concept definition}), thus 
indicating unambiguously how the content of the field in a metadata description should be interpreted'' \cite{Broeder+2010}. 

%This approach of integrating prerequisites for semantic interoperability directly into the process of metadata creation is fundamentally different from the traditional methods of schema matching that try to establish pairwise alignments between already existing schemas -- be it algorithm-based or by means of explicit manually defined crosswalks\cite{Shvaiko2005}. 

While the primary registry for data categories used in CMD is the \xne{ISOcat} Data Category Registry (cf. \ref{def:DCR}), other authoritative sources are accepted (so-called ``trusted registries''), especially the set of terms maintained by the Dublin Core Metadata Initiative \cite{DCMI:2005}.

Once the profiles are defined they are transformed into a XML Schema, that prescribes the structure of the instance records.
The generated schema also conveys as annotation the information about the referenced data categories.


\subsection{CMD Profiles }
In the CR 124\footnote{All numbers are as of 2013-06 if not stated otherwise} public Profiles and 696 Components are defined. Table \ref{table:dev_profiles} shows the development of the CR and DCR population over time.

Next to the `native' CMD profiles a number of profiles have been created that implement existing metadata formats, like OLAC/DCMI-terms, TEI Header or the META-SHARE schema. The resulting profiles proof the flexibility/expressi\-vi\-ty of the CMD metamodel. The individual profiles differ also very much in their structure -- next to flat profiles with just one level of components or elements with 5 to 20 fields (\concept{dublincore}, \concept{collection}, the set of \concept{Bamdes}-profiles) there are complex profiles with up to 10 levels (\concept{ExperimentProfile}, profiles for describing Web Services ) and a few hundred elements. The biggest single profile is currently the remodelled maximum schema from the META-SHARE project \cite{Gavrilidou2012meta} for describing corpora, with 117 distinct components and 337 elements (or 419 components and 1587 elements when expanded\footnote{The reusability of components results in an element expansion, i.e., elements of a component (e.g. \concept{Contact}) included by three other components (\concept{Project}, \concept{Institution}, \concept{Access}) will appear three times in the instantiated record.}).


\begin{table}
\caption{The development of defined profiles and DCs over time}
\label{table:dev_profiles}
%  \begin{tabular}{ l | r | r | r | r }
  \begin{tabular}{ l  r  r  r  r }

    \hline
date     & 2011-01 & 2012-06 & 2013-01 & 2013-06  \\
    \hline
Profiles & 40 & 53 & 87 & 124 \\
Distinct Components & 164 & 298 & 542 & 828 \\
Expanded Components & 1055 & 1536 & 2904 & 5757 \\
Distinct Elements & 511 & 893 & 1505 & 2399 \\
Expanded Elements & 1971 & 3030 & 5754 & 13232 \\
Distinct data categories & 203 & 266 & 436 & 499 \\
Data categories in the Metadata profile & 277 & 712 & 774 & 791 \\
Ratio of elements without DCs & 24,7\% & 17,6\% & 21,5\% & 26,5\% \\
Components with DCs & 28 & 67 & 115 & 140 \\

    \hline
  \end{tabular}
\end{table}


\subsection{Instance Data}

%\todoin{ add historical perspective on data - list overall}

The main CLARIN OAI-PMH harvester\footnote{\url{http://catalog.clarin.eu/oai-harvester/}}
collects records from 69 providers on daily basis. The complete dataset amounts to 540.065 records.
16 of the providers offer CMDI records, the other 53 provide OLAC/DC records\label{info:olac-records}, that are being converted into the corresponding CMD profile after harvesting. Next to these 81.226 original OLAC records, there a few providers offering their OLAC or DCMI-terms records already converted into CMDI, thus all in all OLAC, DCMI-terms records amount to 139.152.
On the other hand, some of the comparatively few providers of `native' CMD records expose multiple profiles (e.g. Meertens Institute uses 12 different profiles.) So we encounter both situations: one profile being used by many providers and one provider using many profiles.


\begin{table}
\caption{Top 20 CMD profiles, with the respective number of records}
\label{tab:cmd-profiles}
\begin{center}
  \begin{tabu}{ r l }
    \hline
\rowfont{\itshape\small} \# records & profile \\
    \hline
155.403 & Song \\
138.257 & Session \\
92.996 & OLAC-DcmiTerms \\
46.156 & DcmiTerms \\
28.448 & SongScan \\
21.256 & SourceScan \\
19.059 & LiteraryCorpusProfile \\
16519 & Source \\
13626 & imdi-corpus \\
10610 & media-session-profile \\
7961 & SongAudio \\     
7557 & SymbolicMusicNotation \\
4485 & LCC DataProviderProfile \\
4485 & SourceProfile \\
4417 & Text \\
1982 & Soundbites-recording \\
1530 & Performer \\
1475 & ArthurianFiction \\
939 & LrtInventoryResource \\
873 & teiHeader \\
    \hline
  \end{tabu}
\end{center}
\end{table}

\begin{table}
\caption{Top 20 CMD collections, with the respective number of records}
\begin{center}
  \begin{tabu}{ r l }
    \hline
\rowfont{\itshape\small} \# records & colleciton \\
    \hline
243.129 & Meertens collection: Liederenbank \\
46.658 & DK-CLARIN Repository \\
46.156 & Nederlands Instituut voor Beeld en Geluid Academia collectie \\
29.266 & childes \\
24.583 & DoBeS archive \\
23.185 & Language and Cognition \\
14.593 & talkbank \\
14.363 & Acquisition \\
14.320 & Institut fÃŒr Deutsche Sprache, CLARIN-D Zentrum, Mannheim \\
12.893 & MPI CGN \\
10.628 & Bavarian Archive for Speech Signals (BAS) \\
7.964 & Pacific And Regional Archive for Digital Sources in Endangered Cultures\\
7.348 & WALS RefDB \\
5.689 & Lund Corpora \\
4.640 & Oxford Text Archive \\
4.492 & Leipzig Corpora Collection \\
3.539 & Institut fÃŒr Deutsche Sprache, CLARIN-D Zentrum, Mannheim \\
3.280 & A Digital Archive of Research Papers in Computational Linguistics \\
3.147 & CLARIN NL \\
3.081 & MPI fÃŒr Bildungsforschung \\    
\hline
  \end{tabu}
\end{center}
\end{table}

We can also observe a large disparity on the amount of records between individual providers and profiles. Almost half of all records is provided by the Meertens Institute (\textit{Liederenbank} and \textit{Soundbites} collections), another 25\% by MPI for Psycholinguistics (\textit{corpus} + \textit{Session} records from the \textit{The Language Archive}). On the other hand there are 25 profiles that have less than 10 instances. This can be owing both to the state of the respective project (resources and records still being prepared) and the modelled granularity level (collection vs. individual resource).



\section{Other LRT Metadata Formats and Collections }
\label{sec:lrt-md-catalogs}

Next to CLARIN and CMDI, there is a hoist of related previous and concurrent work. In the following, we briefly introduce some  formats and data collections established in the field and, where applicable, we also sketch the ties with CMDI and existing integration efforts.

As for comprehensive overview of formats and standards, the CLARIN deliverable \textit{Interoperability and Standards} \cite{CLARIN_D5.C-3} provides overview of standards, vocabularies and other normative/standardization work in the field of Language Resources and Technology. And \textit{Seeing standards: a visualization of the metadata universe} by Riley and Becker \cite{Riley2010seeing} pus the overwhelming amount of existing metadata standards into a systematic comprehensive visual overview analyzing the use of standards from four aspects: community, domain, function, and purpose. Though despite its aspiration on comprehensiveness it leaves out some of the formats relevant in the context of this work: IMDI, EDM, ESE.


\subsection{Dublin Core metadata terms}
The work on this metadata format started in 1995 at Metadata Workshop\furl{http://dublincore.org/workshops/dc1/} organized by OCLC/NCSA in  Dublin, Ohio, USA. Nowadays maintained by Dublin Core Metadata Initiative.

It is a fixed set of terms for a basic generic description of a range of resources (both virtual and physical) coming in two version\furl{http://dublincore.org/documents/dcmi-terms/}: 
\begin{description}
\item[Dublin Core Metadata Element Set (DCMES) ] namespace: \code{/elements/1.1/}\\
the original set 15 terms, standardized as IETF RFC 5013, ISO Standard 15836-2009 and NISO Standard Z39.85-2007
\item[Dublin Core metadata terms ]  namespace: \code{/terms/} \\
the extended `Qualified' set of 55 terms, extending the original 15 ones (replicating them in the new namespace for consistency)
\end{description}

The DCMI terms format is very widely spread nowadays. Thanks to its simplicity it is used as the common denominator in many applications, content management systems integrate Dublin Core to use in \code{meta} tags of served pages (\code{<meta name="DC.Publisher" content="publisher-name" >}), it is default minimal description in content repositories (Fedora-commons, DSpace). It is also the obligatory base format in the OAI-PMH protocol. The OpenArchives register\furl{http://www.openarchives.org/Register/BrowseSites} lists more than 2100 data providers.

There are multiple possible serializations, in particular a mapping t RDF is specified\furl{http://dublincore.org/documents/dcq-rdf-xml/}.
Worth noting is Dublin Core's take on classification of resources\furl{http://dublincore.org/documents/resource-typelist/}.

The simplicity of the format is also it's main drawback when considered as metadata format in the research communities. It it too general to capture all specific details, individual research groups need to describe different kinds of resources with.

\subsection{OLAC}
\label{def:OLAC}

\xne{OLAC Metadata}\furl{http://www.language-archives.org/}format \cite{Bird2001} is a application profile\cite{heery2000application}, of the \xne{Dublin Core metadata terms}, adapted to the needs of the linguistic community. It is developed and maintained by the \xne{Open Language Archives Community} providing a common platform and an infrastructure for ``creating a worldwide virtual library of language resources'' \cite{Simons2003OLAC}. 

The OLAC schema \furl{http://www.language-archives.org/OLAC/1.1/olac.xsd} extends the dcterms schema mainly by adding attributes with controlled vocabularies, for domain specific semantic annotation (\code{linguistic-field}, \code{role}, \code{linguistic-type}, \code{language}, \code{discourse-type}).

\begin{quotation}
 Uniform description across archives is ensured by limiting the values of certain metadata elements to the use of terms from agreed-upon controlled vocabularies. [\dots] OLAC adds encoding schemes that are designed specifically for describing language resources, such as subject language and linguistic data type. 
\end{quotation}

\lstset{language=XML}
\begin{lstlisting}[label=lst:sampleolac, caption=Sample OLAC record]
<olac:olac>
   <creator>Bloomfield, Leonard</creator>
   <date>1933</date>
   <title>Language</title>
   <publisher>New York: Holt</publisher>
</olac:olac>
\end{lstlisting}

OLAC provides a ``search over 100,000 records collected from 44 archives\furl{http://www.language-archives.org/archives}, covering resources in half of the world's living languages''. 

Note, that OLAC archives are being harvested by CLARIN harvester and OLAC records are part of the CMDI joint metadata domain (cf. \ref{tab:cmd-profiles}, \ref{reports:OLAC}).



\subsection{TEI / teiHeader}
\label{def:tei}

\begin{quotation}
 The Text Encoding Initiative (TEI) is a consortium which collectively develops and maintains a standard for the representation of texts in digital form \dots  [Next to] its chief deliverable is a set of Guidelines which specify encoding methods for machine-readable texts, chiefly in the humanities, social sciences and linguistics, \dots the Consortium provides a variety of TEI-related resources, training events and software. [abgridged]
\end{quotation} 

TEI is a de-facto standard for encoding any kind of digital textual resources being developed by a large community since 1994. It defines a set of elements to annotate individual aspects of the text being encoded. For the purposes of text description, metadata encoding (of main concern for us) the complex top-level element \code{teiHeader} is foreseen. TEI is not prescriptive, but rather descriptive, it does not provide just one fixed schema, but allows for a certain flexibility wrt to elements used and inner structure, allowing to generate custom schemas adopted to projects' needs. Thus there is also not just one fixed \code{teiHeader}. 

Some of the data collections encoded in TEI are die Korpora des DWDS\furl{http://www.dwds.de}, Deutsches Textarchiv\furl{http://www.dwds.de/dta} \cite{Geyken2011deutsches}, Oxford Text Archives\furl{http://ota.oucs.ox.ac.uk/}

There has been an intense cooperation between the TEI and CMDI community on the issue of interoperability and multiple efforts to express teiHeader in CMDI were undertaken (cf. \ref{results:tei}) as a starting point for integrating TEI-based data into the CLARIN infrastructure. 


\subsection{ISLE/IMDI -- The Language Archive}

\xne{IMDI}\furl{http://www.mpi.nl/imdi/} (\xne{EAGLES/ISLE Meta Data Initiative}) is an elaborate format for detailed descriptions of multi-media/multi-modal language resoruces developed within the corresponding project\cite{wittenburg2000eagles} 2000 to 2003.

To serve the main goal of the project, easing access to language resources fostering the reuse, resource description in this new format were created for a number of collections and were made available via a dedicated \xne{IMDI browser}\furl{http://corpus1.mpi.nl/ds/imdi_browser/}, that allowed browsing the collection structure as well as complex advanced search over the deeply structured metadata. Also a metadata editor was developed for generating records in this format, with provisions for offline field-work and synchronization with the repository.

The project lead and responsible for running the repository and whole infrastructure was the Technical Group at MPI for Psycholinguistics, who has engaged in a number of projects aimed at building a stable technical infrastructure for long-term archiving and work with language resources since its foundation (together with the Institute itself) in 1970s\furl{http://tla.mpi.nl/home/history/}. Recently, the group and the established infrastructure has been renamed to \xne{TLA -- The Language Archive}\furl{http://tla.mpi.nl/} ``Your partner for language data, tools and archiving'', where on one platform both the hoist of language resources and their description are preserved and provided as well as tools for working with this data is offered. The archive is also an aggregator itself, offering various collection from different (also external) projects (like DOBES, CGN, RELISH, etc.).

IMDI can be seen as predecessor of CMDI, the team of the TG being the driving force behind the development of both. A \xne{imdi-session} profile, the corresponding IMDI to CMDI conversion 
as well as the transformed records were among the first to be added to the new CMD Infrastructure in 2010. The statistics
of CMDI records list round 138.000 \xne{Session} records and round 13.000 \xne{imdi-corpus} records, modelling the collections for the sessions. Also, the metadata editor \xne{Arbil} was refactored to work with the new data model.


\subsection{META-SHARE}
\label{def:META-SHARE}

META-SHARE was the subproject (2010-2013) of META-NET, a Network of Excellence consisting of 60 research centres from 34 countries, that covered the technical aspects.


\begin{quotation}
META-SHARE is an open, integrated, secure and interoperable sharing and exchange facility for LRs (datasets and tools) for the Human Language Technologies domain and other applicative domains where language plays a critical role.

\end{quotation}

Within the project META-SHARE a new metadata format was developed\cite{Gavrilidou2012meta}. Although inspired by the Component Metadata, META-SHARE metadata imposes a single large schema for all resource types with a subset of core obligatory elements and with many optional components. 
%In cooperation between metadata teams from CLARIN and META-SHARE 

The original META-SHARE schema actually accomodates four models for different resource types. Consequently, the model has been expressed as 4 CMD profiles each for a distinct resource type however all four sharing most of the components, as can be seen in figure \ref{fig:resource_info_5}. The biggest single profile is currently the remodelled maximum schema from the META-SHARE project for describing corpora, with 117 distinct components and 337 elements. When expanded, this translates to 419 components and 1587 elements. However, many of the components and elements are optional (and conditional), thus a specific instance will never use all the possible elements. (See \ref{reports-meta-share} for more details about the format based on its integration into CMDI)

The technical infrastructure of META-SHARE represents a distributed network of repositories consists of a number of member repositories, that offer their own subset of resource\furl{http://www.meta-share.eu/}.

Selected member repositories\footnote{7 as of 2013-07}  play the role of managing nodes providing ``a core set of services critical to the whole of the META-SHARE network''\cite{Piperidis2012meta}, especially collecting the resource descriptions from other members and exposing the aggregated information to the users.
The whole network offers approximately 2.000 resources (the numbers differ even across individual managing nodes).

One point of criticism from the community was, the fact, that META-SHARE infrastructure does not provide any interface to the outer world, such as a OAI-PMH endpoint.

%? MetaShare ontology\furl{http://metashare.ilsp.gr/portal/knowledgebase/TheMetaShareOntology}


\subsection{ELRA}

European Language Resources Association\furl{http://elra.info} ELRA, offers a large collection of language resources (over 1.100) with focus on spoken resources, but also written, terminological and multimodal resources, mostly under license for a fee (although selected resources are available for free as well).
The available datasets can be search for via ELRA Catalog\furl{http://catalog.elra.info/}
Additionally ELRA runs the so-called \xne{Universal Catalog} -- a repository comprising information regarding Language Resources (LRs) identified all over the world.

\begin{quotation}
ELRA's missions are to promote language resources for the Human Language Technology (HLT) sector, and to evaluate language engineering technologies.

ELDA\furl{http://www.elda.org/} - Evaluations and Language resources Distribution Agency -- is ELRA's operational body, set up to identify, classify, collect, validate and produce the language resources which may be needed by the HLT -- Human Language Technology -- community. 

ELDA handles the practical and legal issues related to the distribution of language resources, provides legal advice in the field of HLT, and 
drafts and concludes distribution agreements on behalf of ELRA.
\end{quotation}

\subsection{LDC}

Linguistic Data Consortium\furl{http://www.ldc.upenn.edu/} hosted by University of Pennsylvania is another provider/aggregator of high quality curated language resources. The data is provided for a fee, more than 650 resources have been made available since 1993. The catalog is freely accessible. The metadata is additionally aggregated by OLAC archives.

\section{Formats and Collections in the World of Libraries}
\label{sec:lib-formats}

There are at least two reasons to concern ourselves with the developments in the world of Libraries and Information Systems (LIS): the long tradition implying rich experience and the fact, that almost all of the resources in the libraries are language resources. This argument gets even more relevant in the light of the efforts to digitize large portions of the material pursued in many (national) libraries in the last years (cf. discussion on Libraries partnering with Google). And given the amounts of data, even the sole bibliographic records constitute sizable language resources in they own right.

%\item[LoC] Library of Congress \url{http://www.loc.gov}
%\item[EU-Lib] European Library \url{http://www.theeuropeanlibrary.org/portal/organisation/handbook/accessing-collections\_ en.htm}
%\item[europeana] virtual European library - cross-domain portal \url{http://www.europeana.eu/portal/}
%\end{description}

\subsection{Formats  -- MARC, METS, MODS}

There is a long tradition of standardized metadata formats in the world of Libraries and Information Systems (LIS), major role in the standardization being assumed for decades by the Library of Congress\furl{http://www.loc.gov/standards/}.

The \xne{MARC}\furl{www.loc.gov/marc/} set of formats (being used since 1970s ) ``are standards for the representation and communication of bibliographic and related information in machine-readable form''. A number of variants developed over the years, the most widely spread is \xne{MARC 21} since 1999 -- is the standard format used for communication among libraries around the world.

MARC 21 consists of 5 ``communication formats'' for specific types of data (Bibliographic, Authority Data, Holdings Data, Classification, and Community Information), are widely used standards for the representation and exchange of bibliographic, authority, holdings, classification, and community information data in machine-readable form. In 2002, the Library of Congress developed the \xne{MARCXML} schema for representing MARC records in XML; 

\xne{METS -- Metadata Encoding and Transmission Standard} - a format from the family of Library of Congress standards (since 2001) for encoding descriptive, administrative, and structural metadata regarding objects within a digital library.
It is dedicated primarily to capture the structure of the digital objects, ``record the various relationships that exist between pieces of content, and between the content and metadata that compose a digital library object'' \cite{mets2010manual}.
A METS record acts as a flexible container that accomodates other pieces of data (different levels of metadata and encoded objects themselves or references to those) in external formats\furl{http://www.loc.gov/standards/mets/mets-extenders.html}.

Number of tools have been developed to author and process \xne{METS} format\furl{http://www.loc.gov/standards/mets/mets-tools.html} and numerous projects (online editions, DAM systems) use METS for structuring and recording the data\footnote{\url{http://www.loc.gov/standards/mets/mets-registry.html} though seems rather outdated} among others also \xne{austrian literature online}\furl{http://www.loc.gov/standards/mets/mets-registry.html}

\xne{Metadata Object Description Schema} - ``is a schema for a bibliographic element set that may be used for a variety of purposes, and particularly for library applications''. It is a simplified subset of MARC 21 using  language-based tags rather than numeric ones,
more than Dublin Core. One of endorsed schemas to extend (be used inside) METS.

There have been efforts to create a conceptually more sound base for the bibliographic data -- in 1998 \xne{Functional Requirements for Bibliographic Records} (FRBR) \cite{FRBR1998} was published, an abstract model for the data expressed as an Entity Relationship Model and a standard based on FRBR, the \xne{Resource Description and Access} (RDA) has been proposed as an comprehensive standard for resource description and discovery, that however was confronted with opposition from the LIS community, questioning the need of abandoning established cataloging practices \cite{gorman2007rda}.
And although there is still work on RDA, among others by the Library of Congress, there has been no wider adoption of the standard by the LIS community until now.

\subsection{ESE, Europeana Data Model - EDM}

Within the big european initiative \xne{Europeana} (cf. \ref{lit:digi-lib}) information about digitised objects are collected from a great number of cultural institutions from all of Europe, currently hosting information about 29 million objects from 2.200 institutions from 36 countries\furl{http://www.pro.europeana.eu/web/guest/content}.

For collecting metadata from the content providers, Europeana originally developed and advised the common format \xne{ESE Europeana Semantic Elements}\furl{http://pro.europeana.eu/ese-documentation} a Dublin Core-based application profile\furl{www.europeana.eu/schemas/ese/ESE-V3.4.xsd}. Soon it became obvious, that this format is too limiting and work started on a Semantic Web compatible RDF-based format -- the Europeana Data Model EDM\furl{http://pro.europeana.eu/edm-documentation} \cite{isaac2012europeana,haslhofer2011data,doerr2010europeana}.
EDM is fully compatible with ESE, which is (and will be) accepted from the providers. There is also already a SPARQL endpoint\furl{http://europeana.ontotext.com/sparql} to explore the Europeana data in the new format.
%https://github.com/europeana

%%%%%%%%%%%%%%%%%%
\section{Controlled Vocabularies, Reference Data, Ontologies}
\label{refdata}

One goal of this work being the groundwork for exposing the discussed dataset in the Semantic Web
one preparatory task is to identify external semantic resources like controlled vocabularies or ontologies that the dataset could be linked with\footnote{Similar activity of inventarizing vocabularies and thesauri was conducted in the context of the \xne{Europeana} initiative
\url{http://europeanalabs.eu/wiki/WP12Vocabularies}\url{https://europeanalabs.eu/wiki/DesignSemanticThesauri}}. 

Conceptually, we want to partition these resources in two types. On the one hand abstract concepts constituting all kinds of classifications, typologies, taxonomies. On the other hand named entities that exist(ed) in real world, like persons, organizations or geographical places. Main motivation for this distinction is the insight, that while for named entities there is (mostly) ``something'' in the (physical) world that gives a solid ground for equivalence relations between references from different sources (sameAs), for concepts we need to accept a plurality of existing conceptualizations and while we can (and have to) try to identify relations between them, the equivalence relation is inherently much weaker. This insight entails a partly different approach -- simply put, while we can aspire to create one large list/index encompassing all named entities, we have to maintain a forest of conceptual trees.

In the following we inventarize such resources (cf. tables \ref{table:data-ne}, \ref{table:data-concepts}) covering the domains expected to be needed for linking the original dataset. (Information about size of the dataset is meant rather as a rough indication of the "general weight" of the dataset, not necessarily a precise up to date information.) The acronyms in the tables are resolved in the glossary \ref{table:vocab-glossary}
How this resources will be employed is discussed in \ref{sec:values2entities}.
Additionally, some verbose commentary follows.

%\subsubsection{Named entities}

The largest controlled vocabularies of named entities are the authority files of (national) libraries. These are further aggregated into the so-called Virtual International Authority File, a huge resource, with entries from different authority files referring to the same entity being merged. This resource can be explored via a search interface and there is also a search service for applications.
Other general large-scale resources are the vocabularies curated and provided by Getty Research Institute\furl{http://www.getty.edu/research/tools/vocabularies/index.html}, however there is only a limited free access and licensed and fee for full access. But recently there work was announced to publish the vocabularies as LOD\furl{http://www.getty.edu/research/tools/vocabularies/lod/index.html}

Regarding existing domain-specific semantic resources \texttt{LT-World}\footnote{\url{http://www.lt-world.org/}},  the ontology-based portal covering primarily Language Technology being developed at DFKI\footnote{Deutsches Forschungszentrum fÃŒr KÃŒnstliche Intelligenz, \url{http://www.dfki.de}},  is a prominent resource providing information about the entities (Institutions, Persons, Projects, Tools, etc.) in this field of study. \cite{Joerg2010} 

Also to mention \xne{Yago}, a large knowledge base created by MPI informatik integrating dbpedia, geonames and wordnet\furl{http://www.mpi-inf.mpg.de/yago-naga/yago/} \cite{Suchanek2007yago}.

So we witness a strong general trend towards Semantic Web and Linked Open Data.

%Next to these ``global big players'' there are a number of other initiatives on different scale dedicated to a more specific domain.

%Resources that contain different types of data (e.g. persons, places and classifications like GND or Yago) are divided and mentioned in individual tables by type.

%\subsection{Concepts -- Classifications, Taxonomies, \dots}


\begin{comment}

VoID "Vocabulary of Interlinked Datasets") is an RDF based schema to describe linked datasets\furl{http://semanticweb.org/wiki/VoID}

\subsection{schema.org}
http://schema.org/docs/datamodel.html
http://www.w3.org/wiki/WebSchemas/ExternalEnumerations

microdata or 
http://www.w3.org/TR/rdfa-lite/
 Resource Description Framework in attributes

the entire WorldCat cataloging collection made publicly
available using Schema.org mark-up with library extensions for use by developers and
search partners such as Bing, Google, Yahoo! and Yandex

OCLC begins adding linked data to WorldCat by appending
Schema.org descriptive mark-up to WorldCat.org pages, thereby
making OCLC member library data available for use by intelligent
Web crawlers such as Google and Bing

\end{comment}

\section{Summary}

In this chapter, we gave an overview of the existing formats and datasets in the broad context of Language Resources and Technology.
We also gave an overview of main formats and collections in the domain of Library and Information Services and a inventory of existing controlled vocabularies for named entities and concepts (taxonomies, classifications), needed as input in section \ref{sec:values2entities} about mapping values to entities.



\begin{landscape}
\begin{table}
\caption{Controlled vocabularies of named entities -- Persons, Organizations, Works, Language Names, Geographica}
\label{table:data-ne}
%  \begin{tabu}{  p{0.2\textwidth}  p{0.2\textwidth}  p{0.2\textwidth}   p{0.2\textwidth}   p{0.2\textwidth} }
  \begin{tabu}{  >{\sffamily}l l r X X}
    \hline
\rowfont{\itshape\small} name & provider & size (items / facts)  & description & access \\
    \hline
VIAF & OCLC + NatLibs & $\gg$ 1E7 & union of national authority files & search service, search app \\
GND/p & DNB & 4.6E6 & Persons, universal, lang:de  & \href{http://d-nb.info/standards/elementset/gnd}{GND ontology}\\
GND/k & '' & 1.2E6 & Organizations, universal, lang:de  & \\
GND/w & '' & 193,000 & Works, lang:de  & \\
GND/g & '' & 293.000 & Geographica, lang:de & \\
ULAN & Getty & 202,720 / 638,900 & persons, artists     & \\
TGN & Getty & 992.310 / 1.7E6 & also historical place names & \href{http://www.getty.edu/research/tools/vocabularies/index.html}{web search} \\
%CONA & Getty & & records for cultural works & \\       
dbpedia & Wikipedia & $\sim$ 4E6 & all kinds of entities in up to 111 langs & \href{http://wiki.dbpedia.org/Downloads}{data dumps}, \href{http://dbpedia-live.openlinksw.com/sparql}{live SPARQL endpoint} \\
& & \multicolumn{3}{l}{764,000 persons; 333,000 works; 192,000 organizations; 639,000 geographica } \\
Yago \cite{Suchanek2007yago} & MPI Informatik & 1E7 / 1.2E8 & huge semantic KB (aggregated from Wikipedia, Wordnet, Geonames) & \href{http://www.mpi-inf.mpg.de/yago-naga/yago/downloads.html}{data dumps} \\
\href{http://lt-world.de}{LT-World} & DFKI & 3.300 persons& ontology-based portal for LRT & \href{http://www.lt-world.org/kb/}{portal} \\
& & 4.600 organizations & & \\
Geonames & Geonames & \textgreater 1E7 (2.8E6 / 5.5E6) & "modern" place names & data dump + web service \\
PKND     & prometheus & \textgreater 37,000 & persons, artists & \href{http://prometheus-bildarchiv.de/de/tools/pknd}{XML dump} \\
\href{http://gazetteer.dainst.org/}{iDAI.gazetteer} & DAI &  & archaeologically relevant places & search interface \\
%Pelagios & AIT & 25 datasets & search over 25 datasets of archeologically relevant places & API\furl{https://github.com/pelagios/pelagios-cookbook/wiki/Using-the-Pelagios-API} \\
\href{http://pleiades.stoa.org}{Pleiades} & & 34.000 & A community-built gazetteer and graph of ancient places & CSV, KML and RDF data dumps \\
LCCN & LoC & \textgreater 1.2E7 & identifier for bibliographic records & \href{http://authorities.loc.gov/}{search service}, search app \\
ISO 3166 & ISO & 249 & Official country codes, lang: en, fr &   \\
ISO-639-1& ISO & 185 & basic language codes & \href{http://www.loc.gov/standards/iso639-2/php/English_list.php}{static list} \\
ISO-639-3 & SIL & $\sim$ 7.679 & 3-letter code for every human language & \href{http://www-01.sil.org/iso639-3/}{view/download} \\
CLAVAS & CLARIN & 2.500  & organization names extracted from CMD records & \href{https://openskos.meertens.knaw.nl/}{OpenSKOS} -- search service \\
\hline
\end{tabu}
\end{table}

\begin{comment}
\hline
  \end{tabu}
\end{table}

\begin{table}
\caption{Controlled vocabularies of named entities -- Geographica}
\label{table:data-ne-places}

%  \begin{tabu}{  p{0.2\textwidth}  p{0.2\textwidth}  p{0.2\textwidth}   p{0.2\textwidth}   p{0.2\textwidth} }
  \begin{tabu}{  >{\sffamily}l l r X X}
    \hline
\rowfont{\itshape\small} name & provider & size (items / facts) & description & access \\

\end{comment}


\begin{table}
\caption{Taxonomies, Classifications, Thesauri}
\label{table:data-concepts}
  \begin{tabu}{  >{\sffamily}l l r X X}
    \hline
\rowfont{\itshape\small} name & provider & size (items / facts) & description & access \\
    \hline
AAT & Getty & \href{http://www.getty.edu/research/tools/vocabularies/aat/aat_faq.html}{34,880 / 245,530} & subjects in  art and architecture &  \\
LCSH & LoC &  & subjects, universal & \href{http://fast.oclc.org/searchfast/}{FAST} (Faceted Application of Subject Terminology), \href{http://experimental.worldcat.org/fast/}{Linked Data FAST} \\
LCC  & LoC & & universal hierarchical classification & web app: \href{http://classificationweb.net/}{classification web} \\
GND/s & DNB & 202.000 & subjects (Schlagwörter), universal, lang:de & \\
GTAA & NISL & 3.800 & Subjects, describing TV programs & \href{http://datahub.io/de/dataset/gemeenschappelijke-thesaurus-audiovisuele-archieven}{(RDF) data dumps}, \href{https://openskos.meertens.knaw.nl/}{OpenSKOS} -- search service \\
DDC & OCLC & & universal classification by field of study, multi langs & \href{http://dewey.info/}{dewey.info} \\
UDC & & & & \\
Wiki Categories & Wikipedia & 995,911& classification of Wiki articles as skos:Concepts & SKOS Vocabulary, SPARQL \\
 DBpedia Ontology & Wikipedia & 529 / 2333 & general classification of Wiki articles as ontology & \href{http://wiki.dbpedia.org/Ontology39?v=g9b}{RDF data}, SPARQL\\
ISOcat & (CLARIN) & \textgreater 6,500 & data categories defining (linguistic) concepts & \href{http://www.isocat.org}{web-app}, service \\
Object Names Thes. & British Museum & &  classification of objects in the collection & \\
Material Thes. & British Museum & & classification of material & \\
Thes. Monument Types & British Museum & & types of monuments & \\
Hornbostel-Sachs-Systematik & & 300 categories & classification of musical instruments & \href{http://www.music.vt.edu/musicdictionary/texth/Hornbostel-Sachs.html}{web page} \\
Oberbegriffsdatei  & DMB & & a set of vocabularies for museums, lang:de  & \url{museumsvokabular.de}, PDF, XML dumps\\
Iconclass & RKD & 28,000 & taxonomy of subject of an image &  \href{http://iconclass.org/data/iconclass.20121019.nt.gz}{RDF dump} \\
\href{http://dirt.projectbamboo.org/}{DiRT} & Project Bamboo & 32 categories & taxonomy of research tools (1,200 tools)  &  \\
%Scholarly Methods Taxonomy & DARIAH & 100 & research activities in a 2-level hierarchy and brief scope notes & in preparation \\
\hline
\end{tabu}
\end{table}

\end{landscape}



\begin{table}
\caption{Glossary of acronyms used in the overview of controlled vocabularies (tables \ref{table:data-ne}, \ref{table:data-concepts}) }
\label{table:vocab-glossary}

%  \begin{tabu}{  >{\sffamily}l p{0.8\textwidth}
\begin{tabular}{ >{\sffamily}l p{0.8\textwidth}}
%    \hline
%\rowfont{\itshape\small} name & provider & size (items / facts)  & description & access \\
 %   \hline

AAT & international Architecture and Arts Thesaurus, Getty \\
CONA & Cultural Objects Name Authority \\
DAI & Deutsches ArchÀologisches Institut \\
DDC & Dewey Decimal Classification       \\
DFKI & Deutsches Forschungszentrum fÃŒr KÃŒnstliche Intellligenz \\
DMB & Deutscher Museumsbund \\
DNB & Deutsche National Bibliothek \\
FAST & Faceted Application of Subject Terminology \\
Getty & Getty Research Institute curating the \href{http://www.getty.edu/research/tools/vocabularies/index.html}{vocabularies}, part of Getty Trust \\
GND & \emph{Gemeinsame Norm Datei} - Integrated authority Files of the German National Library \\
GTAA & Gemeenschappelijke Thesaurus Audiovisuele Archieven (Common Thesaurus [for \& Audiovisual Archives) \\
% {quotation} The thesaurus consists of several facets for describing TV programs: subjects; people mentioned; named entities (Corporation names, music bands etc); locations; genres; makers and presentators. \end{quotation} \\
ISO & International Standardization Organization \\
LCCN & Library of Congress Control Number \\
LCC & Library of Congress Classification \\
LCSH & Library of Congress Subject Headings \\
LoC & Library of Congress\furl{http://loc.gov} \\
OCLC & Online Computer Library Center\furl{http://www.oclc.org} -- world's biggest library federation \\
PKND & \href{http://prometheus-bildarchiv.de/de/tools/pknd}{prometheus} KÃŒnstlerNamensansetzungsDatei\\
RKD & Rijksbureau voor Kunsthistorische Documentatie -- Netherlands Institute for Art History \\
TGN & Getty Thesaurus of Geographic Names \\
UDC & Universal Decimal Classification                            \\
ULAN & Union List of Artist Names \\
VIAF & Virtual International Authority File -- union of the authority files of \textgreater 20 national (and prominent research) libraries  \\
\end{tabular} 
\end{table}