source: SMC4LRT/chapters/SMC.tex @ 2696

\chapter{Semantic Mapping Component}


\section{Data Model}

Terms ? 
move to SKOS ?

RDF


\subsection{CMD namespace}
Describe the CMD-format?


\subsection{DCR in SKOS}
\label{dcr-skos}
Describe the mapping from DCR into SKOS

DCR recognizes following types of data categories:
simple, complex: closed, open, constrained, (container)?

\begin{figure*}[!ht] 
\begin{center}
\includegraphics[width=0.7\textwidth]{images/dc_types}
\end{center}
\caption{Data Category types}
\end{figure*}
\todo{cite:  ISOcat introduction at CLARIN-NL Workshop}

The export to CLAVAS-SKOS only considers/regards closed and simple DCs from the metadata profile are exported. 
A closed DC maps to a concept scheme and a simple DC to a SKOS concept in such a concept scheme. 
However it needs to be yet assessed how useful this approach is. In the metadata profile
there are many closed DCs with small value domains. How useful are those
in CLAVAS? 
Originally, the vocabulary repository has been conceived to manage rather large and complex value domains,
that do not fit easily in the DCR data-model.
Therefore a threshold seems sensible, where only value domains with more
then 20, 50 or 100 values are exported.

Open or constrained DCs are not exported as they don't provide anything to a vocabulary. \todo{cite: Menzo2013-03-12 mail}
However, they can become users of a CLAVAS vocabulary. Actually, providing vocabularies for constrained but large and complex conceptual domains is the main motivation for the vocabulary repository.

Currently (before integration of VAS and DCR), the only possibility to constrain the value domain of a data category
is by the means a XML Schema provides, like a regular expression. So for the data category \concept{languageID DC-2482}
the rule looks like:
\lstset{language=XML}
\begin{lstlisting}
        <dcif:conceptualDomain type="constrained">
                <dcif:dataType>string</dcif:dataType>
                <dcif:ruleType>XML Schema regular expression</dcif:ruleType>
                <dcif:rule>[a-z]{3}</dcif:rule>
        </dcif:conceptualDomain>
\end{lstlisting}

A current proposal by Windhouwer\todo{cite: Menzo2013-03-12 mail} for integration with CLAVAS foresees following extension:

\begin{lstlisting}
        <clavas:vocabulary href="http://my.openskos.org/vocab/ISO-639" type="closed"/>
\end{lstlisting}

\code{@href} points to the vocabulary. Actually a PID should be used in the context
of ISOcat, but it is not clear how persistent are the vocabularies. This may pose a problem as part of DC specification may now have a different persistency then the core.

\code{@type} could be \code{closed} or \code{open}. \code{closed}: only values in the vocabulary are
valid. \code{open}: the values in the vocabulary are hints/preferred values. Basically the DC itself is then open.

This would yield a definition of the conceptualDomain for the data category as follows:
 
\lstset{language=XML}
\begin{lstlisting}
  <dcif:conceptualDomain type="constrained">
     <dcif:dataType>string</dcif:dataType>
     <dcif:ruleType>XML Schema regular expression</dcif:ruleType>
     <dcif:rule>[a-z]{3}</dcif:rule>
  </dcif:conceptualDomain>
  <dcif:conceptualDomain type="constrained">
     <dcif:dataType>string</dcif:dataType>
     <dcif:ruleType>CLAVAS vocabulary</dcif:ruleType>
      <dcif:rule>
         <clavas:vocabulary href="http://my.openskos.org/vocab/ISO-639" type="closed"/>
      </dcif:rule>
  </dcif:conceptualDomain>
\end{lstlisting}

I.e. the new rule pointing to the vocabulary would be \emph{added}, so that tools that don't support CLAVAS
lookup but are capable of XSD/RNG validation, can still use the regular expression based definition.

 
\begin{note}

\noindent
something similar for the link to an EBNF grammar in SCHEMAcat: 

%\begin{lstlisting}
\begin{verbatim}
      <scr:valueSchema 
               xmlns:scr="http://www.isocat.org/ns/scr" 
               pid="http://hdl.handle.net/1839/00-SCHM-0000-0000-004A-A" 
               type="ISO 14977:1996 EBNF"/>
\end{verbatim}
%\end{lstlisting}
\end{note}



\subsection{smcIndex}\label{indexes}
In this section we describe \emph{smcIndex} -- the data type for input and output of the proposed application. 
An smcIndex is a human-readable string adhering to a specific syntax, denoting some search index. 
The generic syntax is:
\begin{eqnarray*}
smcIndex ::= context \ contextSep \ conceptLabel
\end{eqnarray*}

We distinguish two types of smcIndexes: (i) \emph{dcrIndex} referring to data categories and (ii) \emph{cmdIndex} denoting a specific
``CMD-entity'', i.e. a metadata field, component or whole profile defined within CMD. The \textit{cmdIndex} can be interpreted as a XPath into the instances of CMD-profiles. In contrast to it, the \textit{dcrIndexes} are generally not directly applicable on existing data, but can be understood as abstract indexes referring to well-defined concepts -- the data categories -- and for actual search they need to be resolved to the metadata fields they are referred by. In return one can expect to match more metadata fields from multiple profiles, all referring to the same data category.

These two types of smcIndex also follow different construction patterns:
\begin{eqnarray*}
smcIndex & ::= & dcrIndex \ | \ cmdIndex  \\
dcrIndex & ::= & dcrID \ contextSep \ datcatLabel \\
cmdIndex & ::= & profile \  \\
                      &  &  | \  [\ profile \ contextSep \ ] \ dotPath \\
dotPath  & ::= & [\ dotPath \ pathSep \ ] \ elemName \\
contextSep & ::= & \texttt{`.`} \ | \  \texttt{`:`} \\
pathSep & ::= & \texttt{`.`} \\
dcrId & ::= & \texttt{`isocat`} \ | \ \texttt{`dc`}
\end{eqnarray*}

The grammar is based on the way indices are referenced in CQL-syntax\footnote{Context Query Language, \url{http://www.loc.gov/standards/sru/specs/cql.html}} (\texttt{dc.title}) and on the dot-notation used in IMDI-browser\footnote{\url{http://www.lat-mpi.eu/tools/imdi}} (\texttt{Session.Location.Country}). 

\textit{dcrID} is a shortcut referring to a data category registry
%\footnote{Next to ISOcat other registries can function as a DCR, e.g., the Dublin Core set of metadata terms.} 
similar to the namespace-mechanism in XML-documents.  \textit{datcatLabel} is the verbose Identifier- (e.g. \texttt{telephoneNumber}) or the Name-attribute (in any available translation, e.g. \texttt{numero di telefono@it}) of the data category.
% While it is desirable to also allow the Name-attribute of the data category (\texttt{telephone number}), especially also the Names defined in other working languages (\texttt{numero di telefono@it, numer telefonu@pl}), special care has to be taken here as these attributes mostly contain white spaces, which could cause problems in downstream components, when parsing a complex query containing such indices. 
\textit{profile} is the name of the profile. % (despite the danger of ambiguity). 
\textit{dotPath} allows to address a leaf element (\texttt{Session.Actor.Role}), or any intermediary XML-element corresponding to a CMD-component (\texttt{Session.Actor})   within a metadata description. %This allows to easily express search in whole components, instead of having to list all individual fields.

Generally, smcIndexes can be ambiguous, meaning they can refer to multiple concepts, or entities (CMD-elements). This is due to the fact that the names of the data categories, and CMD-entities are not guaranteed unique. The module will have to cope with this, by providing on demand the list of identifiers corresponding to a given smcIndex.

%As an important sidenote -- cmdIndexes can be ambiguous, meaning they can refer to multiple entities (metadata fields), examples of valid indexes:
%\begin{verbatim}
%Name
%Actor.Name, Project.Name
%Session.Actor.Name, Drama.Actor.Name
%\end{verbatim}

%So we disambiguate (or narrow down the ambiguity) by prefixing context.


\subsection{Query language}
CQL?


\section{Semantic Mapping on concept level}

merging the pieces of information provided by those,
offering them semi-transaprently to the user (or application) on the consumption side.

a module of the Component Metadata Infrastructure performing semantic mapping on search indexes. This  builds the base for query expansion to facilitate semantic search and enhance recall when querying the Metadata Repository.


In this section, we describe the actual task of the proposed application -- \textbf{mapping indexes to indexes} -- in abstract terms. The returned mappings can be used by other applications to expand or translate the original user query, to match elements in other schemas.
\footnote{Though tightly related, mapping of terms and query expansion are to be seen as two separate functions.}
% \footnote{This primary usage of SMC for work with user-created query strings explains the need for human-readability of the indices.}

In the operation mode, the application accepts any index (\textit{smcIndex}, cf. \ref{indexes}) and returns a list of corresponding indexes (or only the input index, if no correspondences were found):
\newline

\textit{smcIndex $\mapsto$ smcIndex[ ]}
\newline

We can distinguish following levels for this mapping function:

(1) \emph{data category identity} -- for the resolution only the basic data category map derived from Component Registry is employed. Accordingly, only indexes denoting CMD-elements (\textit{cmdIndexes)} bound to a given data category are returned:
\newline

\texttt{isocat.size $\mapsto$ } \newline 
\verb|   [teiHeader.extent, |\newline 
\verb|    TextCorpusProfile.Number]|
\newline

\textit{cmdIndex} as input is also possible. It is translated to a corresponding data category, proceeding as above:
\newline

\texttt{imdi-corpus.Name   $\mapsto$ } \newline 
\verb|   (isocat.resourceName) |$\mapsto$  \newline 
\verb|   TextCorpusProfile.GeneralInfo.Name|
\newline 

(2) \emph{relations between data categories} -- employing also information from the Relation Registry, related (equivalent) data categories are retrieved and subsequently both the input and the related data categories resolved to cmdIndexes:
\newline

\texttt{isocat.resourceTitle  $\mapsto$ } 
\verb|   (+ dc.title) |$\mapsto$  \newline 
\verb|   [imdi-corpus.Title, | \newline 
\verb|    TextCorpusProfile.GeneralInfo.Title,| \newline 
\verb|    teiHeader.titleStmt.title,| \newline 
\verb|    teiHeader.monogr.title]|
\newline 

(3) \emph{container data categories} -- further expansions will be possible once the container data categories \cite{SchuurmanWindhouwer2011} will be used. Currently only fields (leaf nodes) in metadata descriptions are linked to data categories. However, at times, there is a need to conceptually bind also the components, meaning that besides the ``atomic'' data category for \texttt{actorName, there would be also a data category for the complex concept \texttt{Actor}.} 
Having concept links also on components will require a compositional approach to the task of semantic mapping, resulting in:
\newline 
\texttt{Actor.Name $\mapsto$ }\newline
\verb|    [Actor.Name, Actor.FullName, |\newline
\verb|     Person.Name, Person.FullName]|


\subsection*{Extensions}

A useful supplementary function of the module would be to provide a list of existing indexes.
That would allow the search user-interface to equip the query-input with autocompletion. Also the application should deliver additional information about the indexes like description and a link to the definition of the underlying entity in the source registry.

Once there will be overlapping\footnote{i.e. different relations may  be defined for one data category in different relation sets} user-defined relation sets in the Relation Registry an additional input parameter will be required to \emph{explicitly restrict the selection of relation sets} to apply in the mapping function.

Also, use of \emph{other than equivalency relations will necessitate more complex logic in the query expansion and accordingly also more complex response of the SMC, either returning the relation types themselves as well or equip the list of indexes with some similarity ratio.}



\section{Semantic Mapping on instance level}


\subsection{Mapping from strings to Entities}

Find matching entities in selected Ontologies based on the textual values in the metadata records.


Identify related ontologies:
LT-World \cite{Joerg2010}

task:
\begin{enumerate}
\item  express MDRecords in RDF
\item  identify related ontologies/vocabularies (category -> vocabulary)
\item  use a lookup/mapping function (Vocabulary Alignement Service? CATCH-PLUS?)

%\fbox{ function lookup: Category x String -> ConceptualDomain}
\begin{eqnarray*}
lookup(Category, Literal) -> ConceptualDomain??
\end{eqnarray*}


Normally this would be served by dedicated controlled vocabularies, but expect also some string-normalizing preprocessing etc.
\end{enumerate} 


\subsection{Linked Data - Express dataset in RDF}

Partly as by-product of the entities-mapping effort we will get the metadata-description rendered in RDF, linked with 
So theoretically we then only need to provide them ``on the web'', to make them a nucleus of the LinkedData-Cloud.


Technical aspects (RDF-store?) / interface (ontology browser?)

defining the Mapping:
\begin{enumerate}
\item convert to RDF
translate: MDREcord -> [\#mdrecord \#property literal]
\item map: \#mdrecord \#property literal  -> [\#mdrecord \#property \#entity]
\end{enumerate}


\begin{figure*}[!ht]
\includegraphics[width=1\textwidth]{images/SMC_CMD2LOD}
\caption{The process of transforming the CMD metadata records to and RDF representation}
\end{figure*}


\section{Semantic Search}

Main purpose for the undertaking described in previous two chapters (mapping of concepts and entities) is to enhance the search capabilities of the MDService serving the Metadata/Resources-data. Namely to enhance it by employing ontological resources.
Mainly this enhancement shall mean, that the user can access the data indirectly by browsing one or multiple  ontologies, 
with which the data will then be linked. These could be for example ontologies of Organizations and Projects.

In this section we want to explore, how this shall be accomplished, ie how to bring the enhanced capabilities to the user.
Crucial aspect is the question how to deal with the even greater amount of information in a user-friendly way, ie how to prevent overwhelming, intimidating or frustrating the user.

Semi-transparently means, that primarily the semantic mapping shall integrate seamlessly in the interaction with the service, but it shall ``explain'' - offer enough information - on demand, for the user to understand its role and also being able manipulate easily.

?
Facets
Controlled Vocabularies 
Synonym Expansion (via TermExtraction(ContentSet))

\subsection{Query Expansion}


\section{Semantic Mapping in Metadata vs. Content/Annotation}
AF + DCR + RR