wiki:FCS-Specification-ScrapBook

Context Navigation

Version 62 (modified by teckart, 10 years ago) (diff)
--

FCS Specification Scrapbook

Issues with current document

Uncomprehensible and not well structured :(
Resource enumeration (aka scan on fcs.resource) rather complex and unintuitive
Basic KWIC records has no provision for multiple "highlight" hits
No (clear) recommendation for using Resource and ResouceFragment
What about recursiveness in Resource (see current schema)? What is the use case?

General ideas / design goals towards better specification

Define FCS conformance level independent of what SRU/CQL do. Don't call them "level", but maybe something like profile to avoid confusion.
1. Do a basic profile first
2. Do an advanced/extend profile later in a separate specification or specification amendment (which must be, of course, compatible to basic profile)
3. Add provisions to, e.g. explain output, to allow endpoints to indicate the profile, they support
Better structure of document (and don't include aggregation stuff; that's a different specification; implementors of endpoints should not need to worry about aggregator implementation)
Keep XML sanity always in mind (so there are no namespace issues as in CMDI)
Drop resource enumeration in favor of endpoint resource description
Drop the recursiveness of Resource, content models should be: Resource (DataView*, ResourceFragment*) and ResourceFragment (DataView*)
Drop the KWIC data view in favor of HITS data view; the latter will allow for multiple hit highlights
Honor and use extension hooks provided by SRU/CQL
Non-normative stuff
1. Endpoint specific extension hooks, e.g. to avoid tag abuse of DataView. Resource.xsd could provide an extension hook, so arbitrary XML could also be embedded.
2. TODO: Clients can put query parameters at @ref to allow hit highlighting on their systems
Do we to keep Geolocation (KML) as defined Data View in "basic" profile?
TODO: Add to description/change wording of Generic Hits Data View, that it is intended as the lowest common denominator to serialize search results

Proposal for new specification

The following is a proposal for a revisited federated content search specification. When done, cut and paste to the appropriate section of the Wiki and publish on the CLARIN web page.

CLARIN Federated Content Search (CLARIN-FCS)

FCS Specification Scrapbook
CLARIN Federated Content Search (CLARIN-FCS)
Discussion

Introduction

The main goal of CLARIN federated content search (CLARIN-FCS) is to introduce a interface specification, to decouple the search engine functionality from its exploitation, i.e. user-interfaces, third-party applications and to allow services to access search engines in a uniform way.

Terminology

The key words MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL in this document are to be interpreted as described in RFC2119.

Glossary

Aggregator: A module or service to dispatch queries to repositories and collect results.

CLARIN-FCS, FCS: CLARIN federated content search, an interface specification to allow searching within resource content of repositories.

Client: A software component, that implements the interface specification to query Endpoints, i.e. an aggregator or a user-interface.

CQL: Contextual Query Language, previously known as Common Query Language, is a formal language for representing queries to information retrieval systems such as search engines, bibliographic catalogs and museum collection information.

Endpoint: A software component, that implements the CLARIN-FCS interface specification and translates between CLARIN-FCS and a search engine.

Interface Specification: Common harmonized interface and suite of protocols that repositories need to implement.

Search Engine: A software component within a repository, that allows for searching within the repository contents.

SRU: Search and Retrieve via URL, is a protocol for Internet search queries.

Data View: A Data View is a mechanism to support different representations of search results, e.g. a "hits with highlights" view, an image or a geolocation.

Data View Payload, Payload: The actual content encoded within a Data View, i.e. a CMDI metadata record or a KML encoded geolocation.

PID: A Persistent identifier is a long-lasting reference to a digital object.

Repository: A software component at a CLARIN center that stores resources (= data) and information about these resources (= metadata).

Repository Registry: A separate service that allows registering Endpoints and provides information about these to other components, e.g. an aggegator. The CLARIN Center Registry is an implementation of such a repository registry.

Normative References

RFC2119: Key words for use in RFCs to Indicate Requirement Levels, IETF RFC 2119, March 1997,
http://www.ietf.org/rfc/rfc2119.txt

XML-Namespaces: Namespaces in XML 1.0 (Third Edition), W3C, 8 December 2009,
http://www.w3.org/TR/2009/REC-xml-names-20091208/

OASIS-SRU-Overview: searchRetrieve: Part 0. Overview Version 1.0, OASIS, January 2013,
http://docs.oasis-open.org/search-ws/searchRetrieve/v1.0/os/part0-overview/searchRetrieve-v1.0-os-part0-overview.doc (HTML), (PDF)

OASIS-SRU-APD: searchRetrieve: Part 1. Abstract Protocol Definition Version 1.0, OASIS, January 2013,
http://docs.oasis-open.org/search-ws/searchRetrieve/v1.0/os/part1-apd/searchRetrieve-v1.0-os-part1-apd.doc (HTML) (PDF)

OASIS-SRU12: searchRetrieve: Part 2. SRU searchRetrieve Operation: APD Binding for SRU 1.2 Version 1.0, OASIS, January 2013,
http://docs.oasis-open.org/search-ws/searchRetrieve/v1.0/os/part2-sru1.2/searchRetrieve-v1.0-os-part2-sru1.2.doc (HTML) (PDF)

OASIS-CQL: searchRetrieve: Part 5. CQL: The Contextual Query Language version 1.0, OASIS, January 2013,
http://docs.oasis-open.org/search-ws/searchRetrieve/v1.0/os/part5-cql/searchRetrieve-v1.0-os-part5-cql.doc (HTML) (PDF)

SRU-Explain: searchRetrieve: Part 7. SRU Explain Operation version 1.0, OASIS, January 2013,
http://docs.oasis-open.org/search-ws/searchRetrieve/v1.0/os/part7-explain/searchRetrieve-v1.0-os-part7-explain.doc (HTML) (PDF)

SRU-Scan: searchRetrieve: Part 6. SRU Scan Operation version 1.0, OASIS, January 2013,
http://docs.oasis-open.org/search-ws/searchRetrieve/v1.0/os/part6-scan/searchRetrieve-v1.0-os-part6-scan.doc (HTML) (PDF)

LOC-SRU12: SRU Version 1.2: SRU Search/Retrieve Operation, Library of Congress,
http://www.loc.gov/standards/sru/sru-1-2.html

LOC-DIAG: SRU Version 1.2: SRU Diagnostics List, Library of Congress,
http://www.loc.gov/standards/sru/diagnostics/diagnosticsList.html

Non-Normative References

RFC6838: Media Type Specifications and Registration Procedures, IETF RFC 6838, January 2013,
http://www.ietf.org/rfc/rfc6838.txt
RFC3023: XML Media Types, IETF RFC 3023, January 2001,
http://www.ietf.org/rfc/rfc3023.txt
KML: Keyhole Markup Language (KML), Open Geospatial Consortium, 2008,
http://www.opengeospatial.org/standards/kml

Typographic and XML Namespace conventions

The following typographic conventions for XML fragments will be used throughout this specification:

<prefix:Element>
An XML element with the Generic Identifier Element that is bound an XML namespace denoted by the prefix prefix.
@attr
An XML attribute with the name attr
string
The literal string must be used either as element content or attribute value.

Endpoints and Clients MUST adhere the XML-Namespaces specification. The CLARIN-FCS interface specification generally does not dictate whether XML elements should be serialized in their prefixed or non-prefixed syntax, but Endpoints MUST ensure that the correct XML namespace is used for elements and that XML namespaces are declared correctly. Clients MUST be agnostic to which syntax for serializing the XML elements, i.e. if the prefixed or un-prefixed variant was used, and SHOULD operate solely on expanded names, i.e. pairs of namespace name and local name.

The following XML namespace names and prefixes are used throughout this specification. The column "Recommended Syntax" indicates, which syntax variant SHOULD be used by the Endpoint to serialize the XML response.

Prefix	Namespace Name	Comment	Recommended Syntax
`fsc`	`http://clarin.eu/fcs/1.0`	CLARIN-FCS Resources	prefixed
`ed`	`http://clarin.eu/fcs/1.0/endpoint-description`	CLARIN-FCS Endpoint Description	prefixed
`hits`	`http://clarin.eu/fcs/1.0/hits`	CLARIN-FCS Generic Hits	prefixed
`sru`	`http://www.loc.gov/zing/srw/`	SRU	prefixed
`diag`	`http://www.loc.gov/zing/srw/diagnostic/`	SRU Diagnostics	prefixed
`zr`	`http://explain.z3950.org/dtd/2.0/`	SRU/ZeeRex Explain	prefixed
`cmdi`	`http://www.clarin.eu/cmd/`	Component Metadata	un-prefixed
`kml`	`http://www.opengis.net/kml/2.2`	Keyhole Markup Language	un-prefixed

CLARIN-FCS Interface Specification

The CLARIN-FCS interface specification defined two profiles, an extensible result format and a set of required operations. CLARIN-FCS is built on the SRU/CQL standard and additional functionality required for CLARIN-FCS is added through SRU/CQL's extension mechanisms.

Generally, CLARIN-FCS Interface Specification consists of two components, a set of formats and a transport protocol. The Endpoint component is a software component that acts as a bridge between the Formats, that are send by a Client using the Transport Protocol, and a Search Engine. The Search Engine is a custom software component, that allows searching in the language resources of a CLARIN center. The Endpoint basically implements the transport protocol and acts as an mediator between the CLARIN-FCS specific formats and the idiosyncrasies of Search Engines. The following figure illustrates the overall architecture.

                 +---------+
                 |  Client |
                 +---------+
                     /|\
                      |
          -------------------------
         |        SRU / CQL        |
         | w/CLARIN-FCS extensions |
          -------------------------
                      |
                     \|/
 +-----------------------------------------+
 |        |      Endpoint     /|\          |
 |        |                    |           |
 |  ---------------    ------------------  |
 | | Translate CQL |  | Translate Result | |
 |  ---------------    ------------------  |
 |        |                    |           |
 |       \|/                   |           |
 +-----------------------------------------+
                     /|\
                      |
                     \|/
        +---------------------------+
        |       Search Engine       |
        +---------------------------+

The following sections describe the CLARIN-FCS profiles and query and result formats, how SRU/CQL is used as a transport protocol in the context of CLARIN-FCS and the required CLARIN-FCS specific extensions to SRU.

Profiles

CLARIN-FCS defines two profiles:

Basic profile

Endpoints MUST support term-only queries.
Endpoints SHOULD support terms combined with boolean operator queries (AND and OR), including subqueries. Endpoints MAY also support NOT or PROX operator queries. If the Endpoint does not support a query, i.e. the used operators are not supported by the Endpoint, it MUST return an appropriate error message using the appropriate SRU diagnostic.
Examples for valid CQL queries for the basic profile:

cat
"cat"
cat AND dog
"grumpy cat"
"grumpy cat" AND dog
"grumpy cat" OR "lazy dog"
cat AND (mouse OR "lazy dog")

The Endpoint is MUST perform the query on an annotation tier, that makes the most sense for the user, i.e. the textual content for a text corpus resource or the orthographic transcription of a spoken language corpus. Endpoints SHOULD perform the query case-sensitive.
Endpoint MUST NOT silently accept queries that include CQL features besides term-only and terms combined with boolean operator queries, i.e. queries involving context sets, etc.

Extended profile: This profile will support more sophisticated queries such as selecting annotation tiers, expanding of tags, or mapping of data categories.
NOTE: the extended profile is not yet defined and will be part of a future CLARIN-FCS specification.

Endpoints and Clients MUST support the basic profile. For now, Endpoints and Clients MUST NOT claim to support the extended profile.

Result Format

CLARIN-FCS uses a customized format for returning results. Resource and Resource Fragments serve as containers for hit results, that are presented in one or more Data View. The following section describes the Resource format and Data View format and section Operation ''searchRetrieve'' will describe, how hits are embedded within SRU responses.

Resource and ResourceFragment

To encode search results, CLARIN-FCS supports two building blocks:

Resources: A Resource is an searchable entity at the Endpoint, such as a text corpus or an multi-modal corpus. A resource SHOULD be a self-contained unit, i.e. not a sentence in a text corpus or a time interval in an audio transcription.
Resource Fragments: A Resource Fragment is a smaller unit in a Resource, i.e. a sentence in a text corpus or a time interval in an audio transcription.

A Resource SHOULD be the most precise unit of data that is directly addressable as a "whole". A Resource SHOULD contain a Resource Fragment, if the hit consists of just a part of the Resource unit, if the hit is a sentence within a large text. A Resource Fragment SHOULD be addressable within a resource, i.e. it has an offset or a resource-internal identifier. Using Resource Fragments is OPTIONAL, but Endpoints are encouraged to use them. If the Endpoint encodes a hit with a Resource Fragment, the actual hit SHOULD be encoded as a Data View that is encoded in a Resource Fragment.

Endpoints SHOULD always provide a links to the resource itself, i.e. each Resource or Resource Fragment SHOULD be identified by a persistent identifier or providing a URI, that is unique for Endpoint. Even if direct linking is not possible, i.e. due to licensing issues, the Endpoints SHOULD provide a URI to link to a web-page describing the corpus or collection, including instruction on how to obtain it. Endpoints SHOULD provide links that are as specific as possible (and logical), i.e. if a sentence within a resource cannot be addressed directly, the Resource Fragment SHOULD NOT contain a persistent identifier or a URI.

If the Endpoint can provide both, a persistent identifier as well as a URI, for either Resource or Resource Fragment, they SHOULD provide both. When working with results, Clients SHOULD prefer persistent identifiers over regular URIs.

Resource and Resource Fragment are serialized in XML and Endpoints MUST generate responses, that are valid according to the XML schema "Resource.xsd " (download ). A Resource is encoded in the form of a <fcs:Resource> element, a Resource Fragment in the form of a <fcs:ResourceFragment> element. The content of a Data View is wrapped in a <fcs:DataView> element. <fcs:Resource> is the top-level element and MAY contain zero or more <fcs:DataView> elements and MAY contain zero or more <fcs:ResourceFragment> elements. A <fcs:ResourceFragment> element MUST contain one or more <fcs:DataView> elements.

The elements <fcs:Resource>, <fcs:ResourceFragment> and <fcs:DataView> MAY carry a @pid and/or a @ref attribute, which allows linking to the original data represented by the Resource, Resource Fragment, or Data View. A @pid attribute MUST contain a valid persistent identifier, a @ref MUST contain valid URI, i.e. a "plain" URI without the additional semantics of being a persistent reference.

Endpoints MUST use the identifier http://clarin.eu/fcs/1.0 for the responseItemType (= content for the <sru:recordSchema> element) in SRU responses.

Endpoints MAY serialize hits as multiple Data Views, however they MUST provide the Generic Hits (HITS) Data View either encoded as a Resource Fragment (if applicable), or otherwise within the Resource (if there is no reasonable Resource Fragment). Other Data Views SHOULD be put in a place that is logical for their content (as is to be determined by the Endpoint), e.g. a metadata Data View would most likely be put directly below Resource and a Data View representing some annotation layers directly around the hit is more likely to belong within a Resource Fragment.

Example 1:

<fcs:Resource xmlns:fcs="http://clarin.eu/fcs/1.0" pid="http://hdl.handle.net/4711/00-15">
  <fcs:DataView type="application/x-clarin-fcs-hits+xml">
      <!-- data view payload omitted -->
  </fcs:DataView>
</fcs:Resource>

This example shows a simple hit, which is encoded in one Data View of type Generic Hits embedded within a Resource. The type of the Data View is identified by the MIME type application/x-clarin-fcs-hits+xml. The Resource is referenceable by the persistent identifier http://hdl.handle.net/4711/08-15.

Example 2:

<fcs:Resource xmlns:fcs="http://clarin.eu/fcs/1.0" pid="http://hdl.handle.net/4711/08-15">
  <fcs:ResourceFragment>
    <fcs:DataView type="application/x-clarin-fcs-hits+xml">
      <!-- data view payload omitted -->
    </fcs:DataView>
  </fcs:ResourceFragment>
</fcs:Resource>

This example shows a hit encoded as a Resource Fragment embedded within a Resource. The actual hit is again encoded as one Data View of type Generic Hits. The hit is not directly referenceable, but the Resource, in which hit occurred, is referenceable by the persistent identifier http://hdl.handle.net/4711/08-15. In contrast to Example 1, the Endpoint decided to provide a "semantically richer" encoding and embedded the hit using a Resource Fragment within the Resource to indicate that the hit is a part of a larger resource, e.g. a sentence in a text document.

Example 3:

<fcs:Resource xmlns:fcs="http://clarin.eu/fcs/1.0"
              pid="http://hdl.handle.net/4711/08-15" ref="http://repos.example.org/file/text_08_15.html">
  <fcs:DataView type="application/x-cmdi+xml"
                pid="http://hdl.handle.net/4711/08-15-1" ref="http://repos.example.org/file/08_15_1.cmdi">
      <!-- data view payload omitted -->
  </fcs:DataView>
  <fcs:ResourceFragment pid="http://hdl.handle.net/4711/08-15-2" ref="http://repos.example.org/file/text_08_15.html#sentence2">
    <fcs:DataView type="application/x-clarin-fcs-hits+xml">
      <!-- data view payload omitted -->
    </fcs:DataView>
  </fcs:ResourceFragment>
</fcs:Resource>

The most complex example is similar to Example 2, i.e. it shows a hit is encoded as one Generic Hits Data View in a Resource Fragment, that is embedded in a Resource. In contrast to Example 2, another Data View of type CMDI is embedded directly within the Resource. The Endpoint can use this type of Data View to directly provide CMDI metadata about the Resource to Clients. All entities of the Hit can be referenced by a persistent identifier and a URI. The complete Resource is referenceable by either the persistent identifier http://hdl.handle.net/4711/08-15 or the URI http://repos.example.org/file/text_08_15.html and the CMDI metadata record in the CMDI Data View is referenceable either by the persistent identifier http://hdl.handle.net/4711/08-15-1 or the URI http://repos.example.org/file/08_15_1.cmdi. The actual hit in the Resource Fragment is also directly referenceable by either the persistent identifier http://hdl.handle.net/4711/00-15-2 or the URI http://repos.example.org/file/text_08_15.html#sentence2.

Endpoints MUST serialize one Resource for each hit, i.e. they MUST NOT combine several hits in one Resource. E.g., if a query matches five different sentences within one text (= the resource), the Endpoint must serialize five Resource (= one per hit) and embed each within one SRU result (see below).

Data View

A Data View serves as a container for representing search results within CLARIN-FCS. Data Views are designed to allow for different representations of results, i.e. they are deliberately kept open to allow further extensions with more supported Data View formats. The content of a Data View is called Payload. Each Payload is typed and the type of the Payload is recorded in the @type attribute if the <fcs:DataView> element. The Payload type is is identified by a MIME type (RFC6838, RFC3023). If no existing MIME type can be used, implementors SHOULD define a properer private mime type.

The Payload of a Data View can either be deposited inline or by reference. In the case of inline, it MUST be serialized as an XML fragment below the <fcs:DataView> element. This method is the preferred methods payloads that can easily serialized in XML. In the case of by reference, the content cannot easily deposited inline, i.e. it is binary content. In this case, the Data View MUST include a @ref or @pid attribute that links location for Clients to download the payload. This location SHOULD be openly accessible, i.e. data can be downloaded freely without any need to perform a login.

For the basic profile, the Data Views Generic Hits, Component Metadata, Image and Geolocation are defined in this specification. Endpoints MAY define custom Data Views, but Clients conforming to the basic profile MAY choose to ignore them. The Generic Hits Data View is mandatory, thus all Endpoints MUST provide hits represented in the Generic Hits Data View.

NOTE: The examples in the following sections show only the payload with the enclosing <fcs:DataView> element of a Data View. Of course, the Data View must be embedded either in a <fcs:Resource> or a <fcs:ResourceFragment> element. The @pid and @ref attributes have been omitted for all inline payload types.

Generic Hits (HITS)

Description	The representation of the hit
MIME type	`application/x-clarin-fcs-hits+xml`
Payload Disposition	inline

The Generic Hits Data View contains the serialization of a search result hit. It supports multiple markers for suppling highlighting for the hit. Each hit SHOULD be presented within the context of a complete sentence. If that is not possible due to the nature of the type of the resource, the the Endpoint SHALL provide an equivalent reasonable unit of context (e.g. within a phrase of a orthographic transcription of an utterance). All Endpoints MUST provide hits represented in this Data View. The XML fragment of the Generic Hits payload MUST be valid according to the XML schema "DataView-Hits.xsd " (download ).

Example (single hit marker):

<!-- potential @pid and @ref attributes omitted -->
<fcs:DataView type="application/x-clarin-fcs-hits+xml">
  <hits:Result xmlns:hits="http://clarin.eu/fcs/1.0/hits">
    The quick brown <hits:Hit>fox</hits:Hit> jumps over the lazy dog.
  </hits:Result>
</fcs:DataView>

Example (multiple hit markers):

<!-- potential @pid and @ref attributes omitted -->
<fcs:DataView type="application/x-clarin-fcs-hits+xml">
  <hits:Result xmlns:hits="http://clarin.eu/fcs/1.0/hits">
    The quick brown <hits:Hit>fox</hits:Hit> jumps over the lazy <hits:Hit>dog</hits:Hit>.
  </hits:Result>
</fcs:DataView>

Component Metadata (CMDI)

Description	A CMDI metadata record
MIME type	`application/x-cmdi+xml`
Payload Disposition	inline or reference

The Component Metadata Data View allows to embed a CMDI metadata record that applicable to the specific context into the Endpoint response, e.g. metadata about the resource in which the hit was produced. If this CMDI record is applicable for the entire Resource, is SHOULD be put in a <fcs:DataView> element below the <fcs:Resource> element. If it is applicable to the Resource Fragment, i.e. it contains more specialized metadata than the metadata for the encompassing resource, it SHOULD be put in a <fcs:DataView> element below the <fcs:ResourceFragment> element. Endpoints SHOULD provide the payload inline, but Endpoints MAY also use the reference method. If an Endpoint uses the reference method, the CMDI metadata record MUST be downloadable without any restrictions.

Example (inline):

<!-- potential @pid and @ref attributes omitted -->
<fcs:DataView type="application/x-cmdi+xml">
  <cmdi:CMD xmlns:cmdi="http://www.clarin.eu/cmd/" CMDVersion="1.1">
    <!-- content omitted -->
  </cmdi:CMD>
</fcs:DataView>

Example (referenced):

<!-- potential @pid attribute omitted -->
<fcs:DataView type="application/x-cmdi+xml" ref="http://repos.example.org/resources/4711/0815.cmdi" />

Images (IMG)

Description	An image related to the hit
MIME type	`image/png`, `image/jpeg`, `image/gif`, `image/svg+xml`
Payload Disposition	reference

The Image Data View allows top provide an image, that is relevant to the hit, e.g. a facsimile of the source of a transcription. Endpoints MUST provide the payload by the reference method and the image file SHOULD be downloadable without any restrictions.

Example:

<!-- potential @pid attribute omitted -->
<fcs:DataView type="image/png" ref="http://repos.example.org/resources/4711/0815.png" />

Geolocation (GEO)

Description	An geographic location related to the hit
MIME type	`application/vnd.google-earth.kml+xml`
Payload Disposition	inline

The Geolocation Data View allows to geolocalize a hit. If MUST be encoded using the XML representation of the Keyhole Markup Language (KML). The KML fragment MUST comply with the specification as defined by KML.

Example:

<!-- potential @pid and @ref attributes omitted -->
<fcs:DataView type="application/vnd.google-earth.kml+xml">
  <kml:kml xmlns:kml="http://www.opengis.net/kml/2.2">
    <kml:Placemark>
      <kml:name>IDS Mannheim</kml:name>
      <kml:description>Institut für Deutsche Sprache, R5 6-13, 68161 Mannheim, Germany</kml:description>
      <kml:Point>
        <kml:coordinates>8.4719510,49.4883700,0</kml:coordinates>
      </kml:Point>
    </kml:Placemark>
  </kml:kml>
</fcs:DataView>

Endpoint Description

Endpoints need to provide information about their capabilities to support auto-configuration of Clients, This capabilities include, among other information, the Profile that is supported by the Endpoint. The Endpoint Description mechanism provides the necessary facility to provide this information to the Clients. Endpoints MUST encode their capabilities using an XML format and embed this information into the SRU/CQL protocol as described in section Operation ''explain''. The XML fragment generated by the Endpoint for the Endpoint Description MUST be valid according to the XML schema "Endpoint-Description.xsd " (download ).

The XML fragment for Endpoint Description is encoded as an <ed:EndpointDescription> element, that contains the following children:

one <ed:Profile> element (REQUIRED)
The content of the <ed:Profile> element indicates the Profile, that is supported by the Endpoint.
Valid values are:
- basic: the Endpoint supports the basic Profile
NOTE: a future CLARIN-FCS specification will introduce more values.
one <ed:SupportedDataViews> (REQUIRED)
A list of Data Views, that are supported by this Endpoint. This list is composed of one or more <ed:SupportedDataView> elements. The content of a <ed:SupportedDataView> MUST be the MIME type of a supported Data View, e.g. application/x-clarin-fcs-hits+xml.
one <ed:Collections> element (REQUIRED)
A list of (top-level) collections that are available at the Endpoint. The <ed:Collections> element contains one or more <ed:Collection> elements (see below). The Endpoint MUST declare at least one (top-level) collection.

The <ed:Collection> element contains a detailed description of a collection that is available at the Endpoint. A collection is a searchable entity, e.g. a single corpus. The <ed:Collection> has a mandatory @pid attribute, that contains persistent identifier of the collection. This value MUST be the same as the MdSelfLink of the CMDI record describing the collection. The <ed:Collection> element contains the following children:

one or more <ed:Title> elements (REQUIRED)
A human readable title for the collection. A REQUIRED @xml:lang attribute indicates the language of the title. An English version of the title is REQUIRED. The list of titles MUST NOT contain duplicate entries for the same language.
zero or more <ed:Description> elements (OPTIONAL)
An optional human-readable description of the collection. Is SHOULD be at most one sentence. A REQUIRED @xml:lang attribute indicates the language of the description. If supplied, an English version of the description is REQUIRED. The list of descriptions MUST NOT contain duplicate entries for the same language.
zero or one <ed:LandingPageURI> element (OPTIONAL)
A link to a website for this collection, e.g. a landing page for a collection, i.e. a web-site that describes a corpus.
one <ed:Languages> element (REQUIRED)
The (relevant) languages available within the collection. The <ed:Languages> element contains one or more <ed:Language> elements. The content of a <ed:Language> element MUST be a ISO 639-3 three letter language code. This element should be repeated for all languages (relevant) available within the collection, however this list MUST NOT contain duplicate entries.
zero or one <ed:Collections> element (OPTIONAL)
If a collection has searchable sub-collections the Endpoint MUST supply additional finer grained collection elements, which are wrapped in a <ed:Collections> element. A sub-collection is a searchable entity within a collection, e.g. a sub-corpus.

Example 4:

<ed:EndpointDescription xmlns:ed="http://clarin.eu/fcs/1.0/endpoint-description">
  <ed:Profile>basic</ed:Profile>
  <ed:SupportedDataViews>
    <ed:SupportedDataView>application/x-clarin-fcs-hits+xml</ed:SupportedDataView>
  </ed:SupportedDataViews>
  <ed:Collections>
    <!-- just one top-level collection at the Endpoint -->
    <ed:Collection pid="http://hdl.handle.net/4711/0815">
      <ed:Title xml:lang="de">Goethe Corpus</ed:Title>
      <ed:Title xml:lang="en">Goethe Korpus</ed:Title>
      <ed:Description xml:lang="de">Der Goethe Korpus des IDS Mannheim.</ed:Description>
      <ed:Description xml:lang="en">The Goethe corpus of IDS Mannheim.</ed:Description>
      <ed:LandingPageURI>http://repos.example.org/corpus1.html</ed:LandingPageURI>
      <ed:Languages>
        <ed:Language>deu</ed:Language>
      </ed:Languages>
    </ed:Collection>
  </ed:Collections>
</ed:EndpointDescription>

This example shows a simple Endpoint Description for an Endpoint that supports the basic Profile and only provides the Generic Hits Data View. It only provides one top-level collection identified by the persistent identifier http://hdl.handle.net/4711/0815. The collection a title as well as a description in German and English. A landing page is located at http://repos.example.org/corpus1.html. The searchable collection contents are only available in German.

Example 5:

<ed:EndpointDescription xmlns:ed="http://clarin.eu/fcs/1.0/endpoint-description">
  <ed:Profile>basic</ed:Profile>
  <ed:SupportedDataViews>
    <ed:SupportedDataView>application/x-clarin-fcs-hits+xml</ed:SupportedDataView>
    <ed:SupportedDataView>application/x-cmdi+xml</ed:SupportedDataView>
  </ed:SupportedDataViews>
  <ed:Collections>
    <!-- top-level collection 1 -->
    <ed:Collection pid="http://hdl.handle.net/4711/0815">
      <ed:Title xml:lang="de">Goethe Corpus</ed:Title>
      <ed:Title xml:lang="en">Goethe Korpus</ed:Title>
      <ed:Description xml:lang="de">Der Goethe Korpus des IDS Mannheim.</ed:Description>
      <ed:Description xml:lang="en">The Goethe corpus of IDS Mannheim.</ed:Description>
      <ed:LandingPageURI>http://repos.example.org/corpus1.html</ed:LandingPageURI>
      <ed:Languages>
        <ed:Language>deu</ed:Language>
      </ed:Languages>
    </ed:Collection>
    <!-- top-level collection 2 -->
    <ed:Collection pid="http://hdl.handle.net/4711/0816">
      <ed:Title xml:lang="de">Mannheimer Morgen newspaper Corpus</ed:Title>
      <ed:Title xml:lang="en">Zeitungskorpus des Mannheimer Morgen</ed:Title>
      <ed:LandingPageURI>http://repos.example.org/corpus2.html</ed:LandingPageURI>
      <ed:Languages>
        <ed:Language>deu</ed:Language>
      </ed:Languages>
      <ed:Collections>
        <!-- sub-collection 1 of top-level collection 2 -->
        <ed:Collection pid="http://hdl.handle.net/4711/0816-1">
          <ed:Title xml:lang="de">Mannheimer Morgen newspaper Corpus (before 1990)</ed:Title>
          <ed:Title xml:lang="en">Zeitungskorpus des Mannheimer Morgen (vor 1990)</ed:Title>
          <ed:LandingPageURI>http://repos.example.org/corpus2.html#sub1</ed:LandingPageURI>
          <ed:Languages>
            <ed:Language>deu</ed:Language>
          </ed:Languages>
        </ed:Collection>
        <!-- sub-collection 2 of top-level collection 2 -->
        <ed:Collection pid="http://hdl.handle.net/4711/0816-2">
          <ed:Title xml:lang="de">Mannheimer Morgen newspaper Corpus (after 1990)</ed:Title>
          <ed:Title xml:lang="en">Zeitungskorpus des Mannheimer Morgen (nach 1990)</ed:Title>
          <ed:LandingPageURI>http://repos.example.org/corpus2.html#sub2</ed:LandingPageURI>
          <ed:Languages>
            <ed:Language>deu</ed:Language>
          </ed:Languages>
        </ed:Collection>
      </ed:Collections>
    </ed:Collection>
  </ed:Collections>
</ed:EndpointDescription>

This more complex example show an Endpoint Description for an Endpoint that, similar to Example 4, supports the basic profile. In addition to the Generic Hits Data View it also supports CMDI the CMDI Data View. The Endpoint has two top-level collections (identified by the persistent identifiers http://hdl.handle.net/4711/0815 and http://hdl.handle.net/4711/0816. The second top-level collection has two sub-collections, identified by the persistent identifier http://hdl.handle.net/4711/0816-1 and http://hdl.handle.net/4711/0816-2. All collections are described using several properties, like title, description, etc.

Endpoint Custom Extensions

The Endpoint can add custom extensions, i.e custom data, to the Result Format. This extension mechanism can for example be used to provide hints for an (XSLT/XQuery) application that works directly on CLARIN-FCS, e.g. to allow it to generate back and forward links to navigate in a result set.

The Endpoint MAY add arbitrary XML fragments to the extension hooks provided in the <fcs:Resource> element (see the XML schema for "Resource.xsd"). The Endpoint MUST use a custom XML namespace name for the extension. The Endpoint MUST NOT use XML namespace names, that start with the prefixes http://clarin.eu, http://www.clarin.eu/, https://clarin.eu or http://www.clarin.eu/.

The Client MUST ignore any custom extensions it does not understand and skip over these XML fragments when parsing the Endpoint's response.

The appendix contains an example, how an extension could be implemented.

CLARIN-FCS to SRU/CQL binding

SRU/CQL

SRU (Search/Retrieve via URL) specifies a general communication protocol for searching and retrieving records and the CQL (Contextual Query Language) specifies a extensible query language. CLARIN-FCS is built on SRU 1.2. A subsequent specification may be built on SRU 2.0.

Endpoints and Clients MUST implement the SRU/CQL protocol suite as defined in OASIS-SRU-Overview, OASIS-SRU-APD, OASIS-CQL, SRU-Explain, SRU-Scan, especially with respect to:

Data Model,
Query Model,
Processing Model,
Result Set Model, and
Diagnostics Model

Endpoints and Clients MUST use the implement the APD Binding for SRU 1.2, as defined in OASIS-SRU-12. Endpoints and Clients MAY implement APD binding for version 1.1 or version 2.0.

Endpoints and Clients MUST use the following XML namespace names (namespace URIs) for serializing responses:

http://www.loc.gov/zing/srw/ for SRU response documents, and
http://www.loc.gov/zing/srw/diagnostic/ for diagnostics within SRU response documents.

CLARIN-FCS deviates from the OASIS specification OASIS-SRU-Overview and OASIS-SRU-12 to ensure backwards comparability with SRU 1.2 services as they where defined by the LOC-SRU12.

Endpoints or Clients MUST support CQL conformance Level 2 (as defined in OASIS-CQL, section 6), i.e. be able to parse (Endpoints) or serialize (Clients) all of CQL and respond with appropriate error messages to the search/retrieve protocol interface.

NOTE: this does not imply, that Endpoints are required to support all of CQL, but rather that they are able to parse all of CQL and generate the appropriate error message, if a query includes a feature they do not support.

Endpoints MUST generate diagnostics according to OASIS-SRU-12, Appendix C for error conditions or to indicate unsupported features. Unfortunately, the OASIS specification does not provides a comprehensive list of diagnostics for CQL related errors. Therefore, Endpoints MUST use diagnostics from LOC-DIAG, section "Diagnostics Relating to CQL" for CQL related errors.

Operation explain

The explain operation of the SRU protocol serves to announce server capabilities and to allows clients to configure themselves automatically. This operation is used similarly.

The Endpoint MUST respond to a explain request by a proper explain response. As per SRU-Explain, the response MUST contain one <sru:record> element that contains an SRU Explain record. The <sru:recordSchema> element MUST contain the literal http://explain.z3950.org/dtd/2.0/, i.e. the official identifier for Explain records.

According to the Profile supported by the Endpoint the Explain record MUST contain the following elements:

Basic Profile: <zr:serverInfo> as defined in SRU-Explain (REQUIRED)
<zr:databaseInfo> as defined in SRU-Explain (REQUIRED)
<zr:schemaInfo> as defined in SRU-Explain (REQUIRED). This element MUST contain an element <zr:schema> with an @identifier attribute with a value of http://clarin.eu/fcs/1.0 and an @name attribute with a value of fcs.
<zr:configInfo> is OPTIONAL`
An extended profile may define how the <zr:indexInfo> element is to be used, therefore it NOT RECOMMENDED for Endpoints to define custom extensions.
Extended Profile: NOTE: the extended profile is not yet defined and will be part of a future CLARIN-FCS specification.

To support auto-configuration in CLARIN-FCS, the Endpoint provide an Endpoint Description. The Endpoint Description is included in explain response utilizing SRUs extension mechanism, i.e. by embedding an XML fragment into the <sru:extraResponseData> element. The Endpoint MUST include the Endpoint Description only if the Client performs an explain request with the extra request parameter x-clarin-fcs-endpoint-description with a value of true. If the Client performs an explain request without supplying this extra request parameter the Endpoint MUST NOT include the Endpoint Description. The format of the Endpoint Description XML fragment is defined in Endpoint Description.

The following example shows a request and response to an explain request with added extra request parameter x-clarin-fcs-endpoint-description:

HTTP GET request: Client -> Endpoint:

http://repos.example.org/fcs-endpoint?operation=explain&version=1.2&x-clarin-fcs-endpoint-description=true

HTTP Response: Endpoint -> Client:

<?xml version='1.0' encoding='utf-8'?>
<sru:explainResponse xmlns:sru="http://www.loc.gov/zing/srw/">
  <sru:version>1.2</sru:version>
  <sru:record>
    <sru:recordSchema>http://explain.z3950.org/dtd/2.0/</sru:recordSchema>
    <sru:recordPacking>xml</sru:recordPacking>
    <sru:recordData>
      <zr:explain xmlns:zr="http://explain.z3950.org/dtd/2.0/">
        <!-- <zr:serverInfo > is REQUIRED -->
        <zr:serverInfo protocol="SRU" version="1.2" transport="http">
          <zr:host>repos.example.org</zr:host>
          <zr:port>80</zr:port>
          <zr:database>sru</zr:database>
        </zr:serverInfo>
        <!-- <zr:databaseInfo> is REQUIRED -->
        <zr:databaseInfo>
          <zr:title lang="de">Goethe Corpus</zr:title>
          <zr:title lang="en" primary="true">Goethe Korpus</zr:title>
          <zr:description lang="de">Der Goethe Korpus des IDS Mannheim.</zr:description>
          <zr:description lang="en" primary="true">The Goethe corpus of IDS Mannheim.</zr:description>
        </zr:databaseInfo>
        <!-- <zr:configInfo> is REQUIRED -->
        <zr:schemaInfo>
          <zr:schema identifier="http://clarin.eu/fcs/1.0" name="fcs">
            <zr:title lang="en" primary="true">CLARIN Federated Content Search</zr:title>
          </zr:schema>
        </zr:schemaInfo>
        <!-- <zr:configInfo> is OPTIONAL -->
        <zr:configInfo>
          <zr:default type="numberOfRecords">250</zr:default>
          <zr:setting type="maximumRecords">1000</zr:setting>
        </zr:configInfo>
      </zr:explain>
    </sru:recordData>
  </sru:record>
  <!-- <sru:echoedExplainRequest> is OPTIONAL -->
  <sru:echoedExplainRequest>
    <sru:version>1.2</sru:version>
    <sru:baseUrl>https://clarin.ids-mannheim.de/digibibsru</sru:baseUrl>
  </sru:echoedExplainRequest>
  <sru:extraResponseData>
    <ed:EndpointDescription xmlns:ed="http://clarin.eu/fcs/1.0/endpoint-description">
      <ed:Profile>basic</ed:Profile>
      <ed:SupportedDataViews>
        <ed:SupportedDataView>application/x-clarin-fcs-hits+xml</ed:SupportedDataView>
      </ed:SupportedDataViews>
      <ed:Collections>
        <!-- just one top-level collection at the Endpoint -->
        <ed:Collection pid="http://hdl.handle.net/4711/0815">
          <ed:Title xml:lang="de">Goethe Corpus</ed:Title>
          <ed:Title xml:lang="en">Goethe Korpus</ed:Title>
          <ed:Description xml:lang="de">Der Goethe Korpus des IDS Mannheim.</ed:Description>
          <ed:Description xml:lang="en">The Goethe corpus of IDS Mannheim.</ed:Description>
          <ed:LandingPageURI>http://repos.example.org/corpus1.html</ed:LandingPageURI>
          <ed:Languages>
            <ed:Language>deu</ed:Language>
          </ed:Languages>
        </ed:Collection>
      </ed:Collections>
    </ed:EndpointDescription>
  </sru:extraResponseData>
</sru:explainResponse>

Operation scan

The scan operation of the SRU protocol is currently not used in the basic profile of CLARIN-FCS. An extended profile may use this operation, therefore it NOT RECOMMENDED for Endpoints to define custom extensions that use this operation.

Operation searchRetrieve

The searchRetrieve operation of the SRU protocol is used for searching in the Resources that are provided by the Endpoint. The SRU protocol defines the serialization of request and response formats in OASIS-SRU-12. In SRU, search result hits are encoded down to a record level, i.e. the <sru:record> element, and SRU allows records to be serialized in various formats, so called record schemas.

Endpoints MUST support the CLARIN-FCS record schema (see section Result Format) and MUST use the value http://clarin.eu/fcs/1.0 for the responseItemType ("record schema identifier"). Endpoints MUST represent exactly one hit within the Resource as one SRU record, i.e. <sru:record> element.

The following example shows a request and response to an searchRetrieve request with a term-only query for "cat":

HTTP GET request: Client -> Endpoint:

http://repos.example.org/fcs-endpoint?operation=searchRetrieve&version=1.2&query=cat

HTTP Response: Endpoint -> Client:

<?xml version='1.0' encoding='utf-8'?>
<sru:searchRetrieveResponse xmlns:sru="http://www.loc.gov/zing/srw/">
  <sru:version>1.2</sru:version>
  <sru:numberOfRecords>6</sru:numberOfRecords>
  <sru:records>
    <sru:record>
      <sru:recordSchema>http://clarin.eu/fcs/1.0</sru:recordSchema>
      <sru:recordPacking>xml</sru:recordPacking>
      <sru:recordData>
        <fcs:Resource xmlns:fcs="http://clarin.eu/fcs/1.0" pid="http://hdl.handle.net/4711/08-15">
          <fcs:ResourceFragment>
            <fcs:DataView type="application/x-clarin-fcs-hits+xml">
              <hits:Result xmlns:hits="http://clarin.eu/fcs/1.0/hits">
                The quick brown <hits:Hit>cat</hits:Hit> jumps over the lazy dog.
              </hits:Result>
            </fcs:DataView>
          </fcs:ResourceFragment>
        </fcs:Resource>
      </sru:recordData>
      <sru:recordPosition>1</sru:recordPosition>
    </sru:record>
    <!-- more <sru:records> omitted for brevity -->
  </sru:records>
  <sru:echoedSearchRetrieveRequest>
    <sru:version>1.2</sru:version>
    <sru:query>cat</sru:query>
    <sru:xQuery xmlns="http://www.loc.gov/zing/cql/xcql/">
      <searchClause>
        <index>cql.serverChoice</index>
        <relation>
          <value>=</value>
        </relation>
        <term>cat</term>
      </searchClause>
    </sru:xQuery>
    <sru:startRecord>1</sru:startRecord>
    <sru:baseUrl>http://repos.example.org/fcs-endpoint</sru:baseUrl>
  </sru:echoedSearchRetrieveRequest>
</sru:searchRetrieveResponse>

In general, the Endpoint is REQUIRED to accept an unrestricted search and MUST then perform the search operation on all Resources, that are available at the Endpoint. The Client can request the Endpoint to restrict the search to a sub-collection of these Resources. In this case, the Client MUST pass a comma-separated list of persistent identifiers in the x-clarin-fcs-context extra request parameter of the searchRetrieve request. The Endpoint MUST then restrict the search to those Resources, that are identified by the persistent identifiers passed by Client. The Client can extract all valid persistent identifiers from the @pid attribute of the <ed:Collection> element, obtained by the explain request (see section Operation ''explain'' and section Endpoint Description). The list of persistent identifiers can get extensive, but an agent MAY use the POST method instead of GET method for submitting the request.

For example, to restrict the search to the Resource with the persistent identifier http://hdl.handle.net/4711/0815 the Client must issue the following request:

http://repos.example.org/fcs-endpoint?operation=searchRetrieve&version=1.2&query=cat&x-clarin-fcs-context=http://hdl.handle.net/4711/0815

To restrict the search to the Resources with the persistent identifier http://hdl.handle.net/4711/0815 and http://hdl.handle.net/4711/0816-2 the Client must issue the following request:

http://repos.example.org/fcs-endpoint?operation=searchRetrieve&version=1.2&query=cat&x-clarin-fcs-context=http://hdl.handle.net/4711/0815,http://hdl.handle.net/4711/0816-2

If an invalid persistent identifier is passed by the Client, the Endpoint MUST issue a http://clarin.eu/fcs/1.0/diagnostic/1 diagnostic, i.e add the appropiate XML fragment to the <sru:diagnostics> element of the response. The Endpoint MAY treat this condition as fatal, i.e. just issue the diagnostic and perform no search or it MAY treat it a non-fatal and perform the search.

Normative Appendix

List of extra request parameters

The following extra request parameters are used in CLARIN-FCS:

Parameter Name	SRU operations	Allowed values	Description
`x-clarin-fcs-endpoint-description`	explain	`true` All other values are reserved an `MUST` not be used by Clients	If present, the Endpoint `MUST` include an Endpoint Description in the `<sru:extraResponseData>` element of the explain response.
`x-clarin-fcs-context`	searchRetrieve	A comma separated list of persistent identifiers	The Endpoint `MUST` restrict the search to the collections identified by the persistent identifiers

List of diagnoistics

Apart from the SRU diagnostics defined in OASIS-SRU-12, Appendix C and LOC-DIAG, the following diagnostics are used in CLARIN-FCS. The "Details Format" column specifies what SHOULD be returned in the details field. If this column is blank, the format is "undefined" and the Endpoint MAY return whatever it feels appropriate, including nothing.

Identifier URI	Description	Details Format
`http://clarin.eu/fcs/1.0/diagnostic/1`	Persistent identifier passed in for restricting the search is invalid	The offending persistent identifier

Non-normative Appendix

The following sections are non-normative.

Referring to an Endpoint from a CMDI record

Centers are encouraged to provide links to their CLARIN-FCS Endpoints in the metadata records for their resources. Other services, like the VLO, can use this information for automatically configuring an Aggregator for searching resources at the Endpoint. To refer to an Endpoint a <cmdi:ResourceProxy> with <cmdi:ResourceType> set to the value SearchService and a @mimetype attribute with a value of application/sru+xml need to be added to the CMDI record. The content of the <cmdi:ResourceRef> element must contain a URI that points to the Endpoint web service.

Example:

<cmdi:CMD xmlns:cmdi="http://www.clarin.eu/cmd/" CMDVersion="1.1">
  <cmdi:Header>
    <!-- ... -->
    <cmdi:MdSelfLink>http://hdl.handle.net/4711/0815</cmdi:MdSelfLink>
    <!-- ... -->
  </cmdi:Header>
  <cmdi:Resources>
    <cmdi:ResourceProxyList>
      <!-- ... -->
      <cmdi:ResourceProxy id="r4711">
        <cmdi:ResourceType mimetype="application/sru+xml">SearchService</cmdi:ResourceType>
        <cmdi:ResourceRef>http://repos.example.org/fcs-endpoint</cmdi:ResourceRef>
      </cmdi:ResourceProxy>
      <!-- ... -->
    </cmdi:ResourceProxyList>
  </cmdi:Resources>
  <!-- ... -->
</cmdi:CMD>

Endpoint custom extensions

The CLARIN-FCS protocol specification allows Endpoints to add custom data to their responses, e.g. to provide hints to for an (XSLT/XQuery) application that works directly on CLARIN-FCS. It could use the custom data to generate back and forward links for a GUI to navigate in a result set.

The following example illustrates how an extension can be embedded into the Result Format:

<fcs:Resource xmlns:fcs="http://clarin.eu/fcs/1.0" pid="hdl:4711/0815">
    <fcs:DataView type="application/x-clarin-fcs-hits+xml">
      <hits:Result xmlns:hits="http://clarin.eu/fcs/1.0/hits">
        The quick brown <hits:Hit>fox</hits:Hit> jumps over the lazy <hits:Hit>dog</hits:Hit>.
      </hits:Result>
    </fcs:DataView>
    
    <!--
        NOTE: this is purly fictional and only serves to demonstrate how
              to add custom extensions to the result representation
              within CLARIN-FCS.

        A hypothetical Endpoint extension for navigation in a result set:
        it basically provdes a set of hrefs, that a GUI can convert into
        navigation buttions.    
    -->
    <nav:navigation xmlns:nav="http://repos.example.org/navigation">
        <nav:curr href="http://repos.example.org/resultset/4711/4611" />
        <nav:prev href="http://repos.example.org/resultset/4711/4610" />
        <nav:next href="http://repos.example.org/resultset/4711/4612" />
    </nav:navigation>
</fcs:Resource>

Discussion

Thomas (ASV)?: In the 'Basic profile' section it is stated that 'Endpoint must not silently accept queries that include CLQ features besides term-only and terms combined...'. What would be the desired action then (issuing a diagnositic?)?
- Oliver (IDS)?: Right, the desired action is to issue a diagnostic. We need to add that information to the text. In general, if something goes wrong, Endpoints shallissue appropriate diagnostic.
Thomas (ASV)?: Do we really want to force every endpoint to support the Generic Hits dataview? For some resources there may be no 'literal text' (like audio, video, highly specific annotation formats) which could lead to some ugly string serialization instead of using custom extensions and which would leave the client with the interpretation of the output.
- Oliver (IDS)?: I guess so, that's the lowest common denominator for an aggregator. We had the KWIC data view in the old specification, that served the same purpose. However, the latter does not support multiple hits. IMHO, we need to keep this common denominator, even if this is just an approximation for other resource types. Endpoints are always free to also add a more appropriate data view for the resource to their response.
Thomas (ASV)?: Do we want to specify additional payloads for resource types like audio and video (referenced, attributes: mimetype + optional offset information)?
- Oliver (IDS)?: Yes, we certainly can do that, however I would need more input for that.
Thomas (ASV)?: Is it a good idea to specify the capabilities of an endpoint by using profile names (where only one is allowed in the explain response)? In the past it was planed to use some 'feature matrix' to reflect the differences in supported functionality for the endpoints. This would allow to explicitly state which subset of possible functionalities (besides the basic stuff) is supported by an endpoint.
- Oliver (IDS)?: The question is, what different functionality do we want to differentiate. And, of course, we need to encode these features for Clients. I think, the basic profile is quite clear in what Endpoints need to support. All the "interesting" (query expansion, data cats, etc) bits are nowhere clear enough to have them specified. I want to do this in the yet-to-be defined "extended" profile, which could also foresee such a matrix. That could be, e .g. an additional list of <feature> elements that will be part of the endpoint description. However, I want to keep the distinction between a basic profile, which people can implement right ways, and more advanced stuff. We really need to have a normative spec.
  - Thomas (ASV)?: I absolutly agree. Just wanted to point out that maybe we already can think of how these features will be encoded (without a decision of what features will make it into the future extended specification).
    - Oliver (IDS)?: We could already prepare for for feature encoding, by adding adding a <feature> elements to the Endpoint Description. Or better call the capabilities, so something along the lines of:
      <ed:EndpointDescription xmlns:ed="http://clarin.eu/fcs/1.0/endpoint-description"> <ed:Profile>basic</ed:Profile> <ed:Capabilities>  <ed:Capability>http://clarin.eu/fcs/1.0/feature/basic-search</ed:Capability>  <ed:Capability>http://clarin.eu/fcs/1.0/feature/query-expansion</ed:Capability> </ed:Capabilities>  </ed:EndpointDescription>
      If we do this, it to be decided, if we want to keep the <Profile> element or if we decide, that this information is redundant and a profiles will require a certain set of capabilities. However, if we ditch <Profile>, it's not so easy for a Client to decide spot-on what profile is supported by the Endpoint.
      - Thomas (ASV)?: I am not sure if this is a problem, but we could interpret "Capabilities" just as a verbose listing of all capabilities of the endpoint. The <ed:Profile> is just the shorter version regarding our definition of what is "basic" and what is "extended". That way our standard aggregator can just work with the profile name, whereas other clients (that don't care how we interpret these terms) could look if an endpoint supports all functionality they need.
Thomas (ASV)?: The endpoint specification contains information about supported profile and dataviews. This is no problem for an endpoint with only one resource or homogenous resources. When we extend the profile by adding information about annotation tiers this may not be applicable for all provided resources (like an endpoint that provides two corpora, where only one is annotated with POS tags). So maybe some of this information is not specific for the endpoint but for the resource.
- Oliver (IDS)?: My aim was to consider the list of data views as a hint, what data views are available at an Endpoint. The semantics is not, that every resource/collections supports all these data views. It's a little influenced by what SRU does in explain with <zr:schemaInfo>. We could keep it this way, remove it of extend it, so this information can be given on resource/collection level
  - Thomas (ASV)?: Right. We should at least think about it as this information could be quite relevant for the user experience in an aggregator.
    - Oliver (IDS)?: Yes, so do you think we should mark "supported dataviews" per collection/resource? Something along the lines like:
      <ed:EndpointDescription xmlns:ed="http://clarin.eu/fcs/1.0/endpoint-description"> <ed:Profile>basic</ed:Profile> <ed:SupportedDataViews> <ed:SupportedDataView id="dv1">application/x-clarin-fcs-hits+xml</ed:SupportedDataView> <ed:SupportedDataView id="dv2">application/x-cmdi+xml</ed:SupportedDataView> <ed:SupportedDataView id="dv3">image/png</ed:SupportedDataView> </ed:SupportedDataViews> <ed:Collections> <ed:Collection pid="http://hdl.handle.net/4711/0815">  <ed:SupportedDataViews ref="dv1" /> </ed:Collection> <ed:Collection pid="http://hdl.handle.net/4711/0816">  <ed:SupportedDataViews ref="dv1 dv2" /> </ed:Collection> <ed:Collection pid="http://hdl.handle.net/1">  <ed:SupportedDataViews ref="dv1 dv2 dv3" /> <ed:Collections> <ed:Collection pid="http://hdl.handle.net/1/1">  <ed:SupportedDataViews ref="dv1 dv2" /> </ed:Collection> <ed:Collection pid="http://hdl.handle.net/1/2">  <ed:SupportedDataViews ref="dv1 dv3" /> </ed:Collection> </ed:Collections> </ed:Collection> </ed:Collections> </ed:EndpointDescription>
      What about Collections with Sub-Collections? Parent collection would not indicate the supported data views or it would be the union of supported dataviews of their child-collections? Is there a use-case where collection http://hdl.handle.net/1 contains more data (= resources) then union of http://hdl.handle.net/1/1 and http://hdl.handle.net/1/2, i.e. there are resources "in between" the child-collection and it's parent? If so, could they have "conflicting" dataviews?
      - Thomas (ASV)?: I am not sure about that these "invisible" resources. But in general we could think of this as normal inheritance. Every root collection element could specify the minimal set of supported dataviews for all daughter nodes (or if missing it is assumed that all entries in SupportedDataViews? are supported). Every node in the sub-collection tree can overwrite this configuration. When we have Hits as mandatory dataview even for otherwise disjoint sets of dataviews in the sub-collections the root collection can at least provide Hits for everything.
    - Oliver (IDS)?: Shall we foresee some mechanism for Clients to tell the Endpoint in what data views they are interested? E.g. the Endpoint may support Geolocation but the Client does not care or support it and could ask the Endpoint at query time to not serialize the Geolocation data view (= less wasted bytes send over the network).
      - Thomas (ASV)?: I think that this a good idea. Maybe in some cases it could also be useful to provide dataviews only if they are explicitly requested. This could allow adding data views that are too "expensive" (computational or regarding bandwith) to generate for every request.
Thomas (ASV)?: The current (old) solution for exposing granularity and structure of supported collections is a multiple-staged mechanism: the client queries for the first-level structure (=collections) and can explicitly ask the endpoint to give additional information about the internal structure of these collections (and so on...). This is very helpful for endpoints which support queries on detailled subcollections. The proposed solution above would force the endpoint to expose the complete structure of all provided resources in the explain response, which would lead (for example for the endpoint in Leipzig) to very large responses.
- Oliver (IDS)?: True, but the old approach is overly complex. If the response is large (> 100MB), so be it. An efficient Endpoint implementation should do an streaming approach when serializing the response and the Client should not assume, that the response to this information will fit in, let's say,1MB of memory. If it's is hard for the endpoint to compile this list (e.g. it requires complex database queries), it's IMHO again, a matter of the endpoint to cache this information (in memory or disk) and just stream it into the explain response.
Thomas (ASV)?: For endpoints that support many collections it can be a problem to force them to allow unrestricted queries regarding resource selection. In these cases every query would lead to a high load on the local search engine. Instead I would prefer to at least allow using a default collection if no restrictions are provided by the client.
- Oliver (IDS)?: This has already been the case with then old specification. That's more more in line with the searchRetrieve semantics of SRU. We should discuss this within the Taskforce.

Download in other formats:

Plain Text