Semantic Engine
The semantic engine of the i3-Market project acts as a microservice which provides data offering templates including data offering metadata, contract parameters, pricing models and dataset model. The semantic engine has been developed based on Java/Spring Boot, and the database is MongoDB. Objectives
We developed and implemented dedicated software components for Semantic Engine System as SEED, that is in charge to manage the semantic metadata, descriptions, queries, discoveries, retrieve, create and map descriptions and manipulate registries, federated queries and components interactions and interfaces. To make easier the interface and use of functionalities we present the external operations via APIs that are more agnostic and easy to use also for non-semantic experts.
Building Blocks
For the Semantic subsystems in charge of dealing with “Semantic Data Management” we can highlight parts like:
- Semantic Meta Data Storage: This component on receiving semantically annotated data push data into registry data store, for example, mongodb.
- Semantic Mapping: This component does semantic mappings and transform data (json to json-ld document) received from API endpoints.
- Vocabulary Management: This environment keeps and manages all of the vocabularies, defined as i3-Market Semantic model, used in different operations of the semantic engine.
- Offering Registration: The Semantic Engine basically REST APIs exposed as endpoints. Semantic engine exposes different endpoints for offering registration. Examples are: o register data provider, o register data offering of a data provider, o update data offerings, o deleting a data offering o query existing offerings, etc.
- Offering Discovery: The Semantic engine exposes different endpoints for offering discoveries and retrieve. Examples are: o Retrieve list of Data Offerings, o discover data offerings by Providers, o discover data offerings by parameters, o search for particular metadata, etc. The following picture shows a detailed landscape of the current set of microservices (cubes), API’s (little yellow rectangles), components (blue rectangles), and storages (white rectangles) on i3-MARKET.
How the above components interact with each other can be explained with the use of a scenario, for instance, registration of data provider. To complete this task, a data provider fills a template provided by offering registration component. The template for a Data Offering description as a JSON request is Semantically mapped by the Semantic Engine component and is forwarded towards the Semantic Meta Data Storage where this annotated data is pushed into the database. A JSON Data Offering object request is shown in the following example:
After a Semantically annotation the above JSON request is mapped to a Json-ld document for the storage in the metadata registry in MongoDB like:
Technical Requirements
For the Semantic Storage the following high-level capabilities have been defined.
Semantic Meta Data Management
The Semantic engine (SEED) relies on a registry database (MongoDB). All the information for instance, data provider, data offerings, consumer, and querying offering are stored as semantic metadata based on the i3-Market Semantic Models. Metadata Registry Database The Registry Database is responsible to store semantic meta data and process the management of metadatacomponents can access to the data.
Offering registration The Semantic engine will expose APIs to register, query and update offering. A data provider can register offering, for instance, datasets and the price for data, etc. Offering Registration is a functionality that allows the user to manage their data offering. More specifically, it provides the following functionalities: – Register the data offering – Retrieve all the offerings – Update/Delete offerings Subscribe to an offering Offering Discovery The Semantic engine will expose APIs to query the existing offerings in i3-Market backplane. A data consumer can query datasets, prices for any dataset, offering, etc. Offering Discovery is a cfunctionality that allows the Data Consumers to search the offering data available on the marketplace. The Data Consumer has to specify the characteristics of the data they are looking for. The Offering Discovery module will then process the data request and returns a list of available offering data that meet their requirements. More specifically, the Offering Discovery should provide the following functionalities: – Register the consumer data request – Retrieve all the data request of a consumer – Process a data request – Update/Delete a data request – Subscribe to a data request.
Semantic Vocabulary Management I3-Market Semantic Models Management The i3/Market Semantic Models are shared not only with Consortium Partners but also with stakeholders and community. As part of open-source assets, the vocabulary models, documentations, and files used in the i3-Market project are made available. The i3-Market Semantic Model info is the documentation that describes in detail all the taxonomies and vocabularies from needed domains used in i3-Market and that describes and represent all the relationships between them to build the graph representation of the i3-Market semantic model. • The Models Files will be shared in GitHub and GitLab public repositories with releases versions where each section contains the online machine-readable files in OWL and other format for online accessibility, the files are maintained and updated regularly to keep the latest version of the models files up to date. – View, search the concepts of i3-Market – Allow user to propose a new concept – Allow the administrator to add a new concept – Allow the administrator to update/delete an existing concept.
Created: 2022-04-06