Public Deliverables

D7.1.3 Report on Dissemination and Standardisation activities Y3 (May 2014)

This document is a report on dissemination and standardisation activities carried out by the project partners during the final six months of the project. Dissemination covers scientific dissemination, the project cooperation with its Stakeholder Board, as well as external communication and liaisons to other projects. 7 public deliverables have been delivered during the final 6 months. 10 papers and articles have been released in international publications. Project partners have made presentations, presented posters and participated in demonstrations in 7 international symposiums, conferences or workshops (5 further conference papers are accepted for publication after the end of the project) and 3 journal articles have been submitted and we are being reviewed. The work on standardisation has continued during the final 6 months. Several contributions have been made to MPEG and the EBU-AMWA FIMS project. Several TOSCA-MP partners represent the project by participating in meetings to promote the solutions developed by the project.

D4.5 Final Version of Benchmarking Methods (April 2014)

This deliverable discusses how formal task models can be used for the assessment of content annotation and search tools, both on component and system level. It presents an approach for cost assessment and provides experimental results using the proposed methods. It also describes the integration of benchmarking tool into an analysis workflows. Media production processes can benefit from the use of automatic information extraction tools that analyse multimedia content and provide information for content description, indexing and search. However, it is difficult to assess the impact of a specific information extraction tool (e.g., genre classification) on the overall process in terms of quality improvements or cost savings w.r.t. manual processes. One observation is that existing benchmarks of individual components do not always reflect the applicability of the methods for a certain task in a process. In order to address this issue, we propose to assess tools in the context of a specific task of a real media production workflow, rather than evaluating these tools in an isolated lab setting. In this document, we discuss issues of specifying and representing ground truth. The basic approach is to consider a set of edit operations between two metadata documents (of which one might be part of ground truth). These edit operations and related costs can be used to model also cases with multiple choices for ground truth or ground truth items with different confidence levels. We describe the use of task models for benchmarking and simulation. For benchmarking, we model error propagation in content analysis and search tasks. We provide a number of experimental results that show that the approach can be also used in cases where the ground truth is incomplete or no ground truth but only generated results are available for intermediate steps. For simulation, the task model is transformed into a business process, and costs are estimated based on different parameterisation of the process. Finally we discuss the integration of the proposed benchmarking and simulation approaches with the service oriented architecture of TOSCA-MP.

D6.6 Final Report on Integration (April 2014)

This document describes the integrated Proof of Concept (PoC) as it was realised by TOSCA-MP based on the system architecture (as described in D6.3) following the principles of Service-oriented Architecture (SOA). The integration process focussed on the realisation of the so-called Configurable Analysis Task Model as a means of combining the individual formal task models of several usage scenarios in a single analysis workflow, to avoid unnecessary duplication at runtime. The technical integration of the components developed in WP2, WP3, WP5 and WP6 was an iterative process which used formal descriptions of all components and their interfaces to facilitate extensive testing before the actual integration of the web service-based components into the PoC. The technical components of which the PoC is composed are described as well; for components developed in WP2, WP3 and WP5, an overview is given, focussing on the relevant aspects with respect to integration in the PoC; for details the reader is kindly referred to deliverables of the respective WPs. Specifically, the web interfaces for these components are addressed here, as these form the technical basis for their integration in the PoC. The Metadata Production Management Framework (MPMF) as the core component for the integration of components as services, and for orchestration and execution of media analysis processes, is described in detail. It has been realised based on the Red Hat JBoss® Enterprise Application Platform, and contains a multitude of TOSCA-MP specific solutions for a flexible workflow configuration, pre-processing steps (before the actual invocation of analysis services), process logic and an extensive error handling and logging. For the handling of essence and metadata in the distributed PoC, a solution based on URL-exchange between MPMF and analysis services was implemented. The Graphical User Interfaces (GUI) that provide the means for various users to interact with the system, either to control or configure it (administration) or to actually use it (search, annotate etc.) are presented. Each GUI makes use of suitable services to provide its functionality and by doing so serves the realisation of the project’s business goals. For each GUI it is described, how it fits in the respective workflow. The integrated PoC is a fully functional prototype which has already been used on multiple occasions for demonstration and dissemination purposes (e.g. during the project’s field trials). The project partners aim at having it available for a year after the project’s finalisation, to facilitate further dissemination of the results.

D3.4 Networked Media Search Engine, Visualisation and User Feedback v3 (April 2014, May 2014)

D3.4 describes the third and final version of the service-based semantically driven search engine for audio-visual content, the new paradigms for results presentation, and definition of implicit and explicit feedback mechanisms. Firstly, the document provides an incremental and final description of the technical information with regards to the Networked Media Search Engine, including the indexing and search services. Secondly, it describes the user interface for presenting the results of the Networked Search Engine including some new paradigms supporting the clustering of search results and the visual summary of events. Thirdly, it provides information about the exploitation of the implicit and explicit mechanisms to modify the ranking of results.

D2.3 Metadata Extraction, Enrichment and Linking V3 (March 2014, May 2014)

This document reports the work on automatic metadata extraction and enrichment in TOSCA-MP. The reporting period is M20 to the publication date of the present deliverable. Addressed are the contributions to automatic speech metadata extraction, automatic visual metadata extraction, semantic enrichment and linking as well as multimodal content modelling. Furthermore, the deliverable contains an analysis that shows how the technologies investigated and developed in WP2 and other work packages have been selected, with the high-level business goals defined in WP6 early in the project as a starting point. This summarizes information presented in Deliverables D6.1, D6.2 and D2.2.

D6.4.3 Field Trial Report V2 (March 2014)

This document describes the results of the project’s final field trials (also called TOSCA-MP demonstration & test workshop) that were arranged partly at RAI on January 20-21, 2014 and partly at VRT on January 23-24. There were a total of 23 participants to this workshop. Most participants were professionals active in a media archives, or were professional researchers in broadcasters R&D department. The field trials have been organised on a half-day basis. In each of the two locations three half-day sessions have been produced. Each half-day session day was organised as follows. First, the participants received a presentation explaining the goals of the TOSCA-MP project. The consortium was introduced as well as the objectives of the project. Next, the four exercises arranged for the trials were presented. After these general introductions, the participants were asked to try one by one the exercises proposed in each demonstration station. The four exercises proposed for this field trials were: Extend the Results of Content Search Exploiting Multilingual Repositories. Efficient Use of Distributed Workflows for Media Production. Access and Update Rich Annotations for Archived Assets. Assisted Production of Sports Events. The four exercises were built integrating components developed by the project’s partners, making use of the DRF and MPMF subsystems. In order to gather user feedback, a comprehensive questionnaire was prepared with different sections associated to one or more parts of the demonstration. The participants were invited to provide their feedback on these forms. A detailed analysis of their feedback can be found in Section 3 of this deliverable. The general feedback of the participants was quite positive and a lot of feedback has been collected through the questionnaires. The key functionalities of the developed tools shown in the exercises were well received, and the usefulness and meaningfulness of the exercises was acknowledged. The substantive amount of feedback on how to improve the functional aspects of the presented use cases indicates, on the one hand, that, as they are, the developed technologies are not yet ready to be employed in real production, but on the other hand shows very high interest from the professional media production community in their potential usage in current and future scenarios. As a consequence, the impact of receiving this high quality feedback is also particularly relevant to instruct the analysis of the possible exploitation of the project results.

D4.2.3 Test Material Including Ground Truth v3 (February 2014)

This document accompanies the release of version 3 of the mammie platform. Mammie is a platform which is used to distribute test media material in the TOSCA-MP consortium. It is available through a website located at http://tosca-mp.vrt.be/toscamp/. The mammie platform is a web interface to a lightweight media asset management system. It contains a few basic MAM functions, such as access control, ingest, update and download of media assets and the corresponding metadata streams. The metadata is searchable using a basic search engine. This deliverable contains all material that was already present in version 2. In this version, some more content was added and ground truth annotations have been provided for some of the content by different partners. The mammie platform currently contains 5,429 videos which have been provided by the different content providers. This provides a total duration of nearly 1,900 hours of high resolution broadcast video currently available on the platform. The contributions of the different partners namely IRT, RAI and VRT are distributed as follows:

Partner	Number of media items	Total duration
IRT	2923	597 hours
RAI	1541	845 hours
VRT	965	438 hours

The platform is accompanied by a material contract, which each TOSCA-MP partner has to sign before requesting access to the platform. Furthermore, ground truth material has been made available. For a selected portion of the content, speech transcriptions, text translations, visual concepts, genre classification, quality control and shot boundary segmentation has been annotated and made available to the consortium. We refer to section 4.3 for detailed statistics.

D6.5 Proof-of-Concept Integration (December 2013)

D6.5 consists of the prototype that demonstrates the technologies developed in TOSCA-MP. This prototype will be used for dissemination activities and consists of various software tools that have been developed within the project and which have been integrated in a prototypical system called “TOSCA-MP Proof of Concept”. As accompanying information to the prototype, this document gives an overview of the functionality of the integrated Proof of Concept and the TOSCA-MP demonstrator components as they will be available for dissemination activities, amongst others during the TOSCA-MP Field Trials in January 2014. The system integration will be covered in more technical detail in deliverable D6.6 “Final report on Integration” (due in M30 of the project). Relevant links to the corresponding (public) documents and information sources are also given in this document for additional reference.

D 7.1.2 Report on Dissemination and Standardisation Activities Y2 (September 2013)

This document is a report on dissemination and standardisation activities carried out by the project partners during the first year of the project. Dissemination covers the project cooperation with its Stakeholder Board, as well as external communication. Six public deliverables have been delivered during Year 2. Sixteen papers and articles have been released in international publications. Project partners have made presentations, presented posters and participated in demonstrations in eighteen international symposiums, conferences or workshops (three conference presentations already accepted for the next period). The work on standardisation has intensified during Year 2. Several contributions have been made to MPEG and the EBU-AMWA FIMS project also registered by SMPTE. Several TOSCA-MP partners represent the project by participating in meetings to promote the solutions developed by the project. Future dissemination plans until the end of the project are also included.

D6.4.2 Field Trial Report v1 (September 2013)

This document describes the results of the field trials (also called TOSCA-MP demonstration & test workshop) that were held at EBU on Tuesday June 4 at EBU in Geneva. There were 14 participants for this workshop. Most participants were active in a media archive, or were researchers in a broadcasters R&D department. The day was organised as follows. First, the participants received a presentation explaining the goals of the TOSCA-MP project. The consortium was presented as well as the objectives of the project. Next, the methodology of the project with business goals, scenario’s and tasks was presented. After these general presentations, the participants were split up. Some of them assessed the three software demonstrators that were available: The control & config user interface serves as a central hub for monitoring the status of content repositories and invoking workflows for automatic information extraction. The user interface for result verification and assessment visualizes results of a range of automatic video quality analysis algorithms are. The tool provides efficient means for navigating the analysis results, and verifying and correcting detected defects, as well as adding further annotations. An exploratory-based search user interface that supports different visual functionalities for query refinement and enhancement. The tool relies on Semantic Web technologies and combines content-related features and semantic descriptors to improve the accuracy and recall of the search results. It also allows search and filtering based on automatic annotations.

TOSCAMP-D2.2-RD2.2 Automatic Metadata Extraction and Enrichment (June 2013)

This document reports the work on automatic metadata extraction and enrichment in TOSCA-MP. The reporting period is M12 to the publication date of the present deliverable. Addressed are partner contributions to automatic speech metadata extraction, automatic visual metadata extraction, semantic enrichment and linking as well as multimodal content modeling. The deliverable also contains an analysis that puts into relation the developed technologies in WP2, as well as in some other work packages, with the high-level element of the top-down analysis that the TOSCA-MP partners have performed and presented in earlier deliverables.

D3.3 Networked Media Search Engine, Visualization and User Feedback v2 (June 2013)

D3.3 describes the second version of the service-based semantically driven search engine for audio-visual content, the new paradigms for results presentation, and definition of implicit and explicit feedback mechanisms. Firstly, the document completes the architecture presented in D3.2 and provides technical information regarding the exposed web services for indexing and search. Secondly, it describes the user interface for presenting the results of the Networked Search Engine including some new paradigms supporting an exploratory-based approach. Furthermore, it also provides other mechanisms for visualizing the results of feature extraction services. Thirdly, it outlines different sources of information to take into account user feedback and a method to gather dynamic rating of media assets to highlight the most interesting parts but also in relation with other related media assets. Finally, the deliverable provides details for further steps focused on the integration of some individual tools for visualization and the components of the main platform architecture (e.g. the Distributed Repository Framework and the Metadata Production Management Framework).

D6.4.1 Pilot Field Trials (May 2013)

This deliverable describes the results of TOSCA-MP pilot field trials held in February 2013. The field trials consisted of experiments on the subjective evaluation of sharpness. For this purpose, 28 test subjects had to rate videos on sharpness. Two experiments were held: one single stimulus (SS) experiment where the subjects had to state the perceived sharpness on a discrete scale and a second experiment, in which the subjects were presented two videos for which they had to rate the sharpness on a continuous scale. For half of the test subject, an eye tracking system has been used to track the eye movements of the subject during the sharpness assessment. The evaluation confirms a high correlation between the objective sharpness measure and the subjective sharpness assessment of the test subjects. The test subjects were also asked to rate their level of expertise on sharpness evaluation. From the results of the study, we can conclude that expert users tend to be more critical when sharpness is low. From the eye tracking data, it is also confirmed that non-experts tend to focus on high-saliency regions in the video, whereas experts focus on textured regions and edges. The results of this evaluation will be used to support the selection of areas for automatic sharpness detection algorithms and to adapt the sharpness metric to the specific target application. It will also be necessary to perform more trials with more test subjects in order to confirm the results described in this document.

D4.2.2 Test Material including Ground Truth v2 (February 2013)

This document accompanies the release of version 2 of the mammie platform. Mammie is a platform which is used to distribute test media material in the TOSCA-MP consortium. It is available through a website located at http://tosca-mp.lab.vrt.be/toscamp/. The mammie platform is a web interface to a lightweight media asset management system. It contains a few basic MAM functions, such as access control, ingest, update and download of media assets and the corresponding metadata streams. The metadata is searchable using a basic search engine. The mammie platform currently contains 5321 videos which have been provided by the different content providers. This provides a total duration of over 1800 hours of high resolution broadcast video currently available on the platform.

D7.1.1 Report on Dissemination and Standardisation Activities Y1 (October 2012)

D2.1 Automatic Metadata Extraction and Enrichment v1 (October 2012)

This document reports the work on automatic metadata extraction and enrichment in TOSCA-MP. The reporting period is M1 to the publication date of the present deliverable. Addressed are partner contributions to automatic speech metadata extraction, automatic visual metadata extraction, semantic enrichment and linking as well as multimodal content modeling.

D3.2 Networked Media Search Engine, Visualisation and User Feedback v1 (October 2012)

This deliverable contains a description of the different components of the first version of the networked media search engine. For this first iteration, the focus of the work is on having an early running prototype as back-end connecting the available individual components with an existing visualization. Thus, it does not yet include a connection to the TOSCA-MP MPMF. Instead, the different components are integrated point-to-point where necessary. The source for essence and basic metadata is the MAMMIE platform (D4.2.1). In this document, the REST API of MAMMIE is described. Two more metadata sources are coupled with the networked media search engine, namely concept detection and quality analysis. The last information source used for this version comes from multimodal news aggregations, where natural language techniques are used on speech transcriptions and RSS feeds in order to extract named entities and other relevant keywords and topics.

D6.3 System Design (October 2012)

D6.3 System Design – Appendix A

This document describes the overall architecture and design of the TOSCA-MP system. The system design presented in this document provides the current state of the design of the system architecture with all relevant system components of the TOSCA-MP system. The Logical System Design provides the high-level architectural definition. It shows the connections and relations between the four identified main components, i.e. the services which provide specific functionalities, a distributed data repository (DRF), the Metadata Production Management Framework (MPMF) in which an Enterprise Service Bus (ESB) builds the core component for the integration of components as services and the graphical user interfaces (GUIs).

D5.1 Overall Architecture, Interfaces and Protocols of the Distributed Repository Framework (September 2012)

This document describes the overall architecture of the Distributed Repository Framework (DRF) in detail. The overall architecture description is an update of the internal deliverable 5.1.1 taking into accounts the continuing technical discussions and additional arising requirements within the consortium. Significant impact on the update of the overall architecture had the decision to integrate MAMMIE as one external repository on one hand and the results of a survey for mandatory requirements by all partners of the consortium on the other hand. All received and feasible requirements are described under one bullet of the document.

D6.2 Requirements (June 2012)

This deliverable represents a cornerstone of the TOSCA-MP design phase, by providing a set of requirements that links the high-level business analysis phase, that ended with the completion of deliverable 6.1 (Scenarios), with the integration and development phase that will follow this document. This goal is realised through approaching the requirement analysis from a process-orientated perspective, i.e. being guided by the scenarios defined in the first phase and being targeted at identifying the components that together with the reference blocks constituted by the Distributed Repository Framework (DRF) and the Metadata Production and Management Framework (MPMF) will actually implement the scenarios. Together with refining the scope and description of the components already declared in the scenario descriptions, during the analysis several “new” components have been also identified. Together with this identification/refinement a set of behavioural and functional requirements has been performed in the context of the processes that the components are supposed to contribute to.

D4.1 Relevant Tasks in the A/V Media Production Workflow (May 2012, updated Sept 2012)

This document collects a set of real-world tasks in the media production workflow, that are considered relevant for TOSCA-MP, as they can be supported by the research performed in the project. The document collects attributes of these tasks that help to characterise them and to formalise task models and success metrics based on them. The collected information will be used for the orchestration of services performing steps needed in the tasks, and for defining benchmarks for (chains of) components needed to (partly) automate the tasks.

D3.1 State of the Art on Semantic Retrieval of AV Content beyond Text Resources (April 2012, updated Sept 2012)

This deliverable describes the state of the art in the area of semantic retrieval of multimedia (audiovisual) content beyond text resources, i.e., considering the nature of the content to be retrieved. In this line, the characteristics of images, video and audio, are exploited for improving accuracy of retrieval results.

D4.2.1 Test material including ground truth v1 (April 2012)

This document accompanies the release of the mammie platform. Mammie is a platform which is used to distribute test media material in the TOSCA-MP consortium. It is available through a website located at http://tosca-mp.vrt.be/toscamp/

D6.1 Usage Scenarios (December 2011)

This deliverable covers the description of usage scenarios of the system that the TOSCA-MP consortium will develop and demonstrate during the project’s lifetime. It also covers a set of optionally demonstrable scenarios as well as a set of scenarios used only for requirements analysis. It includes analytical considerations about relevance and coverage of planned research advancements w.r.t. identified scenarios.