Task models and benchmarking
Task.based benchmarking
Despite the amount of literature related to task models, few have been applied in a scenario as envisaged by TOSCA-MP. The majority purely targets the design of user interfaces, rather than tools for extracting, searching and organising content related to a task. The approaches coming from the e-learning domain might be more applicable in that context. We will analyse how the existing or similarly structured models can be applied to formalise the tasks in the audiovisual media production process. A possible approach could be transferring models for learning goals used in e-learning to models of completing tasks such as search for audiovisual content.
For benchmarking content segmentation algorithms, TOSCA-MP will advance the state of the art by also researching benchmarking methods for semantic segmentation of less structured content genres than news and sports. The binary judgement about the presence of a concept used in evaluation of concept detection does not take into account the relevance of the clip for a search containing that concept. TOSCA-MP will research how to model this information in the annotations and evaluation measures. The current evaluation methods for instance search and linking treat the problem as a per shot retrieval task, ignoring the context in the video. TOSCA-MP will investigate more meaningful evaluation measures based on the ability to detect relevant other media items. The measures used for speech metadata extraction judge the quality of the transcribed text, however, these measures do not necessarily correlate with the usefulness of the results for indexing and retrieval.
The TRECVID measure for content based copy detection includes a cost that can be tuned to a certain use case, but does not include the temporal precision. TOSCA-MP will determine values for these cost factors for actual tasks in audiovisual media production and investigate the relevance of the temporal precision.
Human perception takes a broader range of factors for determining redundancy into account than is measured by current near duplicate detection methods. MediaEval has organised a boredom detection task, with the goal to automatically determine which videos of a series of video blogs would be found boring by viewers. Both evaluation methods do not well model the need to eliminate redundancy in search results and group similar results. TOSCA-MP will assess the influence of different aspects on the redundancy in search result items and will feed the results back into the development of new methods for near duplicate detection.
In the evaluation method used for known-item search in TRECVID, there was exactly one matching clip, while in practice another clip might be acceptable as well in the context of the production. Modelling this tolerance would help to assess better the value of retrieval results, and TOSCA-MP will work on the development of benchmarking methods taking this into account.
Given the availability of well-defined usage scenarios, sufficient real-world data and professional users, TOSCA-MP can advance the state of the art in the evaluation of interactive search and browsing tools. Novel evaluation measures will be defined on the value of the retrieved results in the production context and validated in user experiments.
Task models and business processes
TOSCA-MP has collected a set of task descriptions (see D4.1), which were put in relation to the processes resulting from the more technical work on requirements analysis. Using the ConcurTaskTree notation coming from model-based UI design, the task descriptions were formalised to obtain machine-readable models, which can be used for service orchestration (by deriving business processes) and benchmarking. An automatic conversion from CTT models to executable business processes in BPMN notation has been developed. The ctt2bpmn1.0 conversion XSL is published under LGPL. The business processes have been used among others for cost simulation of the use of automatic content analysis tools as described in the EuroITV paper.