Semantic Annotation for Process Models: - Department of Computer ...

Chapter 8
Quality Evaluation of the Method
Evaluation of the method and prototype implementation consists of two parts: 1)
quality analysis of semantic annotation framework and method, and 2) applicability
validation based on annotation results. In this chapter, we focus on the quality of our
method. We apply a quality framework — SEQUAL to provide a systematic analysis on
the quality of GPO (General Process Ontology), PSAM (Process Semantic Annotation
Model) and the annotation tool Pro-SEAT. The quality of the work is evaluated based
on the use experience of the exemplar studies, which is also related to the applicability
validation of the work in Chapter 9.
8.1 Evaluation Design
Since the underlying theory of our work is information modeling, we look at the criteria
and metrics of the evaluation in the information modeling discipline. GPO and PSAM
have been created for the knowledge representation of process models. If the PSAM
definition in chapters 4 and 5 is regarded as a modeling language, GPO is the metamodel
of it, and an instance of PSAM is a model. In our exemplar studies, the original
EEML/BPMN process models are translated into the PSAM models in a common process
knowledge modeling language based on GPO. We therefore apply a general quality
framework of models and modeling languages in the evaluation. The quality framework
is based on the semiotic theory 1 , which is also the theoretical basis of our annotation
framework. Hence the whole work is built and evaluated under the same theoretical
foundation. The quality categories are selected from the quality framework and evaluated
through the exemplar studies. The annotation tool is also evaluated according to
the quality framework. However, since it is only a prototype, the performance of the
system and the user interface is not taken into account in the evaluation.
8.2 Setting for the Quality Evaluation
In this section we discuss a general quality framework — SEQUAL [80], which is used
for this evaluation work. A set of facts from the annotation approach and exemplar
1 The quality categories in the quality framework correspond to the semiotic ladder [33].
129