2015-02 How (much) does the presence of a legend in a conceptual model influence its understandability?

Wie: 1 of 2 student(en) HIR/MBE/HW(Inf)
Promotor: Jan Claes

Problem description.
Observation 1. Various conceptual models exist (e.g., data models, process models, value models, etc.)
Observation 2. For some model types multiple modeling languages exist (e.g., BPMN, EPC, Petri-Net, etc.)
Observation 3. Conceptual models are used for various purposes (e.g., software analysis, process redesign, security analysis, etc.)
Observation 4. Various people in organizations come into contact with some kind of conceptual model.
Observation 5. These people have varying levels of knowledge and experience about the models they work with.
The former five observations describe why in organizations it happens that professionals have to be able to understand models they are not used to work with. This forms a challenge that is currently mainly addressed by providing training, rather than researching how to make it easier for novices to interpret conceptual models. We wonder if, how much and how the presence of a legend in conceptual models influence the understandability by model readers that are not used to work with the particular type of conceptual model, modeling language or model purpose.

Research objectives.
The goal of this master dissertation is to investigate how and how much it would help an inexperienced model reader to provide a legend of the utilized symbols. This legend may include the symbol, its name, an explanation of its meaning, an example of its use, etc.

This proposal is for a student who aspires to get a taste of real research practice. We are looking for a student who wants to take control and responsibility for the research. We will not interfere with your planning, ideas, approach, etc. In turn we offer our time, knowledge and support. You will receive a lot of inspiration and advice both on the research content and about how to manage the project, and we will try to offer help and answers when you ask for it.

Concerning the academic article, we expect you to write a 15-page article summarizing the research, which we want to submit to the annual BPM conference if the quality of the research and the text is good. We will do our best to help you reach this goal. We can help you to take crucial decisions, to improve research quality, to structure and write the text of the article.

In the first year, the aim should be to have fully designed the research methodology. Every aspect of the problem and proposed solution method needs to be understood and motivated. By the time of the intermediate evaluation session, you will be asked to answer and defend parts of the problem (Why is this problem (still) relevant? Who will benefit from your research? (How) is this related to legends in geographical maps?) and the proposed solution (Why did you decide to test this? Why do you think it is the best way to test this? How can you be certain about the results? What are the limitations of your approach?) This way we hope to help you to develop a strong methodological approach to the research, which you will need to get it published.

Below are some initial thoughts that can help you to get started. Note however, that in the end we may play the devils advocate and question every step of your research (including our own suggestions).

  • How will we present the legend? Why?
    • What will be displayed (Symbols? Meanings? Examples? Usage rules? Only used symbols or all symbols?)
    • In what order will the elements of the legend be displayed (According to the model? Fixed order? Alphabetical order?)
    • Where will we display the legend (Top, bottom, left, right, horizontal, vertical, multiple columns, etc.)
    • How to decide this (Pilot testing with expert opinions? Literature? Testing multiple alternatives?)
  • How will the effect be tested? Why?
    • How many modeling languages will be tested
    • How many exercises will we develop? Which exercises?
    • How many respondents? Which respondents?
    • Between subjects, within subjects or mixed design?
    • Test existing language with novice modelers or test unexisting language with expert modelers?
  • How will we measure understandability? Why?
    • Define understandability
    • How to measure understandability?
    • "super objective" = prove (e.g., discuss properties of the model and the legend, use psychological and cognitive theories to argument differences in understanding, etc.)
      "objective" = test (e.g., test users" understanding of the produced model, ask questions of the model such as can activity A and E be performed in one execution, can activity A happen before activity B, etc.)
      "subjective" = ask (e.g., ask about opinion of the users, rate understandability of different models)
      "super subjective" = assume (e.g., observe users and make assumptions about why misinterpretations have been made, etc.)
    • How to include differences in participants" capabilities? Measure this? Rules this out? Try to match subjects with similar capabilities? Search for homogeneous test group?
  • Which influencing factors can you define? Why?
    • What is/could be the role of expertise (domain expertise, modeling expertise, expertise in analyzing scheme"s, etc.)? Can we measure this? Can we use these measures to filter, group, control, the participants?
    • What is/could be the role of the complexity of the modeled systems? How should we consider this in developing the research design?
    • Are there other confounding variables?
  • What is the precise research question? Why?
    • How big is the effect (quantitative, predictive)
    • When does the effect happen (scope)
    • Why is there an effect (explanation)
  • Why is this relevant, who will benefit from this?
    • Relevance: which other solution exist, how does our solution compare to them
    • Impact: who will benefit from this

Afterwards, the research has to be executed and the dissertation and the academic article need to be written and submitted.

Here are some examples of the type of article we expect you to write. The articles can be downloaded from http://www.janclaes.info/publications
More references can be found at http://www.janclaes.info/masterproef (ref: business process modeling)

  • Jan Claes, Irene Vanderfeesten, Hajo A. Reijers, Jakob Pinggera, Matthias Weidlich, Stefan Zugal, ... Geert Poels, (2012). Tying process model quality to the modeling process: The impact of structuring, movement, and speed. In A. Barros, A. Gal, & E. Kindler (Eds.), Proc. BPM "12, (Vol. LNCS 7481, pp. 33-48). Springer.
  • Jan Claes, Geert Poels, (2013). Process mining and the ProM framework: an exploratory survey. In M. La Rosa & P. Soffer (Eds.), Proc. BPM "12 Workshops, (Vol. LNBIP 132, pp. 187-198). Springer.
  • Jan Claes, Irene Vanderfeesten, Jakob Pinggera, Hajo A. Reijers, Barbara Weber, Geert Poels, (2013). Visualizing the process of process modeling with PPMCharts. In M. La Rosa & P. Soffer (Eds.), Proc. BPM "12 Workshops, (Vol. LNBIP 132, pp. 744-755). Springer.

Interested? Book your own appointment via janclaes.youcanbookme.com.