Goal Model Construction Study

Note: This work is published in the Proceedings of the 13th International i* Workshop, the abstract and talk slides can be veiwed here.

A Preliminary Study of the Utility of Goal Model Construction
2019 SURF Abstract by Naomi Cebula and Lily Diao

Goal-oriented requirements engineering (GORE) is a subset of requirements engineering, which focuses on the elicitation and analysis of stakeholders’ intentions [1]. Grubb proposed several studies aimed at investigating what utility stakeholders derive from constructing and analyzing goal models [2]. In this project, we designed and conducted an empirical study that explored the construction stage of goal modeling, asking whether stakeholders benefit from manually drawing their model (on paper or in a tool) for the purpose of understanding and generating project scenarios. Specifically, we compared reviewing auto-generated models with manually created ones for the purpose of helping students answer their own self-directed questions, the results of which have implications for goal model adoption and automation.

In order to compare subjects’ self-created models with auto-generated models, we asked subjects to discuss a decision they were considering through an online pre-study questionnaire. We manually constructed a goal model from the pre-study. In a one-hour in-person session, participants first answered questions about a training video to measure their understanding. Participants then constructed a model of their decision while talking out loud. Half of the participants in this study used BloomingLeaf (a web-based goal modeling tool), while the remaining participants drew goal models by hand. We prompted participants with a list of prepared questions to help them add new elements and links. After the participants finished their model, we asked them to compare their model to the one we created from their pre-study questionnaire. We then asked them to extend their preferred model with new insights generated. Finally, we asked participants questions to explicate what utility (if any) they experienced in making their decision.

We recruited eight qualified participants at Smith College in the summer of 2019. Each participant was randomly assigned to either the Paper group or the Tool group. They completed an in-person training session and were asked to construct a goal model either on paper or with Bloomingleaf. We used open coding to find themes and categories for qualitative responses. Early analysis showed participants preferred their own model to the researcher generated one. The analysis is ongoing, and we hope to present results later this year.

References:
[1] J. Horkoff, et al.. Goal-oriented Requirements Engineering: An Extended Systematic Mapping Study. Requir. Eng., 24(2):133–160, 2019.
[2] A.M. Grubb. Reflection on Evolutionary Decision Making with Goal Modeling Via Empirical Studies. In Proc. of RE’18, pages 376–381, 2018.