Evaluation Visualization Overlay (EVO): A User Validation Study
Evaluation Visualization Overlay (EVO): A User Validation Study
2022 SURF Abstract by Yesugen Baatartogtokh, Irene Foster, Joey Elsbernd
BloomingLeaf is a web-based tool that allows users to model and analyze goals that evolve over time. These models consist of actors, links, and intentions that have function types and satisfaction values, which are also known as evidence pairs. By evaluating the changes in intentions over time through path analysis, users are better able to understand future project evolutions. However, running a path analysis means that each intention in the model contains evaluation information for each time point of the path. This information is difficult for the user to remember, which makes decision-making complicated.
Previous work in BloomingLeaf implemented Evaluation Visualization Overlay (EVO) as a feature that assigns each evidence pair a color. In BloomingLeaf’s Modeling mode EVO colors the intentions that have evidence pairs according to the color key. In Analysis view has three mode options: Percent, Time, and State. State mode allows the user to step through each time point in the path with the intentions colored according to their satisfaction value. In Time mode, the intentions are striped chronologically with each color representing a different time point, allowing the user to see the evolution of the model over time at a glance. Percent mode colors the intentions with the percentage of time it has a specific satisfaction value, showing the values according to proportion.
To validate that adding EVO assists users in their model understanding and allows for better and faster decision making, we developed a between-subjects user study that will be taken online. Our research questions are: RQ0: Do modelers perform similarly on understanding tests, given a consistent training protocol RQ1: How does EVO affect modeler’s understanding and comprehension of goal models RQ2: How does EVO affect the speed at which a modeler understands a goal model
RQ1 has understanding and comprehension being broken down into the ability to interpret, exemplify, classify, summarize, infer, and compare. RQ2 looks at the time it took subjects with and without EVO to answer our questions. This summer, we explored empirical methods of evaluation and created our study design. Additionally, we prepared study materials (including videos) and tested these materials with others in the lab. We intent to run our study this fall, which will allow us to validate the usefulness of EVO for users.
