When we say storywise improves your RE process, you don’t just have to take our word for it anymore: We actually had Dipl.-Ing- Dr. techn. Petra Unger look into it and compare the RE process and the quality of specifications with and without storywise in a study. Let’s look at what she found.
storywise VS. Word in the RE process – the numbers at a glance:
- Clarity of requirements +16%
- Completeness of requirements +12%
- Verifiability of requirements +12%
- Contextual logic +3,25%
- Granularity +0,75%
→ Overall quality of the requirements improved by 9% when written with storywise!
In her BA thesis, Petra Unger investigated how the quality of requirements specifications changes when they are created with an AI-supported tool such as storywise versus when created with traditional methods (Microsoft Word). A total of 9 requirements engineers with at least 2 years experience in the field was split into two groups and asked to create specifications for the same time tracking software.
These specifications were then evaluated on 5 aspects (clarity, completeness, verifiability, contextual logic, and granularity) and given a grade from 1 to 5, with 1 being bad and 5 being good.
And the results were actually really cool – not just in terms of proving what we already know, but also in terms of feedback we got from the requirements engineers who participated. Because they loved storywise!
Let’s dive into the most interesting findings of the study.
Finding #1: storywise AI can level differences between skill levels
“The AI in storywise is more than just a text generator, it acts as a sort of cognitive framework that facilitates a systematic capture of content.”
– Petra Unger
While the study showed that on average, the group working with storywise AI created better requirements than the one working in Microsoft Word, the overall quality improvement was moderate. However, a more detailed look at the results achieved by the individual participants showed that there was less fluctuation in the quality produced by the group working with storywise AI.
This is particularly interesting because it indicates that storywise and its AI support act as a systemic framework that makes it easier to keep everything in mind. This is probably most valuable for less experienced requirements engineers as it increases the quality of work, but also has great potential for more experienced professionals as it reduces the risk of potentially costly errors.
Finding #2: storywise AI is most valuable for clarity, completeness & verifiability of requirements
“storywise acts as a sort of formal quality control: the tool reduces ambiguities, promotes easy to understand sentence structure and drives the user’s focus to concrete, verifiable statements.”
– Petra Unger
One thing the study showed was that storywise complements human expertise rather than replacing it: on the one hand, the quality differences between the two groups were negligible in the categories contextual logic and granularity. This shows that AI support doesn’t have a strong impact on the aspects of requirements engineering that are related to domain knowledge and context.
On the other hand, the study shows that AI support does have a positive effect on those aspects of RE that are related to linguistic precision and explicit structure. This indicates that storywise AI improves requirements by a) making implicit information explicit, b) structuring information, and c) providing specifications in a verifiable format.
Finding #3: most participants would like to use storywise in their daily workflows
When asked whether they would be interested to use storywise in their work and whether they have further feedback for us, most answers were positive and said things like:
- “I really like it and would be interested to integrate storywise in my daily workflows.”
- “I prefer standardised requirements and therefore appreciate workflows with AI support.”
- “Definitely useful for the quick classification and creation of stories.”
- “Great tool, although it could be more intuitive sometimes.”
- “A valuable addition to determine the parameters of new projects.”
- “I think it’s a great tool that helps in creating succinct requirements.”
And in addition to that great feedback, one thing made us particularly happy: 4 in 5 people said that the 10-minute intro to storywise was enough for them to work with the tool!
We also asked the participants of the study how happy they were with the support our tool provided, and 40 % said that they were “extremely satisfied” with storywise!
Study details – the hard facts
If you care about the details of the study but don’t want to read the entire thesis, here’s a breakdown of how the study was conducted.
| Research question | How does an AI-supported workflow impact the level of detail in software specification requirements compared to traditional methods? |
|---|---|
| Participants | 9 requirements engineers:
|
| Execution | The study was conducted via Microsoft Teams in three sittings. Each meeting started with a 5-minute intro to the study, followed by a 10-minute intro to storywise. After that, the participants had 25 minutes to create their software requirements and another 5 minutes to fill out a statistics and feedback questionnaire. |
| Task | The participants were asked to create requirements for a time tracking software in natural language. |
| Evaluation criteria |
|
| Findings | The study shows that AI-supported tools can improve the quality of software requirements, particularly in terms of clarity, completeness and verifiability of the specifications. Both groups achieved similar results in contextual logic and granularity, indicating that these aspects require human expertise. In addition, the study showed less variation in the group working with the AI-supported tool, suggesting a standardizing effect of AI that helps balance out different levels of skill or experience. |
If you do want to check out the entire thesis (only available in German), you can do so here.
