Using an AI Chatbot to Refine Gherkin Specifications Based on Stakeholder Comments

Abstract

A software project begins with capturing visions and requirements in an understandable format, e.g., vision videos, which represent complex software-based processes. This video invites comments from stakeholders for validating the vision. However, the easy-to-watch videos must be translated to easy-to-validate requirements, which can be written in the semi-formal Gherkin specification.

Requirements engineers use a video and obtained comments to refine Gherkin specifications. However, this is a demanding task. AI chatbots such as ChatGPT can be used in the refinement. We investigated the effectiveness of using ChatGPT. Two requirements engineers used textual description of a vision video and comments of 12 stakeholders to refine Gherkin specifications with and without ChatGPT 3.5. We asked for stakeholders' opinions on the refined specifications with and without ChatGPT.

Results show that (1) the understandability, and (2) the stakeholders' satisfaction on refined specifications with and without ChatGPT do not differentiate significantly; (3) ChatGPT generated detailed specifications but made formulation errors. We suggest using an AI Chatbot and learning from its answers to achieve stakeholders' satisfaction.