A Hybrid Framework for COSMIC Measurement: Combining Large Language Models with a Rule-Based System

EasyChair Preprint 14110, version 2

Abstract

Accurate Functional Size Measurement (FSM) is crucial for effective project management and resource allocation in software development. The COSMIC FSM provides a consistent and standardized method, allowing for objective comparisons and accurate estimation of effort. However, manual FSM is often time-consuming, error-prone, and subject to assumptions and human bias. Automating this process can greatly benefit organizations but introduces challenges associated with handling requirements expressed in Natural Language (NL). In this paper, we explore the integration of Large Language Models (LLM) with rule-based systems to enhance the automation of the COSMIC Function Point Measurement. Our hybrid framework combines the contextual understanding and adaptability of LLMs, such as GPT-4, with the precision and consistency of rule-based engines crucial for tasks requiring strict adherence to rules and regulations. This approach enables a more robust analysis, processing of NL requirements, and precise application of the COSMIC measurement method. The effectiveness and feasibility of this approach are demonstrated through a series of experiments conducted on COSMIC public case studies.
Our results show a promising F1-score of 0.95 in identifying COSMIC key components using GPT-4, an F1-score of 0.97 in deducing data movements using the rule-based system, and a global error rate of only 6.6% in measuring COSMIC function points using the integrated framework. These results underscore the potential for more reliable and efficient system analysis through the synergistic capabilities of LLMs and rule-based systems.

Keyphrases: Automation, COSMIC FSM, GPT-4, LLM, Prompting, rule-based system

@booklet{EasyChair:14110,
  author    = {Safae Laqrichi},
  title     = {A Hybrid Framework for COSMIC Measurement: Combining Large Language Models with a Rule-Based System},
  howpublished = {EasyChair Preprint 14110},
  year      = {EasyChair, 2024}}