Eliminating Requirement Bugs: A Tutorial on AWS’s Hybrid AI–Solver Approach

Overview

Most software defects are not born in code—they originate in the requirements that steer development. AWS research shows that nearly 60% of software requirements contain hidden bugs: contradictions, ambiguities, or gaps that slip into design and testing, only to surface as costly production faults. Traditional debugging then forces teams to backtrack weeks of work to trace the issue back to a misread specification.

Eliminating Requirement Bugs: A Tutorial on AWS’s Hybrid AI–Solver Approach — Source: thenewstack.io

To address this, AWS introduced Requirements Analysis as a new feature in its Kiro agentic development platform. The solution is not more AI—it’s a hybrid of large language models (LLMs) and a 50-year-old logic engine called an SMT (satisfiability modulo theories) solver. This tutorial explains how to apply this neurosymbolic approach to your own projects, whether you use AWS Kiro or implement a similar pipeline yourself.

Prerequisites

Before diving into the step-by-step guide, ensure you have the following:

Basic understanding of software requirements engineering (e.g., use cases, user stories, functional/non-functional specs).
Familiarity with AWS Kiro platform (or access to an equivalent tool that can combine LLMs with automated reasoning).
Access to an SMT solver such as Z3 (open-source) or the one built into AWS Automated Reasoning services.
An LLM API key (e.g., from AWS Bedrock, OpenAI, or Anthropic) for rewriting requirements.
Programming environment (Python 3.8+ recommended) for tying the steps together—though the concept works in any language.

Step-by-Step Instructions

Step 1: Collect Natural-Language Requirements

Start with raw, ambiguous specifications. For example:

"The system must allow users to edit their profiles, but only during business hours."

This hides ambiguities: What time zone? What about admin override? Are logs required? Write down at least five such requirements as a test set.

Step 2: Rewrite Requirements with an LLM

Use an LLM to convert vague statements into precise, testable criteria. Prompt the model:

Prompt: "Rewrite the following requirement to be unambiguous, testable, and complete. List any missing assumptions or implied conditions: [original requirement]"

Example output:

"A user may edit their profile fields (name, email, phone) only between 08:00 and 18:00 UTC, Monday to Friday, excluding public holidays observed in the user’s account region. The system shall log all edit attempts and return an error if the time constraint is violated."

This step leverages the LLM’s ability to infer context and fill gaps, but the output is still natural language—not yet machine-checkable.

Step 3: Translate Requirements into Formal Logic

Now convert each rewritten requirement into a formal representation using a logic language compatible with SMT solvers. For the example above, you might use first-order logic or a fragment like:

(declare-const edit_time Int)
(declare-const day_of_week Int)
(declare-const is_holiday Bool)
(assert (and (>= edit_time 8) (<= edit_time 18)))
(assert (and (>= day_of_week 1) (<= day_of_week 5)))
(assert (not is_holiday))
(assert (not (= edit_time 0))) ; simplified

If you have multiple requirements, merge them into a single set of assertions. This step may require domain expertise—tools like AWS’s automated reasoning service can assist with translation.

Step 4: Run the SMT Solver

Feed the formal assertions into an SMT solver (e.g., Z3). The solver will attempt to find a satisfying assignment (a model) that meets all constraints. If it fails to find any model, you have a contradiction. If it finds multiple models, you may have ambiguity.

from z3 import *
edit_time = Int(’edit_time’)
day = Int(’day’)
holiday = Bool(’holiday’)
s = Solver()
s.add(And(edit_time >= 8, edit_time <= 18))
s.add(And(day >= 1, day <= 5))
s.add(Not(holiday))
if s.check() == sat:
    print("Consistent:", s.model())
else:
    print("Contradiction detected!")

The solver will report either sat (consistent) or unsat (contradiction). For multiple requirements, you may get a sat result but with unexpected variable assignments that reveal hidden gaps.

Step 5: Interpret and Fix Bugs

When the solver finds a contradiction, it means no implementation can satisfy all requirements simultaneously. For example, one requirement says “edit allowed only during business hours” and another says “admins can edit 24/7.” The solver proves impossibility. AWS Kiro presents these as plain-language, two-option questions, often solvable in 10–15 seconds each. You then adjust the requirements—either remove the contradiction or add an explicit priority rule.

Common Mistakes

Skipping the LLM rewrite step – Feeding raw ambiguous requirements directly to the solver yields trivial contradictions or false positives.
Over-constraining the logic – Adding too many assumptions (e.g., hardcoding time zones) can mask real contradictions. Keep variables flexible.
Ignoring the solver’s unsat core – When unsat is returned, inspect which assertions caused the conflict. Most SMT solvers provide an unsat core to help.
Treating the solver as a bug finder only – The solver also identifies gaps: missing conditions that lead to unintended behaviors (e.g., no rule for holiday handling).
Not iterating – Requirements are living documents. Re-run the pipeline after each revision.
Assuming the LLM is enough – As AWS emphasizes, the LLM alone cannot prove correctness; the solver is needed for formal validation.

Summary

AWS’s Requirements Analysis feature in Kiro demonstrates that combining modern LLMs with classic automated reasoning—specifically SMT solvers—can catch up to 60% of requirement bugs before a single line of code is written. The three-stage pipeline (LLM rewrite, formal translation, SMT solving) turns vague specs into provably consistent rules. This neurosymbolic approach is not only for AWS users; you can adapt it using open-source tools like Z3 and any LLM. The key is to treat requirements as code: test them early, formally, and often.

Tags: