Skip to main content

Replace Vibes with Tools

Replace Vibes with Tools

Intent

When a part of your problem is well-specified and deterministic, don’t use a generative approach to try to find a solution. Create a reusable, reproducible tool that your generative assistant can use.

Motivation

Not all problems are best solved using generative techniques. The field of software engineering has built up a barrage of automated tools and approaches to deterministically support software systems and their creation. These tools include:

  • Package-management tools, to control the software supply chain.
  • Build tools, to automate software construction.
  • Testing tools, to repeatably verify or validate software.
  • Analysis tools, to uncover API misuse, security vulnerabilities, software bugs, and other issues.
  • Reporting tools, to summarise status.
  • Work-tracking tools, that record tasks, their assignment to agents (human or otherwise), and their status.
  • Scripts and commands that generate source code from models, or prepare resources and other artefacts.
  • Continuous integration and delivery tools, that automate invocation of other tools during the software development lifecycle.
  • Deployment tools, to install a software system onto the target computer(s).

The above list is far from exhaustive. When you use any deterministic, automatic tool in your process, the result reliably depends (assuming there aren’t any bugs in the tool’s implementation) on the tool input, project context, and environment.

Your coding assistant might be able to replicate the tool’s behavior, or discover the correct way to use the tools, but it might land on a slightly different invocation than you expect: for example, using the test runner in a way that only runs some of the tests. Additionally, its trial-and-error process to find a way to use the tool might add lots of tokens to its context, leading to attention issues on its main task.

Give the coding assistant access to a tool that performs the deterministic sub-task in exactly the way you want. Control the tool’s output, so that the model receives relevant information that doesn’t overwhelm its context. Package the tool in an agent skill or an MCP server so that the model can discover and use it.

Applicability

Use Replace Vibes with Tools to encapsulate a deterministic task in your workflow. This task might be a step in the software development process, for example building your application or running tests. Alternatively, the task might be something from the broader context of your work, like generating a status report or presentation for colleagues, or providing updates to an issue-tracking tool.

Consider Replace Vibes with Tools particularly for tasks where the model makes a lot of failed attempts at correctly generating a solution, like updating a structured project definition (for example an Xcode project, BUCKFILE, or Maven pom.xml).

Replace Vibes with Tools enables so called “lights-out” development (where coding agents work on software without human intervention) by ensuring that the agent completes critical process steps and sequence points in a deterministic fashion.

Consequences

Replace Vibes with Tools offers the following benefits:

  • Reduce the context cost to your coding assistant of discovering how to complete steps in your workflow.
  • Gain deterministic control over steps in your workflow.
  • Provide reusable recipes for tasks that you share with your team’s coding assistants or with “lights-out” development agents.

Implementation

Create a tool, for example a shell script or an interpreted program, that performs the reproducible step in your workflow. Give the tool the following characteristics:

  • A help command that lists the other commands available, with their arguments.
  • Distinct return codes for success and failure.
  • Terse output, for example no output on success, and relevant diagnostic information on failure.

Provide your coding assistant with access to the tool and information about how to use it. Two options are the Model Context Protocol, and Agent Skills standard.

Example

Run my tests

LLMs, including Claude Sonnet 4.6 and Qwen 3 Coder Next, demonstrate problems with using automated tests to guide development. Both models typically make code changes without running tests. If prompted to use the tests to guide implementation, they generate incorrect commands that only run some of the tests, or even create commands that run individual named tests without checking for regressions by running other tests.

On one project, I created an agent skill with scripts to run the unit tests and integration tests:

#!/bin/sh

xcodebuild -project AppScript/AppScript.xcodeproj -scheme AppScriptTests test

The source code is available at https://codeberg.org/leeg/patterns-examples/src/branch/main/run-tests-skill/scripts/unit-tests.sh.

Where the prompt tells the agent precisely the command to run:

---
name: run-tests
description: Use this skill to correctly run integration tests and unit tests
license: AGPL-3.0-or-later
user-invokable: true
---
# AppScript test-runner skill

This skill provides scripts to run the unit tests and integration tests for this project. You must **always** run these scripts to get the results of tests; don't create other commands.

These scripts run _all_ of the tests of the specified type. It's always good practice to run all tests when you make a change, so you can check for unexpected regressions in behavior.

## Run the unit tests

Run the script at `scripts/unit-tests.sh`, relative to the base directory for this skill.

## Run the integration tests

Run the script at `scripts/integration-tests.sh`, relative to the base directory for this skill.

The source code is available at https://codeberg.org/leeg/patterns-examples/src/branch/main/run-tests-skill/SKILL.md.

The agent—Claude Code, or Mistral Vibe—automatically discovers the skill and gets the model to generate the correct commands to run tests. Alternatively, I can get the model to use the skill directly via a “slash command”, for example:

/run-tests unit tests

Intervene when your assistant has trouble performing a task, so you Replace Vibes with Tools and complete the task correctly.

Remember What We Did to find the end state of an interaction that forms the basis for Replacing Vibes with Tools.

You Reflect to discover whether vibing instead of tool-use is the cause of your problem.

No comments to display

Back to top