Fiber Product
Engineering case study 0112 min read

Stripe agent benchmark

One workflow.
Nine tools.
25 paired runs.

We gave the same model the same multi-step Stripe task through two interfaces. Fiber completed every run with one call, using 95.2% fewer tokens and producing zero Stripe API errors.

Answer, with scope

Fiber helped decisively on efficiency and error exposure. Observed completion improved from 24/25 to 25/25, but that one-run difference is too small to generalize beyond this benchmark.

See results Download data
Literal
9 Stripe tools
96%
task completion
11.7 calls / run
Fiber
1 reviewed workflow
100%
task completion
1.0 call / run
01Observed result

The model did less work—and made fewer mistakes.

95.2%
fewer tokens
39,764 → 1,904 / run
91.4%
fewer tool calls
11.7 → 1.0 / run
74.1%
lower latency
53.1s → 13.7s / run
33 → 0
Stripe API errors
across 25 runs / interface
Tool calls / run
91.4% fewer
Literal
11.7
Fiber
1.0
Agent tokens / run
95.2% fewer
Literal
39,764
Fiber
1,904
Agent latency / run
74.1% lower
Literal
53.1s
Fiber
13.7s
Exact aggregateLiteralFiber
Completed trials24 / 2525 / 25
Total tool calls29225
Stripe API errors330
Total agent tokens994,10647,605
Aggregate agent latency1,326.971s343.415s
Cleanup remediation1 run0 runs
02Method

Same model. Same task. Same Stripe transport.

Five frozen catalog tasks were repeated five times per interface. Every trial created a product, attached a one-time price, created and retrieved a payment link, verified ownership and relationships, then left all created resources inactive. Surface order was counterbalanced across pairs.

Literal interface

09 tools
  • Create, retrieve, and update product
  • Create, retrieve, and update price
  • Create, retrieve, and update payment link
  • The model sequences and repairs every step

Fiber interface

01 tool
  • One confirmation-gated catalog workflow
  • The same nine Stripe operations underneath
  • Code verifies ownership and relationships
  • Cleanup executes even when a step fails
Model
qwen3.5:27b-q4_k_m-262k
Temperature
0
Task set
5 × 5 pairs
Transport
Stripe test mode
literal: create_product → create_price → create_link → retrieve × 3 → deactivate × 3
fiber:   run_catalog_workflow({ confirmed: true }) → assertions + cleanup
03Failure analysis

The only failed run was not a scoring judgment.

In catalog-200, repetition 4, the literal agent created an inactive price, received a Stripe rejection, then created a second active price while recovering.

The payment link eventually worked, and remediation left every created object inactive. The deterministic evaluator still rejected the trial because the task forbade duplicate side effects. The agent’s final prose claimed success; the resource evidence did not.

17

Stripe: “Payment Links cannot include deleted or archived prices.”

16

Stripe: “Payment Links cannot include a price with an inactive product.”

Failed row: stripe-catalog-200 / repetition 4 / 13 calls / 46,657 tokens / remediation verified.

04Public dataset

All 50 trial measurements, without Stripe object IDs or model transcripts.

The downloadable JSON contains every task ID, repetition, interface, pass/fail result, tool-call count, API-error count and message, token count, latency, remediation flag, and final cleanup state. Raw provider responses and sandbox resource identifiers remain excluded.

Download JSON Inspect source protocol
Inspect all 50 sanitized trial rowsExpand +
TaskRepInterfaceResultCallsAPI errorsTokensLatencyCleanup
catalog-04901FiberPass101,90114.2sVerified
catalog-04901LiteralPass13245,71260.1sVerified
catalog-04902LiteralPass13245,46854.4sVerified
catalog-04902FiberPass101,89113.5sVerified
catalog-04903FiberPass101,88013.2sVerified
catalog-04903LiteralPass13245,66560.8sVerified
catalog-04904LiteralPass12141,69455.5sVerified
catalog-04904FiberPass101,89213.6sVerified
catalog-04905FiberPass101,90313.7sVerified
catalog-04905LiteralPass13245,84160.8sVerified
catalog-09901LiteralPass9029,14639.9sVerified
catalog-09901FiberPass101,89813.6sVerified
catalog-09902FiberPass101,88913.4sVerified
catalog-09902LiteralPass9029,11939.8sVerified
catalog-09903LiteralPass9029,05639.4sVerified
catalog-09903FiberPass101,89313.6sVerified
catalog-09904FiberPass101,89213.7sVerified
catalog-09904LiteralPass13245,76860.3sVerified
catalog-09905LiteralPass13245,87355.5sVerified
catalog-09905FiberPass101,90213.3sVerified
catalog-20001LiteralPass13245,91459.9sVerified
catalog-20001FiberPass101,89713.4sVerified
catalog-20002FiberPass101,89913.6sVerified
catalog-20002LiteralPass13246,15960.9sVerified
catalog-20003LiteralPass13245,86359.9sVerified
catalog-20003FiberPass101,89713.6sVerified
catalog-20004FiberPass101,90413.6sVerified
catalog-20004LiteralFail13246,65763.1sVerified
catalog-20005LiteralPass13245,58458.0sVerified
catalog-20005FiberPass101,89713.5sVerified
catalog-35001LiteralPass13245,77761.5sVerified
catalog-35001FiberPass101,90813.7sVerified
catalog-35002FiberPass101,92714.3sVerified
catalog-35002LiteralPass13245,54459.8sVerified
catalog-35003LiteralPass13245,71255.5sVerified
catalog-35003FiberPass101,89513.3sVerified
catalog-35004FiberPass101,90813.5sVerified
catalog-35004LiteralPass13245,81459.4sVerified
catalog-35005LiteralPass13245,83361.2sVerified
catalog-35005FiberPass101,93515.1sVerified
catalog-125001LiteralPass9015,61639.2sVerified
catalog-125001FiberPass101,92314.4sVerified
catalog-125002FiberPass101,92214.0sVerified
catalog-125002LiteralPass9029,09241.2sVerified
catalog-125003LiteralPass9029,09641.6sVerified
catalog-125003FiberPass101,92014.1sVerified
catalog-125004FiberPass101,91714.1sVerified
catalog-125004LiteralPass9029,07740.3sVerified
catalog-125005LiteralPass9029,02639.0sVerified
catalog-125005FiberPass101,91513.4sVerified
05Interpretation

What this result does—and does not—establish.

Supported by this benchmark

  • Fiber reduced model-visible orchestration from nine tools to one reviewed workflow.
  • Every Fiber trial completed in one call with zero observed Stripe API errors.
  • Tokens and local-model latency fell substantially on this task family.
  • Deterministic cleanup prevented abandoned active resources.

Not established here

  • A universal reliability improvement across APIs, models, or task types.
  • A statistically decisive completion-rate lift from a 25-pair sample.
  • Hosted-model dollar savings; local inference cost was recorded as $0.
  • Hardware-independent latency; only paired relative latency is meaningful.
06Reproduce

The protocol was frozen before the result existed.

The repository pins the model, runtime, seed, context length, task files, tool schemas, source fixture, assertions, cleanup policy, and publication requirements. The publisher recomputes the public aggregate from raw redacted evidence and refuses partial suites.

npm run benchmark:stripe:check
npm run benchmark:stripe:case-study
npm run benchmark:stripe:publish -- artifacts/stripe-case-study/<run>
Protocol SHA-2561529b4c07f25b420d75718d0be0c0af1b06a5724c3fdc5a9c98efbd8489a8799
Tasks SHA-256f0336dc11a3988068afdb08f01eaba00fe59468bad702e2145e606a0b0614df9
Literal schema SHA-25669e676c7efe8f2fee5ab59ad7cec82bd988125375f5dba5d0754aed1dc177a20
Fiber schema SHA-256c3787d02e4b189be8be14c2c06c075b23bf5bf8531d6f66f164f3cd7542026e6