Amazon says no human wrote or reviewed the code for its new cloud testing tool

Amazon released AWS Emulator on Nov. 11, a local tool that imitates AWS services so developers can test cloud applications on their own machines. The company says autonomous AI agents wrote and reviewed its implementation.

In Amazon's engineering account, engineer Matthew Goldberg says his team neither wrote nor reviewed the emulator code. He also describes the substantial engineering work needed to turn an experiment into a product, including the guardrails and checks that determine which changes can ship.

Amazon describes three teams of agents: one manages fidelity tests, another finds differences between the emulator and real AWS, and a third fixes individual issues. Changes face checks for compilation, coding standards, security and regressions before merging, according to the company.

The test agents work from public AWS API models and verify their scenarios against the real cloud. The public evaluation suite says every fidelity test was checked against at least two sandboxed AWS accounts.

What a passing test establishes

AWS calls the degree of agreement with its cloud services “fidelity.” A test passes when the emulator matches AWS for every call in that workflow, including status codes, resulting resources, side effects and errors, according to its fidelity documentation. Returning success where AWS returns an error fails that test.

The boundary is coverage. The evaluation repository explicitly says behavior without a fidelity test is unverified. Passing a scenario establishes a match for that scenario; it does not establish that every way an application uses AWS will work.

A passing workflow is evidence for that scenario. It does not validate all AWS behavior or application security.

Diagram shows AWS's pass rule, not a particular test result or measured coverage. Application security still requires testing on real AWS before production.

Sources and evidence

Sources

1. AWS fidelity definition
2. AWS evaluation suite
3. AWS security limitations
4. AWS Emulator intended use

Supporting facts

• AWS defines a fidelity-test pass as matching AWS for every call in the scenario, including status codes, resources, side effects, and errors, and publishes tests the emulator does not yet pass. [1]
• The public evaluation suite says behavior not covered by a fidelity test is not verified; a passed scenario therefore does not establish equivalence across all AWS behavior. [2] [1]
• AWS warns against using the emulator to validate application security: it does not check AWS KMS key policies or Amazon S3 bucket policies, and a request AWS denies may succeed in the emulator. [3]
• AWS instructs users to test their whole application and security settings on real AWS before production; the emulator is intended for building and testing, not production workloads. [3] [4]

Method

Qualitative diagram of AWS's published fidelity definition and testing limits. Branches compare the real-AWS reference with emulator replay; they do not represent simultaneous live tests, amounts or coverage proportions.

Scope

Diagram shows AWS's pass rule, not a particular test result or measured coverage. Application security still requires testing on real AWS before production.

Infographic by CryptoSlate

Security has a separate limit. AWS's security guidance warns against using the emulator to determine whether an application is secure. It does not check AWS KMS key policies or Amazon S3 bucket policies, and a request denied by AWS may succeed locally.

Agent review of implementation security therefore cannot be read as validation of a user's application permissions. The emulator is for building and testing, not production workloads. AWS instructs developers to test their whole application and security settings on real AWS before production.

For builders adopting agent-written software, the release makes the acceptance process as relevant as the code's authorship: which behaviors are checked, what remains outside coverage, and where the developer must still test against the real service.