Structuring AI-assisted software development: a case study of City App Framework

1 Introduction

AI coding agents now build working software from a short description. What they lack is structure that lasts beyond one session: the facts of the project, the preferences and limits of the person they work for, and the lessons of earlier sessions. Developers supply that structure as text: context files such as AGENTS.md, collections of rules, and logs of past mistakes. That text costs tokens and attention on every run, and controlled studies have found that context files change task success little while raising cost [10] [11].

City App Framework is an attempt to give AI-assisted software and web development a lasting structure. Its name is its metaphor: a city runs on laws, customs and shared infrastructure, and a project built by AI agents needs the same. Version 1 (August 2025) specified a create-city-app scaffolder that would lay out a new project, with the human as mayor and AI agents as citizens; it was never built. Version 2 (April 2026) cast the human as a sponsor directing an autonomous council of AI departments. Version 3 (April to May 2026) shipped a universal AGENTS.md of 178 lines, about 40 convention documents on how to structure code, components, tests, releases and interfaces, decision patterns, templates, a 55 KB journal and a demo site.

By September 2026 the ecosystem had moved on: AGENTS.md had become a Linux Foundation standard read by most agents [1], Claude Code loaded it natively [2], scaffolders generated it [17], and "state the goal, the system builds it" had shipped in several products [13]. That raised the question this case study answers: what structure does a framework still need to give an AI coding agent, and which parts of City App's structure actually worked? We ask five questions:

1. RQ1. Does a framework's structure reach the agent at all?
2. RQ2. Which rules about how to build still change what a current model does, and which are redundant?
3. RQ3. How should the lessons of one session reach the next?
4. RQ4. Does splitting the work across agent roles, such as a separate reviewer, help on small tasks?
5. RQ5. Can a framework rebuilt from these answers be verified on real web and command-line projects?

Section 3 describes the resulting framework first, so the reader knows where the evidence leads. Sections 4 and 5 give the method and the evidence for RQ1 to RQ4, and Section 6 evaluates the framework on real projects to answer RQ5.

2 Background

Instruction files. AGENTS.md is a plain Markdown file of project instructions that Codex, Cursor, Copilot, Gemini CLI, Grok Build and others read at the start of a session; it became a Linux Foundation standard in December 2025 [1]. Claude Code reads AGENTS.md on its own only when there is no CLAUDE.md. A CLAUDE.md that tells Claude in words to read AGENTS.md leaves the decision to the model, while an @AGENTS.md import loads the file [2]. Vendor guidance recommends keeping such files short and specific [3] [5].

Enforcement. Claude Code hooks are scripts that run before and after tool calls and when the agent tries to finish. A hook can block an action or ask the human, whatever the agent intends [4]. Claude Code's auto mode blocks force-pushes and production deploys but allows installing new dependencies [9].

Related findings. Practitioners describe "harness engineering" and "compound engineering": when an agent makes a mistake, add a check that makes the mistake impossible, and write prose only when a check can't [14] [15] [16]. ImpossibleBench found that agents edit or skip tests they cannot pass, and that a plain instruction to report such tests reduces this [12]. Anthropic reported that agents split by role spent more tokens on coordination than on the work itself [8].

3 The framework

City App Framework v4 is a Claude Code plugin plus a few files installed in each project. It keeps the city metaphor in the one form the evidence supports: laws are enforced, customs are advice. Its design follows three principles, each traced to results in Section 5.

3.1 Principles

• Structure the process, not the code. Agents already wrote working, dependency-free code without being told how, and v3's rules on scope and dependencies had no measurable effect (Section 5.2). So the framework does not prescribe code style or folder layout; it structures how work is specified, checked and remembered.
• Enforce what must never happen; advise the rest. The must-nevers are hooks the agent cannot talk its way past. Everything else is a short file the agent actually loads (Figure 2).
• Keep a rule only while a test shows it works. Every line of advice was measured, and the framework ships the measuring tool so users can re-test their own rules when models change.

3.2 How work flows

Figure 1 shows the development loop the framework gives a project. Each stop leaves something checkable behind, so the next session starts from evidence rather than from memory.

3.3 Laws and customs

3.4 The commands

The plugin's eight commands (Table 1) carry out the loop in Figure 1. The ones that run test sessions show their plan and ask before using the user's usage.

4 Method

The framework's design comes from a set of experiments on its predecessor, v3, and on drafts of v4. This section describes them; Section 5 gives the results.

4.1 Subject app and harness

The test app, habit-cli, is a zero-dependency Node.js 22 command-line habit tracker with 7 tests. One experiment (E7) started from an empty project instead. Each trial was one fresh, isolated, headless session of Claude Code 2.1.283 working in its own copy of the app. Sessions could edit their folder and run an allowlist of commands; nothing bypassed Claude Code's permission checks. Most runs used one Claude model; one larger and one smaller Claude model were used for spot checks. Exact model versions are recorded with the raw results.

4.2 Conditions

4.3 Tasks

4.4 Measures

Every run was scored by fixed checks only: hidden acceptance tests for each task, diff size, files touched, dependencies added, whether tests were written and passed, number of turns and, for the vague request, whether the final message laid out options. Each scorer was first run against a correct and a deliberately broken reference solution. The options measure reads text, so it was also checked against labeled example replies; Section 5.5 describes a bug it had.

4.5 Experiments and analysis

That is 285 single sessions, plus 16 review trials of three sessions each. We report counts. Where a setup is compared with bare, we use a two-sided Fisher's exact test and give p when it is below 0.05. With 3 to 8 trials per cell only large effects are detectable, so the absence of a difference is weak evidence.

5 Results: the evidence

Each subsection answers one research question and ends with the design decision it led to.

5.1 RQ1: does the structure reach the agent?

In practice, v3's core structure (its rules on overbuilding, escalation and communication) sat behind a link and never reached the agent. Inspecting v3 found more defects of the same kind: its GROK.md was read by nothing, because Grok Build reads AGENTS.md; its project script could write a new project into the filesystem root when the parent folder was missing; and its 55 KB journal was never opened in any run.

In the framework: CLAUDE.md is one line that imports AGENTS.md, so the project's facts load in every session, and nothing important sits behind a link.

5.2 RQ2: which rules change how agents build?

The bare model already did the job. Every setup, bare included, passed every hidden acceptance check, including the trap of habit names with spaces in dates, and no run added a dependency, even for a web server or date parsing. v3's rules on scope and dependencies therefore had nothing left to fix. Test writing did change: a one-line "add a test for new logic" rule raised it from 17 of 25 runs to 25 of 25, while v3's full rules lowered it to 11 of 25. v3 as shipped used 24% more turns than bare, with no gain in quality. Turn counts are rough, because the sandbox denied shell commands containing $(...), which some setups used more often for manual checks.

The vague request is where rules mattered. Bare agents quietly picked one reading ("list today's unfinished habits") and never mentioned the larger options, such as notifications or scheduling. v3's escalation table ("when in doubt, ask") made agents stop and ask even when they judged the delay harmless; one wrote "Impact of delay: none blocking" and still built nothing. One specific rule, "build only the smallest uncontroversial part, then list 2 to 3 options with your pick", produced both progress and a surfaced decision every time.

In the framework: no rules on code style or layout. Overbuilding and new dependencies keep one short line each, with the dependency rule backed by a hook; the working agreement keeps the test rule and the vague-request rule, and the escalation table is gone.

5.3 RQ3: how should lessons carry over?

In a simulated past session, the user had said: use parseArgs from node:util for command-line flags, not hand-rolled parsing. The new session was asked to add a --min-streak <n> flag.

v3's learning loop had no path back into the next session: its journal was write-only. A rule prevents the mistake if it is read. A check catches it even when nothing is read: in 4 of the 5 check runs the agent first made the mistake, the test failed with its fix-it message, and the agent corrected itself. Following the lesson also improved the feature, since --min-streak=2 worked only when parseArgs was used.

In the framework: the journal is gone. /city-app:lesson turns each correction into a test, a blocked command or one AGENTS.md line, and logs it.

5.4 RQ4: does a separate reviewer help?

On small, clearly specified tasks the model was right alone 15 times out of 16. The reviewer never raised a false alarm, but it never caught a bug either, and reviewing took more work than building. The one miss was an unwritten requirement: the app already allowed habit names with spaces, but the task didn't say so, and the reviewer checked only what the task spelled out. This agrees with reports that reviewers and evaluators pay off on long, complex work rather than small tasks [6] [7].

In the framework: no council of roles. One reviewer is available on request, and unwritten requirements go into AGENTS.md as one-line gotchas ("habit names can contain spaces").

5.5 The framework under the same tests (E5)

In the framework: the test harness ships with it, as /city-app:rules:test and /city-app:rules:prune, because it caught two problems in the framework's own drafts.

5.6 Other models (E6)

With the larger model, the framework produced the small part plus options on the vague request in 3 of 3 runs (bare: 0 of 3) and smaller changes (serve: 35 added source lines against 64). The smaller model followed the vague-request rule poorly: it asked without building in 2 of 3 runs and built without offering options in the third (bare: built only, 3 of 3). No model added a dependency, with or without the framework. The vague-request rule helps mid-sized and larger models; smaller models may over-ask.

5.7 A short spec, a working app (E7)

For a small app with a clear spec, "state what you want and get it" is now default behavior, with or without a framework. What a framework can still add is what the spec leaves out: the user's preferences, guardrails, and lessons from earlier sessions.

In the framework: /city-app:start keeps the spec-first build and adds the proof: each requirement becomes a failing acceptance test before any code is written.

5.8 From evidence to design

6 Evaluation on real projects

6.1 Live checks

Each command and hook was checked in real Claude Code sessions on small projects, scored by fixed checks in the same way as the experiments. Table 13 gives the latest results, including the ones that fell short.

6.2 A web app and a command-line app

Two small projects in the repository were built with the framework. demo/habit-web is a web app with no front-end framework that uses every part of it. Its npm test runs the front-end checks: accessibility, console errors and layout on 3 pages at 3 screen sizes, a comparison against approved screenshots, and a test that fails on colors typed outside the design tokens; its own tests show the UI check catching 4 planted bugs, each with a fix-it message. demo/bookmarks-cli is the command-line app that start built from the E7 spec, tests first.

On habit-web, rules:test measured two AGENTS.md lines, 3 runs with and 3 without each. "Colors and spacing come from the tokens" made no difference (3 of 3 either way: agents followed the existing tokens anyway), so it was cut and replaced by the token test. "Ask before changing how streaks are counted" was unclear (2 of 3 either way; a unit test that pinned the rule did most of the work), so it was kept, and a later rules:prune re-test gave 2 of 3 again.

6.3 Defects found while validating

Validation also surfaced five defects in the project's own process. Each is now covered by a test or a rule, and all are logged in the lessons log.

• Three live checks could pass without the behavior under test: two because the demo project already contained the target state, one because an agent writing "npm test is failing" in its own words matched the hook's message. Offline tests now show that each check fails on a wrong outcome.
• A release shipped without a version bump, so claude plugin update reported "already at the latest version" and existing installs kept the old commands.
• One merge reached the main branch with a flaky failing test. Merges now wait for the full test run.
• The previous white paper went on describing v3 after the rebuild. A test now fails when this paper's commands, version or links drift from the plugin.
• The framework's own UI check found an accessibility defect in this page's first draft: a scrolling code block that keyboard users couldn't reach.

7 Discussion

What a framework should structure now. v3 tried to structure the code: naming, components, layouts, scope. In our tests that structure was either never read (the convention documents sat behind links no agent opened) or redundant (the rules we could measure, on scope and dependencies, changed nothing, because the agents already wrote working, dependency-free code). What they lacked was structure around the code: facts they could not infer, limits they must not cross, proof that the work is done, and memory of past corrections. v4 puts the framework's effort there, and each of its parts traces to a measured result (Table 12).

Reachability comes before content. v3's best rules had no effect because they never loaded. Whether a file loads depends on tool details, such as a CLAUDE.md pointer against an import, that change between versions [2]. The first thing to verify about any part of a framework is that the agent sees it.

Enforce the must-nevers; let checks act as memory. Hooks hold whatever the model reads or decides. A failing test also teaches: in E3 it carried a lesson into the next session with no written rule at all, which matches the practice of turning each mistake into a check [14] [15].

Front-end quality needs checks, not advice. On the web demo, a rule asking agents to use the design tokens made no difference, because they copied the tokens already in the code; a test that fails on a raw color guards the same thing even when the code stops setting the example. The framework treats accessibility and unintended visual changes the same way: as checks in npm test, not as advice.

Specific beats general. The rules that worked name a concrete action: build the smallest part and list options, add a test for new logic. This fits advice drawn from thousands of AGENTS.md files to be short, concrete and specific [18].

Text metrics are fragile. A detector bug nearly led us to drop a rule that worked (Table 10). Before trusting a measure that reads an agent's words, read a sample of the raw replies.

Specs are cheap now; context is not. A 4-sentence spec gave a working app with or without a framework, so for small builds, heavier spec-driven processes (still rated "assess" by the Thoughtworks Radar [19]) add little. What agents lack is what no spec contains: a user's standing preferences, limits and past corrections.

8 Limitations

• The experiments used one small Node.js command-line app and one empty project, and the evaluation two small demo projects. Larger codebases, or web apps built on a front-end framework such as React, may need structure this study did not test.
• Most runs used one model and one version of Claude Code, with 3-trial spot checks on two other models. The results should be re-run as models change.
• Cells had 3 to 8 trials, so only large effects are detectable. The evaluation of v4 used 1 to 3 runs per check: enough to show that each part works, not to estimate how reliably.
• Headless runs can't ask a person mid-task, which shapes how "asking" shows up.
• The options measure is a text heuristic, checked against labeled examples but still a heuristic.
• Several 2026 studies were available only as abstracts or secondary coverage [10] [11] [12], and each is a single, unreplicated study.
• The same project built the framework, the checks and the graders. Three graders proved too loose during validation (Section 6.3), and others may be.

9 Conclusion

City App Framework set out to give AI-assisted software and web development a structure. Testing its rulebook showed that most of that structure did nothing with current models: it never loaded, restated what the models already do, or over-corrected. The structure that survived is different in kind. It tells the agent what it must know (a short project file it actually loads), what it must never do (hooks), and what it must prove (acceptance tests written from the spec before the code, and front-end checks), and it keeps lessons as checks rather than notes. That is the framework now. The test harness ships with it, because the evidence expires: a rule that helps one model can be dead weight on the next.

10 Availability

The framework, the harness, the scorers, the raw per-run results and this paper are in the repository; the findings report has every number above, and the results folder holds the raw data. To install the framework in Claude Code:

claude plugin marketplace add balbonits/city-app-framework
claude plugin install city-app@city-app-framework

Then run /city-app:setup inside a project. To re-run the experiments, from a copy of the repository (each command prints how many sessions it ran):

node experiments/validate-scorer.mjs /tmp/check
node experiments/run.mjs --tasks remind --arms bare,kit --trials 5
node experiments/report.mjs

Acknowledgments. The experiments, the framework and this paper were produced with Claude Code, directed and reviewed by the author. Every number comes from scored runs stored in the repository.

Cite as: Dilig, J. (2026). Structuring AI-assisted software development: a case study of City App Framework (white paper, framework v0.2.0). https://github.com/balbonits/city-app-framework

References

Marked [abstract] where only the abstract or a search excerpt was read, and [secondary] where only third-party coverage was.

1. Linux Foundation. "Linux Foundation announces the formation of the Agentic AI Foundation." 9 December 2025. linuxfoundation.org
2. Anthropic. Claude Code documentation: memory, AGENTS.md and imports. code.claude.com/docs/en/memory
3. Anthropic. Claude Code documentation: best practices. code.claude.com/docs/en/best-practices
4. Anthropic. Claude Code documentation: permissions and hooks. code.claude.com/docs/en/permissions, code.claude.com/docs/en/hooks
5. Anthropic. "Effective context engineering for AI agents." 29 September 2025. anthropic.com
6. Anthropic. "Effective harnesses for long-running agents." 26 November 2025. anthropic.com
7. Anthropic. "Harness design for long-running apps." 24 March 2026. anthropic.com
8. Anthropic. "Building multi-agent systems: when and how to use them." 23 January 2026. claude.com
9. Anthropic. "Claude Code auto mode." 25 March 2026. anthropic.com
10. Gloaguen et al. "Evaluating AGENTS.md." arXiv:2602.11988, February 2026. [abstract, secondary]
11. Lulla et al. A study of AGENTS.md and agent efficiency. arXiv:2601.20404, January 2026. [abstract]
12. Zhong et al. "ImpossibleBench." arXiv:2510.20270, October 2025. [abstract]
13. S. Yegge. Gas Town. github.com/steveyegge/gastown
14. Every. The compound-engineering plugin. github.com/EveryInc/compound-engineering-plugin
15. M. Hashimoto. "My AI adoption journey." 5 February 2026. [secondary]
16. B. Böckeler. "Harness engineering." martinfowler.com, 2026; and OpenAI, "Harness engineering," February 2026. [secondary]
17. Vercel. AGENTS.md generation in create-next-app. github.com/vercel/next.js/pull/89850
18. GitHub. "How to write a great AGENTS.md: lessons from over 2,500 repositories." 19 November 2025. github.blog
19. Thoughtworks Technology Radar. Spec-driven development (Assess). November 2025. [abstract]