same soup, different spoon

A few weeks back I wrote about how I had been rebuilding the same app over and over with different agent team configurations.

In my workflow, everything I create was built feature by feature, step by step. Myself and my team of role-based agents plan it, architect it, build it, test it, review it, close it, capture all the documentation along the way - and the study was me looking for a pat on the back of “Wow - this is so good. No room for improvement here.”

That… Is not what I learned.

What I learned was that while I was producing incredibly detailed documentation, and product that looked good/worked. I was doing so at an incredible premium. I learned this out of curiosity; when I simply gave Fable the original technical scope of work I used to produce a project of mine that took about 2 days, and it knocked it out in 45 minutes.

Now there’s an important distinction here - because yes Fable built it and it worked. But to a different standard. There were a few security gaps in the build, which could easily have been fixed with a second pass - but the more frustrating thing to me was that I only got about half of what I normally produce. There wasn’t any documentation - no record of what was built. Fable made a lot of decisions and none of them were captured anywhere, I had the car but not the manual.

But still… 2 days to 45 minutes. I had to be able to find a healthy middle ground right?

So here’s where I started experimenting. I ran a similar pass with Opus instead of Fable - nearly identical results.

I thought okay, maybe it’s the continuity? So I decided to try and go hands off. I took my regular workflow, friday (named so because I originally built it on a friday) and gave both Fable & Opus god mode in it - built the whole thing again, but this time, break it down into features like we normally do, and iterate through each feature’s stages of planning, architecture, building, testing, reviewing, closing out the documentation - don’t ask me any questions - make the decisions yourself.

What happened here? I got some gain in speed and I got all of my documentation, BUT what I gained in speed, I lost in continuity & usability. Both of the user interfaces produced were not of the same quality of when I let Fable & Opus loose on their own without breaking things down into features and without asking them for documentation.

What appeared to work well was the agent having continuous build context across the features, tying each feature seamlessly into the next. So then I tried the third configuration, dedicated lanes with continuous context in each one - but still going feature by feature. Lane 1 was Build - build a feature, hand it off to a separate agent in Lane 2 for testing, loop back to Lane 1 if it needs fixing, then onto Lane 3 for review, looping back to Lane 1 & 2 as needed - all the while creating the documentation along the way.

Sweet. Christ.

The time suck. I abandoned these runs because after several hours we were only 4 or 5 features in. This was not the result I was looking for.

It seemed like there was not going to be a way for me to have my cake and eat it too. I had to think about it differently.

My goals:

• Fast(er)
• Well Documented
• Usable
• Repeatable

So I redesigned my loop from the ground up. Completely changed my strategy for documentation, and abandoned the per feature breakdowns which seemed to be hurting efficiency.

Instead of breaking the project down into high level features, and iterating over each feature with a detailed loop to build & document it. I decided to front load as much of that process as I could (ye olde waterfall approach, I know). In my previous approach, I had the flexibility to steer the build in flight if I knew it was heading off the rails because I was only building one feature at a time - and I had constructed a methodology for reassessing and redirecting the build. Now, I was going to lose that flexibility, so I had to be damn sure that I was ready before I pulled the trigger.

My technical scope of work changed shape - and it became a much more intense collaborative session, attacking the same product from as many angles as possible. The important thing though, was not to get too prescriptive with the details of implementation up front - more to capture the make or break decisions, must have functionality. The goals, the constraints, not the granular steps to achieve them. This would become the project’s “oracle” document; architecture decisions that we were able to capture would be captured up front as ADRs, and I’d have to figure out how to capture the others later.

Once the doc was built, it would be the basis for the builder’s plan - and I suppose this is important - I realize that not every project can be built as a one-shot. Some will just be too big. BUT what I had learned was that when I define the seams, they’re the seams that make sense to me (the human) but maybe not the seams that make sense to the model. The model needed to set up its own plan based on the requirements, draw its own seams and manage the handoffs on its own to ensure continuity. So that’s exactly what happens. This kicks off with the builder creating its own plan & task list, then kicks off a combined testing, building, and architecture lane. Failing tests are written & committed first, then we build until green - a hook prevents the builder from editing a committed test and forces it to be surfaced if the test was legitimately wrong and needs to be rewritten. Supplementary architecture decisions that are hard to reverse get surfaced throughout the build, recorded with the rest of the ADRs - smaller decisions that happen throughout the build are narrated by the builder and then automatically synthesized from the transcript & build telemetry after the fact.

All of this is orchestrated by a team lead that maintains context through the whole build, and delegates to an agent team. Fresh context agents are used for adversarial reviews and security hardening passes. I’m too lazy to write about how we manage patches, bugs, increments - but I have defined all of that too and tied it into the workflow.

The end result? I can move a hell of a lot faster, at a significantly lower token spend, while still maintaining the documentation bar that I set for myself. Is it perfect? Definitely not, but it’s progress - not perfection. What I find more interesting is not necessarily the building itself - but the process by which the building is done. Things can happen so quickly and easily now, it’s easy to lose track of the why and come back to a codebase of pure chaos if we’re not deliberate about how we build with AI - and I want my at-home AI software factory to be nice and tidy. Anyway, I’m sure I’ll be back here in a few weeks with some new revelation - but for now, this is where I’m at. Temporarily satisfied.