Measuring What Works

Asked a founder the other day how she knew AI was working for her agency. She said the team was using it. I asked how she knew it was actually improving things. She paused for a bit and said she could feel it.

I hear this a lot. Agencies that have adopted AI tools, sometimes quite seriously, but have no way of measuring whether the investment is paying off. The team is faster, probably. The work is better, possibly. But there's no baseline and no tracking, so it's all gut feel.

One agency I worked with decided to measure it properly. They picked three workflows that they'd rebuilt around AI, content research, first-draft production, and competitive analysis, and tracked time-to-completion and rework rates for eight weeks. Before and after.

Content research dropped from an average of six hours to about ninety minutes. First-draft production was faster but the rework rate went up, which meant the net saving was smaller than they'd expected. Competitive analysis was roughly the same speed but noticeably more thorough.

The interesting thing was that having the numbers changed the conversation entirely. They stopped arguing about whether AI was worth it and started arguing about where to apply it next. The investment case for the next phase wrote itself because they had actual evidence instead of enthusiasm.