Voice and tone analyzer
A self-serve tool to help teams test model responses against guidelines
- Role
- Content designer
- Company
- OKX
- Stack
- Claude
- Python
- GitHub Actions
- Status
- Prototyped concept
Problem
AI made content creation faster, but teams lacked a way to check whether generated content followed brand guidelines.
Solution
Turning our voice, tone, and brand guidelines into a self-serve evaluation tool
A real failure message behind the problem
In this review, I traced what the failure message left unanswered: where the money was, what fees applied and how to get help. Tone alone couldn't resolve those gaps.
What I did
Step 1. Deterministic checks
Started by reviewing known issues
I used market-specific research to train the LLM to flag confusing financial terms.
Step 2. Content architecture
Made writing guidelines usable by AI
I structured 17 rules with IDs, component scope, and good/bad examples, separating what code could check from what required AI judgment.
Step 3. Deterministic checks
Clear constraints, predictable output
I used repeatable code checks for character limits, contrast, terminology and formatting. These constraints have a clear answer rather than depending on a model's judgment.
Step 4. Evaluation and iteration
The misses were the useful part
To test the checks, I collected 74 real strings from the product and wrote down the correct answer for each one: flag this, pass that. I built the rules against 50 of them and kept 24 aside, so I could test on strings the rules had never seen.
Then I ran the tool and compared its answers to mine. Every mismatch was a finding: sometimes a rule caught something I'd marked wrong, and sometimes my label was the problem. Fixing those mismatches, not chasing a score, is what improved the checks.
- development examples matched
- 73/74
- held-out examples matched
- 23/24
- documented known gap
- 1
Step 5. Human-in-the-loop design
A better tone can't change the facts
These prototype screens show the review surface: tone feedback, flagged issues and a before-and-after rewrite. The displayed scores aren't part of my evaluated rule-check results.
The prototype exposes issues and suggested rewrites. For the next step, I would make acceptance and reasoned rejection part of the review, so factual errors become evaluation examples.
Proposed review flow · illustrative output
Original
Order failed. Your recurring buy plan is still active.
Suggested rewrite
Your order is processing. Your recurring buy plan is active.
The rewrite sounds calmer but changes “failed” to “processing.” That's a factual error, not a tone improvement.
How I organized design-review annotations
My annotation categories separate content, accessibility and interaction issues, including new and existing keys. This is design-review organization, not a feedback flow implemented in the analyzer.
What this proves
Content design is part of the system
I defined the standards, structured the guidance, built the prototype and used evaluation failures to improve it. The work moves review toward self-serve checks rather than repeated interpretation.
Demonstrated
Rule architecture, code-based quality checks, labeled comparisons and iterative correction.
Still to validate
AI tone quality, hallucination handling and the proposed reviewer-feedback flow. The AI layer hasn't been evaluated yet.