Issue #1: 

Response 

issues

The model’s responses were verbose, cluttered, and overly formal, often using language that sounded intelligent but failed to communicate efficiently. Phrases like “please consider adjusting the required parameters to achieve the desired outcome” made simple tasks feel dense and bureaucratic.

This over-formality created a poor user experience where users had to parse long, complex sentences to extract basic information. The communication style projected effort instead of expertise. It slowed workflows, caused cognitive fatigue, and made the assistant feel mechanical rather than capable.

The challenge was clear: How do we make the AI sound intelligent but not academic, professional but human-readable?

Process

Once the assistant’s personality traits (courteous, semi-friendly, competent, concise) were defined, I led a focused effort with two content designers to rebuild the assistant’s communication model.

We began by writing golden path conversations – ideal interactions that demonstrated how the assistant should communicate across a variety of real user scenarios. These examples helped us capture the right balance of tone, precision, and brevity before writing prompts.

To validate the language direction, we partnered with UX research to test responses with real representative users to determine:

1. - How clear and direct the assistant’s responses felt.
2. - Whether the tone aligned with user expectations for the tool.
3. - What level of detail provided value without creating overload.

We tested variations across tone, sentence length, and structural complexity, collecting both quantitative ratings and qualitative feedback.

Solution

Insights from testing directly informed the refinement of system prompts and training data.

• Prompts were rewritten to encourage active voice, remove redundancy, and avoid filler language.
• The model was guided to front-load key information, keeping secondary context brief and optional.
• Example-based learning from golden paths reinforced ideal phrasing and cadence.
• 
  

The result was a communication framework that produced responses that were concise, confident, and context-aware, mirroring how a skilled human professional would respond under time pressure.

Impact

• Reduced response length without loss of meaning, improving information retrieval time.
• Improved readability and comprehension, especially for users multitasking or skimming content.
• Higher satisfaction scores in UXR studies, with users describing responses as “clear,” “efficient,” and “to the point.”
• Decreased revision cycles for content and prompt tuning, since tone and structure were now standardized.
• 
  

In essence, the assistant moved from sounding like a document to sounding like a colleague who knows exactly what you need.

Issue #2: 

Inaccuracy

Accuracy was mission-critical. Unlike a casual assistant or entertainment chatbot, this product existed to help users make real advertising decisions – where incorrect data or misleading information could directly impact business outcomes, financial results, and user trust.

Early tests revealed significant issues: the model sometimes fabricated metrics, misstated campaign details, or overconfidently presented guesses as facts. These failures weren’t just usability concerns, they were also credibility risks.

The design challenge was to create a system where accuracy was measurable, improvable, and sustainable, without slowing iteration or adding unnecessary manual overhead.

Process

Our approach combined automation, human feedback, and prompt design to systematically drive accuracy improvements.

1. 
   
2. Defining What “Accurate” Meant
3. I worked with product and engineering partners to clearly define what qualified as an accurate response for this domain. This included identifying acceptable tolerances for contextual responses (e.g., estimates, summaries, or advice) and separating factual accuracy from stylistic preference.
4. 
   
5. Labeling and Benchmarking
6. We began by reviewing and labeling large batches of model responses, categorizing them by factual correctness, clarity, and confidence level. These human-labeled examples became our initial truth set and was the baseline data used for model tuning and evaluation.
7. 
   
8. LLM-as-a-Judge Implementation
9. To scale evaluation, I collaborated with an engineer to design a system prompt for a “judge model.” This secondary LLM was tasked with identifying factual claims in the primary model’s output and rating their accuracy against source data.

• 
  
• We generated synthetic datasets containing both accurate and deliberately inaccurate responses.
• 
  
• The judge used structured evaluation criteria to assign each response an accuracy confidence score, flagging uncertain or unverifiable statements.
• 
  
• These outputs informed retraining and fine-tuning cycles for the assistant model.
• 
  

1. Human Rater Guidelines
2. In parallel, I authored rater guidelines for the human feedback phase. These guidelines defined clear, unambiguous evaluation criteria and examples for each accuracy tier (e.g., “completely accurate,” “partially accurate,” “inaccurate”).
3. The clarity of these instructions was critical, since external raters would be performing the evaluations. I ran guerrilla tests with internal content designers, engineers, and PMs to ensure the guide was intuitive and free of interpretation gaps before deploying it to external raters.

Solution

By merging automated judgment (LLM-as-a-judge) with structured human evaluation, we built a feedback loop that continuously identified and corrected factual weaknesses in the model.

• The system prompts established logical constraints and reinforced accurate phrasing patterns.
• The synthetic datasets improved detection of subtle inaccuracies and overconfident tone.
• The rater guidelines ensured consistent human evaluation standards across multiple testers.
• 
  

This ecosystem became our accuracy evaluation framework – a repeatable process that could scale across new product areas and model versions.

Impact

• Marked reduction in factual errors and unsupported claims in model outputs.
• Improved reliability in user-facing responses, strengthening user confidence and adoption.
• Faster iteration cycles, as automated judging surfaced problem areas early without waiting for manual review.
• Cross-functional alignment between content design, product, and engineering, grounded in a shared understanding of what “accurate” meant in practice.

Ultimately, this work ensured the assistant could be trusted as a professional-grade AI partner – one that users could rely on for decision-making, not just conversation.