You launch your AI product. Users love the demo. You're excited.
Three months later, you're stuck. The model hasn't improved. User complaints pile up. Competitors who launched after you are somehow better.
What happened?
You forgot to build the flywheel.
Most teams treat AI product launches like traditional software: build it, ship it, move on. But AI products are fundamentally different. They should get better with every user interaction.
The companies winning in AI aren't necessarily the ones with the best initial models. They're the ones who built data flywheels that turn every user interaction into continuous improvement.
After building AI products across 8 portfolio companies, we've learned: The flywheel isn't a nice-to-have. It's the durable advantage.
What Is a Data Flywheel?
A data flywheel is a self-reinforcing system where:
- Users interact with your AI product
- Data is logged (queries, responses, feedback, context)
- Quality is evaluated (automated + human review)
- Models improve (fine-tuning, prompt optimization, retrieval improvements)
- Better product attracts more users
- Cycle repeats, faster each time
The magic happens when this cycle compounds. Each iteration makes the next one easier and more effective.
The Netflix Example Everyone Knows
Netflix didn't win by having the best initial content library. They won because:
- Users watch → Netflix logs what they watch, skip, rewatch
- Netflix learns → Recommendations improve
- Better recommendations → Users watch more
- More watching → More data → Better recommendations
The flywheel made Netflix's product uniquely valuable. Every hour of viewing made their recommendations better than any competitor could replicate.
The AI Engineering Version
For AI products, the flywheel looks like this:
Month 1:
- 1,000 users
- Log 10,000 interactions
- Manually review 500 samples
- Identify 3 failure modes
- Ship fixes
Month 3:
- 5,000 users (better product attracted more)
- Log 50,000 interactions
- Automated evals flag issues
- LLM judge pre-screens data
- Human review focuses on edge cases
- Fine-tune model on curated data
- Ship improvements weekly
Month 6:
- 20,000 users
- Log 200,000 interactions
- Automated pipeline catches 85% of issues before users notice
- Continuous fine-tuning
- Model specialized to your users' exact needs
- Competitors can't catch up (you have proprietary data + refined process)
Our Flywheel Framework: The 6 Stages
After implementing flywheels across portfolio companies, we've standardized on a 6-stage process:
Stage 1: Comprehensive Logging
What to log:
- User query (exact input)
- Retrieved context (for RAG systems)
- Model response
- Latency metrics
- Costs (tokens, API calls)
- User feedback (explicit: thumbs up/down; implicit: did they retry, edit, copy, etc.)
- Metadata (user ID, session ID, timestamp, model version)
Implementation:
# Every AI interaction gets logged
log_entry = {
"query_id": generate_id(),
"timestamp": datetime.now(),
"user_id": user.id,
"query": user_input,
"retrieved_chunks": chunks, # For RAG
"model_response": response,
"model_version": "gpt-4o-2024-08-06",
"latency_ms": latency,
"cost_dollars": calculate_cost(tokens),
"user_feedback": None, # Will be updated if user reacts
"session_context": session.context
}
await db.insert("ai_interactions", log_entry)
Key principle: Log everything. Storage is cheap. Missing data is expensive.
One of our portfolio companies didn't log retrieved chunks for their RAG system. When users complained about wrong answers, we couldn't tell if it was retrieval or generation failing. We had to rebuild 3 months of logs by replaying queries. Don't make this mistake.
Stage 2: User Feedback Collection
The easiest signal: binary feedback.
Every AI response gets thumbs up/down buttons. Simple. Fast. Users actually do it.
Our learnings:
- Thumbs up/down generates 10x more feedback than "rate 1-5 stars"
- Users will click thumbs down but won't write explanations
- Follow up thumbs down clicks with optional feedback form (5-10% fill it out, but those are gold)
More sophisticated signals:
-
Implicit feedback (doesn't require user action):
- Did user copy the response? (Good sign)
- Did user immediately ask follow-up? (Could be good or bad)
- Did user retry with rephrased query? (Usually bad)
- How long did they read the response? (Short read = probably bad)
-
Explicit multi-dimensional feedback (for power users):
- Accuracy: 1-5 stars
- Clarity: 1-5 stars
- Completeness: 1-5 stars
We use binary feedback for 95% of users, multi-dimensional for internal testing and select beta users.
Stage 3: Automated Quality Evaluation
You can't manually review 10,000 interactions per day. Automation is mandatory.
Our eval stack:
3a. Rule-Based Checks (Fast, Cheap)
def evaluate_response(response, context):
checks = {
"no_hallucination": not contains_facts_outside_context(response, context),
"appropriate_length": 50 < len(response.split()) < 500,
"no_refusal": "I cannot" not in response,
"no_apology": not response.startswith(("I'm sorry", "I apologize")),
"citations_present": "[" in response and "]" in response,
"no_markdown_errors": validate_markdown(response)
}
return checks
3b. LLM-as-a-Judge (More Expensive, Catches Nuanced Issues)
We use Claude Sonnet 4.0 to evaluate:
- Factual accuracy
- Tone alignment with brand voice
- Completeness (did we answer the full question?)
- Clarity for target audience
Key insight: LLM judges need to be aligned with your domain expert. We follow Hamel Husain's "critique shadowing" approach:
- Domain expert labels 100-200 examples (Pass/Fail + detailed critique)
- Use those examples as few-shot demonstrations in judge prompt
- Test judge against expert on held-out set
- Iterate until >85% agreement
3c. Specialized Checks (Domain-Specific)
For our medical documentation AI:
- Medical term hallucination check
- Dosage accuracy verification
- Citation verification (do cited studies exist?)
- Plain language check (no unexplained jargon)
For our legal contract AI:
- Clause completeness check
- Jurisdiction-specific requirement validation
- Risk flag detection
Stage 4: Human Review (Strategic Sampling)
Automated evals catch most issues. Humans catch the subtle ones that matter most.
Our human review strategy:
Daily (15-30 minutes):
- Review 20 randomly sampled interactions
- Review all negative feedback from that day
- Review all automated eval failures
Weekly (2 hours):
- Deep dive on 50-100 interactions
- Identify new failure modes
- Update automated eval criteria
- Label data for fine-tuning
Monthly (half day):
- Analyze trends over time
- A/B test evaluation
- Update domain expert alignment
- Review competitor products
Key principle: Human review should focus on learning, not labeling. If you're spending hours labeling data that could be automated, your process is wrong.
Stage 5: Model & System Improvements
This is where the flywheel produces tangible value.
5a. Prompt Engineering
Based on eval data, we continuously refine prompts:
# Example: Legal contract AI prompt evolution
# Version 1 (Launch):
"Analyze this contract and answer the user's question."
# Version 2 (After 1 week of data):
"Analyze this contract and answer the user's question.
Cite specific sections. Use plain language."
# Version 3 (After 1 month):
"You are a legal contract analyst. Analyze the contract below
and answer the question precisely.
Requirements:
- Cite specific sections with [Section X]
- Explain in plain language (avoid legalese)
- If asking about risks, list them as bullet points
- If information is missing, explicitly state that
Contract: {contract}
Question: {question}"
# Version 4 (After 3 months - Dynamic Few-Shot):
"You are a legal contract analyst.
Here are examples of good analyses:
{retrieve_similar_good_examples(question)}
Now analyze this contract:
{contract}
Question: {question}"
Our velocity: We iterate on prompts weekly based on eval data.
5b. Retrieval Improvements (For RAG Systems)
Data reveals retrieval failures:
- Chunking issues: Relevant info split across chunks → Adjust chunk size/overlap
- Embedding issues: Semantic search misses obvious keywords → Add hybrid search (vector + keyword)
- Ranking issues: Relevant chunks not in top-k → Implement re-ranking
Example from medical documentation RAG:
- Month 1: Vector search only. Precision: 72%
- Month 2: Added keyword search. Precision: 81%
- Month 3: Added cross-encoder re-ranker. Precision: 89%
- Month 4: Fine-tuned embedding model on domain data. Precision: 93%
Each improvement was data-driven from production failures.
5c. Fine-Tuning
Once you have curated data (labeled good/bad by automated evals + human review), fine-tuning becomes viable.
Our fine-tuning cadence:
- First fine-tune: After 500-1,000 curated examples
- Subsequent fine-tunes: Monthly or when we accumulate 200+ new curated examples
What we fine-tune for:
- Tone and style (brand voice)
- Output formatting (structured responses, consistent citations)
- Domain-specific knowledge (legal terminology, medical protocols)
- Error correction (don't repeat past failures)
Results from legal contract AI:
- Before fine-tuning: GPT-4 with prompt engineering. 83% eval pass rate.
- After fine-tuning: GPT-4o fine-tuned on 2,000 examples. 91% eval pass rate.
- Bonus: Fine-tuned model is 30% cheaper (fewer tokens needed in prompt).
Stage 6: Continuous Monitoring & Iteration
The flywheel doesn't stop. You monitor and iterate forever.
What we monitor:
Daily Dashboards:
- Eval pass rates (by model version)
- User feedback distribution (% thumbs up vs down)
- Latency (P50, P95, P99)
- Cost per query
- Error rates (by error type)
Weekly Reviews:
- New failure modes
- Eval drift (are our evals still measuring what matters?)
- User behavior changes
- Competitor analysis
Monthly Reviews:
- Overall quality trajectory
- ROI of improvements (did that fine-tune justify the cost?)
- Flywheel velocity (are we improving faster or slower?)
- Strategy adjustments
Real Results: The Flywheel in Action
Case Study 1: Legal Contract Analyzer
Context: AI that analyzes commercial contracts for risk, obligations, and key terms.
Flywheel Timeline:
Month 1 (Launch):
- 50 beta users
- GPT-4 with engineered prompts
- Manual review of all outputs
- Pass rate: 78%
Month 2:
- 200 users
- Added automated evals
- LLM-as-a-judge deployed
- Identified: Model misses non-standard clause language
- Improved prompt with examples of non-standard language
- Pass rate: 83%
Month 3:
- 800 users
- 8,000 interactions logged
- Fine-tuned GPT-4o on 1,200 curated examples
- Pass rate: 91%
- User satisfaction: 87% thumbs up
Month 6:
- 3,500 users
- 35,000 interactions logged
- Continuous fine-tuning pipeline
- Pass rate: 94%
- User satisfaction: 92% thumbs up
- Competitor enters market with GPT-4 + prompts (78% pass rate)
- Grey Haven product demonstrably better with 6 months of flywheel data
Key Insight: After 6 months, our model is specialized to legal contract analysis in ways no competitor can replicate without the same data.
Case Study 2: Medical Documentation Assistant
Context: AI that helps doctors find dosage guidelines, treatment protocols, and contraindications.
Flywheel Impact:
Without Flywheel (First Attempt):
- Launched with GPT-4
- No logging beyond basic analytics
- Doctors complained: "It's often wrong but I don't know when"
- Abandoned after 2 months
With Flywheel (Second Attempt):
- Day 1: Comprehensive logging
- Week 1: Automated safety checks (no dosage hallucinations)
- Week 2: LLM-as-a-judge for medical accuracy
- Week 3: Daily human review by MD on staff
- Month 2: Fine-tuned on 600 curated examples
- Month 3: Continuous improvement pipeline
Results:
- Zero hallucinated dosages (caught by automated checks)
- 94% accuracy on medical facts (LLM judge + human review)
- 89% thumbs up from doctors
- Model learns hospital-specific protocols and preferences
The difference? The flywheel turned every interaction into improvement. Doctors trust it because it visibly gets better.
When Flywheels Fail (And How to Fix Them)
Not all flywheels work. Here's what we've learned from failures:
Failure Mode 1: Logging Without Analysis
Mistake: Team logs everything but never looks at data.
Symptom: Gigabytes of logs, zero insights.
Fix: Schedule mandatory weekly data reviews. If no one looks at data, you don't have a flywheel. You have expensive storage.
Failure Mode 2: Analysis Without Action
Mistake: Team reviews data, identifies issues, but never ships fixes.
Symptom: Great insights, same bugs persist for months.
Fix: Tie data reviews to deployments. Every data review should produce a concrete action (prompt update, fine-tune, new eval, etc.).
Failure Mode 3: Improvement Without Measurement
Mistake: Team ships changes but doesn't measure impact.
Symptom: Constant iteration, unclear if product is actually better.
Fix: A/B test everything. Every prompt change, every fine-tune, every retrieval update. Measure impact on eval pass rates and user satisfaction.
Failure Mode 4: Optimization for the Wrong Metric
Mistake: Team optimizes for automated eval pass rate but users aren't happier.
Symptom: Eval scores improve, user satisfaction stagnates or declines.
Fix: Regularly validate that automated evals correlate with user satisfaction. If positive feedback rate doesn't increase when eval pass rate increases, your evals are wrong.
Building Your Flywheel: The Pragmatic Approach
Don't try to build the perfect flywheel from day 1. Start simple, iterate.
Week 1: Logging
- Log every AI interaction (query, response, metadata)
- Add thumbs up/down buttons
- Deploy to 10-50 users
Goal: Establish data pipeline.
Week 2-3: Manual Review
- Review 100% of interactions daily
- Identify top 3 failure modes
- Fix obvious bugs
Goal: Understand what good looks like.
Week 4: Automated Evals (Level 1)
- Write 10-20 rule-based checks
- Run on all logged data
- Track pass rates over time
Goal: Scale evaluation beyond manual review.
Week 5-6: LLM-as-a-Judge
- Domain expert labels 100-200 examples
- Build LLM judge prompt
- Test against expert labels
- Iterate to >85% agreement
Goal: Automate nuanced evaluation.
Week 7-8: Continuous Improvement Pipeline
- Weekly prompt updates based on eval data
- Sample-based human review (not 100%)
- Deploy improvements confidently
Goal: Establish rhythm of data → insights → improvements.
Month 3+: Advanced Techniques
- Fine-tuning on curated data
- Dynamic few-shot retrieval
- Automated data labeling
- Cross-model evaluation
Goal: Compound improvements accelerate.
The Compounding Effect
Here's why the flywheel is so powerful:
Month 1: You learn 10 things about your product's failures. Month 2: You fix those 10 things. You learn 15 new things (you're looking at more data). Month 3: You fix those 15 things. You learn 20 new things (even more data, better evals).
By Month 6, you've:
- Fixed 60+ issues
- Built 100+ automated evals
- Fine-tuned your model 3-4 times
- Specialized to your users' exact needs
Competitors launching now need to:
- Learn the same 60+ things (takes 6 months)
- Build the same evals (takes months)
- Collect their own data (can't access yours)
You're 6 months ahead. That gap only widens.
The Advantage Is the Data + Process
People obsess over model selection: "Should we use Claude or GPT-4?"
That's the wrong question.
The right question: "How do we continuously improve whatever model we choose?"
A team with GPT-3.5 + a great flywheel will beat a team with GPT-4o + no flywheel.
Why? Because:
- The flywheel team's model improves weekly
- The flywheel team's prompts are specialized to their users
- The flywheel team learns from every interaction
- The flywheel team builds a durable advantage (proprietary data + refined process)
The model is a commodity. The flywheel is the advantage.
Common Objections (And Our Responses)
"We don't have enough users yet."
Start with 10 users. That's enough to begin logging, get initial feedback, and identify failure modes. The flywheel works at any scale.
"We don't have engineers to build this."
Start manual. Log to spreadsheets. Review data weekly. Ship fixes. The process matters more than the tooling.
"Our users won't give feedback."
Make feedback absurdly easy. Thumbs up/down buttons everywhere. You'll get 5-10% feedback rate. That's enough. Also leverage implicit feedback (did they copy the response, retry, etc.).
"Fine-tuning is expensive."
Start with prompt engineering. That's free and can get you 70-80% of the way. Fine-tune only when you have 500+ curated examples and prompt engineering plateaus.
"What if our data is too noisy?"
All data is noisy. That's why you have evals, human review, and curation. The flywheel filters noise into signal.
The Truth About AI Products
AI products don't succeed because they're perfect at launch.
They succeed because they get better faster than competitors.
The flywheel is how you get better faster.
- Netflix beat Blockbuster not with better initial content, but with a flywheel that made recommendations unbeatable.
- Spotify beat Pandora not with better initial playlists, but with a flywheel that learned every user's taste.
- Your AI product will beat competitors not with a better initial model, but with a flywheel that makes your product uniquely yours.
Start building your flywheel today. Log everything. Review data. Ship improvements. Repeat.
The compounding starts now.
Want to see how we implement flywheels across our portfolio? Check out our RAG Evals post or reach out to discuss your AI product.
