When Fine-Tuning Actually Wins

Good question, and the honest answer is uncomfortable.

Most companies shouldn’t fine-tune first. Prompting is faster, RAG is cheaper to start with, and both let you iterate without owning any infrastructure.

But there are specific cases where fine-tuning is the only right call. And once you see them clearly, the choice stops feeling like a religious debate and starts feeling like an engineering one.

Why prompting and RAG quietly lose

Prompting puts your logic in plaintext at runtime. Anyone who intercepts the request, leaks the system prompt, or jailbreaks the model sees your secret sauce.

RAG still needs a capable base model to reason over retrieved documents. Garbage in, garbage out. If the model cannot reason, the documents do not save you.

Both add latency and token cost on every single call. The bill compounds with scale.

And neither changes how the model thinks. They only change what it sees.

For a lot of tasks that is enough. For some it is not.

SecOps cases: cost reduction and risk reduction

Triage classifier. Train a small model to classify incoming alerts as critical, noise, or worth investigating. A fine-tuned small model handles this at roughly $0.000001 per call with no external API dependency. Compare that to paying a frontier model around $0.01 per call for ten million alerts a month. That is $100k a month versus about $100.

Log parsing. Security logs are messy, vendor specific, and rarely standard. A fine-tuned model learns your log format once. No prompt engineering, no few-shot examples eating context, no hallucinated field names. Faster, cheaper, and more accurate than prompting a general model to guess.

Phishing detection. Train on your company’s actual email corpus plus confirmed phishing samples. The model learns your internal communication style. It can flag this looks almost like how our CFO writes, but not quite. RAG cannot do this because the knowledge is not a document. It is a pattern.

Dividend-generating cases

Vertical SaaS co-pilot. If you sell software to dentists, lawyers, or logistics companies, fine-tune on domain language once and ship a model that speaks the jargon natively. Competitors using generic prompting sound generic. You sound like you have been in the industry for a decade. That is a moat.

Reducing inference costs at scale. Fine-tune a small model in the seven billion parameter range to do what you would otherwise need a seventy billion parameter model or a frontier API for, on your specific task. A product doing fifty million calls a month can cut costs by ten to fifty times by distilling that behavior into a fine-tuned open model. That is direct margin, not a science project.

Private or on-prem requirement. Healthcare, finance, defense. Data cannot leave the building. Prompting a cloud API is a compliance violation. A fine-tuned model running locally is the only option. That is not a preference, it is a contract requirement. Fine-tuning unlocks entire market segments that prompting cannot touch.

Removing the prompt as an attack surface. If your product relies on a system prompt to constrain behavior, that prompt can be leaked, injected against, or jailbroken. A fine-tuned model has the behavior baked in. There is no prompt to extract or override. For a security product, that is significant.

The honest summary

The takeaway

Fine-tuning is an infrastructure investment. You pay once, you own the behavior, and you run it forever.

Prompting and RAG are operating costs. They compound with scale, with traffic, and with every new feature that depends on them.

The right answer is rarely all of one. The right answer is knowing which lever you are pulling, and why.

Most teams should start with prompting. A few should move to RAG. A small but growing number should fine-tune.

The trick is being honest about which one you actually are.