Key takeaways
- Generative AI pays off on frequent, text-heavy tasks where a person reviews the output, such as document extraction, knowledge search and draft replies.
- Exact calculations, low-volume tasks and high-stakes decisions without human review are poor uses and usually waste money.
- Start with good prompts, use RAG when the model needs to know your content, and fine-tune only when it needs to behave a certain way.
- Record a baseline for two to four weeks before the pilot, and count review time and running costs when you measure savings.
- A narrow pilot on one use case can reach a scale, adjust or stop decision in about eight weeks.
Most companies have now tried generative AI for business in some form: a chatbot pilot, Copilot licenses, a developer experimenting with an API key. Far fewer can point to a process that is measurably cheaper or faster because of it. The gap usually isn't the technology. It's picking the wrong problems, skipping the data work, and never measuring the result. This guide covers where large language models earn their keep, where they don't, and how to run a pilot that gives you a real answer.
Where generative AI pays off
The best use cases share a pattern: lots of unstructured text, a task people do many times a week, and a human who reviews the output before it matters. Some examples we'd put near the top of any list:
- Pulling data out of documents. Invoices, purchase orders, bills of lading, insurance forms and contracts arrive in dozens of layouts. A model can extract the fields into structured data, with low-confidence results routed to a person.
- Internal knowledge search. Staff ask questions in plain English and get answers drawn from your policies, product manuals and past tickets, with links to the source document.
- Support triage and draft replies. Incoming emails and tickets get classified, routed and paired with a suggested answer that an agent edits and sends.
- First drafts. Proposals, job descriptions, product descriptions, RFP responses and routine reports. The model gets you from blank page to 70% done, and a person finishes it.
- Summaries. Long email threads, meeting transcripts, case files and call notes condensed into what happened and what needs to happen next.
- Software development. Code assistants speed up routine coding, test writing and working in unfamiliar codebases, as long as every change still goes through review.
Where it doesn't pay off
Just as useful is knowing where to say no. These are the patterns we see waste the most money:
- Exact calculations and rules. Pricing, tax, payroll and inventory math should run in normal code. A language model can call that code, but it shouldn't do the arithmetic itself.
- High-stakes decisions with no human review. Credit approvals, medical guidance and legal conclusions need an accountable person. Use AI to prepare the case, not to decide it.
- Low-volume tasks. Automating something that happens ten times a month rarely covers the build and running costs.
- Processes nobody has defined. If three employees handle the same request three different ways, AI will add a fourth. Fix the process first.
- Messy, out-of-date source content. A knowledge assistant built on a SharePoint site full of old policy versions will confidently quote the wrong one.
- Unsupervised public-facing chatbots. A bot that can promise refunds or discuss topics outside its lane is a liability without tight guardrails and testing.
| Use case | Fit | Why |
|---|---|---|
| Invoice and document extraction | Strong | High volume, clear right answer, easy to review exceptions |
| Internal knowledge assistant | Strong, if content is clean | Saves search time, answers can cite sources |
| Support draft replies | Strong | Agent stays in control, quality is easy to measure |
| Sales and marketing first drafts | Moderate | Saves time, but value depends on how much editing is needed |
| Financial calculations | Poor | Deterministic code is cheaper and always correct |
| Autonomous customer-facing decisions | Poor to risky | Errors are costly and hard to catch in time |
RAG vs fine-tuning
Once you pick a use case, the next question is how to get the model to know about your business. There are three main options, and most companies should try them in this order.
1. Good prompts and structured inputs
Start here. Clear instructions, a few worked examples and a defined output format solve a surprising share of problems with no extra infrastructure. Document extraction and classification often need nothing more.
2. Retrieval-augmented generation (RAG)
RAG means the system searches your documents for the passages relevant to a question, then gives those passages to the model along with the question. The model answers from your content rather than from memory. It's the right choice when the knowledge changes often, when answers need to cite sources, and when different users are allowed to see different documents. Most internal assistants should be built this way. The hard part isn't the model. It's cleaning the content, splitting documents sensibly, keeping the index current and respecting permissions.
3. Fine-tuning
Fine-tuning trains a model further on your own examples. It's useful for teaching a consistent format, tone or specialized classification, especially at high volume where a smaller tuned model can be cheaper to run. It's a poor way to teach facts, because the knowledge is frozen at training time and the model can't tell you where an answer came from. We rarely recommend fine-tuning as a first step.
Rule of thumb: use RAG when the model needs to know things, and fine-tuning when it needs to behave a certain way. Many successful systems use neither and just have well-built prompts.
Keeping your data private
Data privacy is the most common reason AI projects stall in legal review. It's solvable, but it needs a deliberate design.
- Use business terms, not consumer apps. Enterprise API offerings from the major providers, and services like Azure OpenAI running inside your own Azure tenant, generally state that your inputs aren't used to train their models. Read the actual terms, including data retention and where data is processed.
- Keep the permission model. A RAG system must only retrieve documents the person asking is allowed to see. If it ignores permissions, it becomes the fastest way to leak salary data across the company.
- Minimize what you send. Strip or mask personal and financial data the task does not need before it reaches the model.
- Log and review. Record prompts, retrieved sources and outputs so you can audit answers and investigate problems, and set a retention period for those logs.
- Plan for prompt injection. Documents and emails can contain text that tries to redirect the model. Limit what actions the system can take, and require a human to approve anything that sends, pays or deletes.
- Check your regulations. HIPAA, financial rules and state privacy laws still apply when a model is involved. If protected health information is in scope, you need a provider that will sign a Business Associate Agreement.
Measuring ROI
If you don't measure the process before the pilot, you can't prove anything after it. Pick a specific task and record a baseline for two to four weeks:
- Volume: how many times the task happens per week.
- Time per task: minutes of staff time, measured rather than estimated.
- Quality: error rate, rework or customer complaints tied to the task.
- Cycle time: how long the customer or internal requester waits.
Then measure the same things during the pilot, and add the running cost: model usage, hosting and the time people spend reviewing outputs. The simple version of the math is hours saved per month, multiplied by a loaded hourly cost, minus the monthly running cost, compared against the build cost. Do the math honestly. If a person still has to check every output carefully, the saving may be much smaller than the demo suggested. Also track adoption. A tool that saves time for the four people who use it and is ignored by the other forty is not yet a success.
A pilot plan you can run in about eight weeks
- Weeks 1–2: Pick one use case with clear volume and a clear owner. Record the baseline. Collect a test set of real examples, 50–200 of them, with the correct answers.
- Weeks 2–3: Check the data. Confirm you can access the documents or records, clean up obvious problems, and get legal and security sign-off on the data flow.
- Weeks 3–5: Build the simplest working version. Start with prompts, add retrieval if needed, and test against your example set. Track accuracy on those examples every time you change something.
- Weeks 5–7: Put it in front of a small group of real users in their normal workflow, with review built in. Collect feedback and measure the same numbers as the baseline.
- Week 8: Decide. Compare results with the baseline and the running cost, and choose to scale, adjust or stop. Stopping is a valid result, and much cheaper than finding out a year later.
Keep the pilot narrow. A single process measured properly will teach you more than a company-wide chatbot that everyone tries once. Once the first use case is working and measured, the second one goes much faster, because the data access, security review and evaluation approach are already in place.
Where to start this quarter
Make a list of the text-heavy tasks your team repeats every week. Rank them by volume and by how easy it is to tell a right answer from a wrong one. The top of that list is almost always a better first project than whatever looked best in a vendor demo.
If you want help picking the use case or running the pilot, we build these systems on Azure OpenAI and other major model providers, connected to the data you already have. Email sales@tag4tech.com or use our contact page.