Why a careful tool still gets things wrong
The New Zealand public service guidance on generative AI uses the OECD definition: a hallucination is output that is incorrect but convincing7. That second word is the problem. Claude does not flag a guess with a different tone of voice. A made-up clause number, a misread total or a court decision that never existed arrives in the same polished paragraph as everything else.
Anthropic is direct about this. Its own documentation lists ways to reduce hallucinations, and then warns that those techniques do not remove them entirely, so critical information still needs to be validated, especially for high-stakes decisions1. The National Cyber Security Centre's guide for small businesses puts hallucinations alongside prompt injection as a reliability risk, describing answers that sound correct but are not true9.
In practice the errors cluster in predictable places. Watch numbers, dates, names, quotations, references to legislation, and any claim about what a document says when Claude has not been given that document. Rewording your own email is low risk. Asking Claude what the law requires, from memory, is where the trouble starts.
What New Zealand law and regulators expect
If an answer involves personal information, the Privacy Act 2020 applies regardless of which tool produced it. Information privacy principle 8 says an agency must not use or disclose personal information without taking reasonable steps to make sure it is accurate, up to date, complete, relevant and not misleading6. A Claude summary of a staff member's file, or a draft letter about a customer's account, is exactly the kind of output that principle covers.
The Office of the Privacy Commissioner spells out what it expects of an organisation using generative AI. Its list includes procedures for checking accuracy before information is used or disclosed, and human review of outputs before anyone acts on them5. The Commissioner also notes that these tools often produce confident errors of fact or logic, and can repeat bias5.
The courts have gone further for one profession. The judiciary's guidelines for lawyers say practitioners are responsible for the accuracy of everything they put before a court or tribunal, and must check any chatbot output, including citations, first8. The same guidelines warn that chatbots can invent cases, quotes and legislation that look as if they came from a real source8. Even if you never go near a courtroom, that warning describes the failure you are guarding against.
MBIE's responsible AI guidance for businesses is voluntary, but it frames the same idea as good practice: be clear about why you use AI, build it on sound business foundations, and manage risks across the life of the system10. A checking routine is one of the cheapest risk controls you can put in place.
Sort the work by risk before deciding how hard to check
Not every output deserves the same scrutiny. Checking a reworded paragraph line by line wastes time, and waving through a tax calculation is reckless. Give your team three simple tiers and a rule for each.
- Low risk: Claude reshapes text you wrote or already approved, such as tidying an internal email or turning bullet notes into a paragraph. The author reads it once before sending.
- Medium risk: Claude summarises or extracts from a document you supplied, such as meeting minutes, a supplier contract or a tender. The author checks every number, name and date against the source.
- High risk: anything about law, tax, health, safety, money owed, or a named person, and anything going to a client or regulator. A second person who knows the subject checks it, and the source of every factual claim is recorded.
- Off limits without a source: legal or regulatory statements produced from Claude's general knowledge with no document or cited web page behind them.
Set Claude up so mistakes are easier to see
The best check starts before the prompt. Most hallucinations happen when Claude fills a gap from its general training rather than from your material, so the first job is to give it the material and tell it to stay inside it.
Use a Project for any repeated task. In claude.ai, open Projects, choose New Project, then use Set project instructions to describe how Claude should behave in every chat inside it4. Upload the documents it should rely on into the project knowledge, and Claude uses them as context in each conversation4. On Team and Enterprise plans you can share a project with colleagues on view or edit permissions, so everyone works from the same source files4.
Anthropic's documentation recommends a handful of habits that translate directly into project instructions1. Give Claude explicit permission to say it does not know. For long documents, ask it to pull out word-for-word quotes first and base its answer only on those quotes. Ask it to cite a supporting quote for each claim, and to withdraw any claim it cannot support. Tell it to use only the documents provided, not its general knowledge.
When the answer genuinely needs current information from the web, turn on web search: select the plus button at the bottom left of the chat window and choose Web search2. If you have the new Claude experience there is no toggle, because Claude searches the web when it helps2. Responses then carry direct citations and source links you can open2. On Team and Enterprise plans an Owner must first switch this on for the organisation under Organization settings, then Capabilities2. For a bigger question, the Research option on paid plans runs a chain of searches and returns a report with citations, and it needs web search enabled to work3.
- Add this line to your project instructions: If the documents do not answer the question, say so plainly instead of guessing.
- Add this line too: Quote the passage you relied on, with its heading or page, after each factual statement.
- Keep superseded versions of policies and price lists out of the project knowledge, so Claude cannot quote an old figure.
A five-step check before anything leaves the business
This routine works for any medium or high risk output. Print it, pin it in the team's shared Project instructions, and teach it in the first week of any rollout.
- Step 1. Trace every claim. For each number, name, date and quotation, find the line in the source document or the cited web page. If you cannot find it, delete it.
- Step 2. Open the links. A citation only helps if someone opens it and confirms the page says what Claude claims it says.
- Step 3. Ask Claude to audit itself. In a fresh message, ask it to list every factual claim in its draft with the supporting quote. Gaps in that list are where errors hide1.
- Step 4. Rerun the question. Anthropic suggests running the same prompt more than once and comparing the results, because inconsistent answers can signal a hallucination1.
- Step 5. Sign it off. The person who sends the work owns it. Record who checked high risk output and what they checked it against.
Worked example: an accounting practice in Hamilton
Picture an eight-person Hamilton accounting practice that uses Claude to prepare year-end summary letters for small business clients. The first attempt goes wrong in a quiet way: Claude rounds a depreciation figure and states a filing deadline from memory. Neither error is dramatic, and both would have gone to a client.
The fix takes an afternoon. The practice manager builds a Project for year-end letters, uploads the firm's letter template and engagement terms, and writes instructions that say Claude must take every figure from the client's uploaded trial balance and quote the line it used. Deadlines are removed from Claude's job entirely and taken from the firm's own compliance calendar. The reviewing accountant now checks each figure against the quoted line, which is more targeted than rereading the whole letter cold.
Because the letters contain client personal information, the practice also writes down its accuracy step, which is the kind of procedure the Privacy Commissioner expects to see5. If a client later questions a letter, the firm can show what was checked and by whom.
Worked example: a Christchurch builder reading consent conditions
Picture a Christchurch residential builder with a small office team using Claude to turn long building consent documents into site checklists for foremen. Here the risk is an omission, not an invention: a missed inspection hold point costs days.
The office manager asks Claude to extract every condition as a direct quote before writing anything, which follows Anthropic's advice for long documents1. The checklist is then built only from those quotes, each tagged with its page number. The site manager ticks each quote against the consent before the checklist is issued. Anything Claude marks as unclear goes to the council, not to a guess.
Make checking a team habit, not a heroic effort
Checking fails when it depends on one careful person. Write two sentences into your AI use policy: who checks which tier of work, and that unchecked high risk output does not leave the business. Keep a shared log of errors Claude made and how they were caught. After a month the log tells you which tasks need tighter instructions and which are safe to speed up.
Keep client and staff personal information out of personal Claude accounts. If someone pastes a customer file into the wrong place and serious harm is likely, the Privacy Act's breach rules apply, and the Commissioner expects notification as soon as practicable11. The public service guidance sums up the right attitude for any business: users should understand the data they give these tools, and understand, check and agree with what comes back7.