My Local LLM Kept Inventing Naan — Building a Reliable Food-Delivery Agent with Spring AI, MCP & Ollama

A food-delivery agent built with Spring AI, MCP and Ollama, and the guardrails a 7B model needed before I'd trust it with an order

I asked my food-delivery agent for one butter chicken and two naan. It answered with a confident little menu: naan from Spice Route at Rs.120, total Rs.618, shall I place the order?

There is no naan on Spice Route's menu. There is no naan on any menu. The price was made up, the total was made up, and the model was very sure of both.

That moment is the reason this post exists. I set out to build a small agent with Spring AI, the Model Context Protocol (MCP) and a model running locally through Ollama. The plumbing worked quickly. What took the time was everything the model did after the plumbing worked: announcing orders it never placed, reusing old order IDs, and telling me it had cancelled things it hadn't touched.

This is the story of the build, and of the guardrails that made it something I'd actually let take an order.

The finished app. The chat is on the left; the order card on the right is live and can be cancelled while the order is still being prepared.

What I built

A chat UI (Angular) talks to a Spring Boot backend. The backend hands the conversation to a local model, qwen2.5-coder:7b, and the model decides which tools to call: search restaurants, look up a dish, place a Cash on Delivery order, cancel it. The tools live in a separate MCP server, also Spring Boot, which the backend discovers when it starts.

It is a proof of concept, so the restaurant data is mocked: four restaurants, 17 dishes, and an order that "delivers" itself over about 70 seconds. There is no real Zomato or Swiggy behind it.

Why not connect to the real thing? Both gate their APIs behind partner onboarding. Swiggy has published MCP servers, which is exactly the kind of integration this project imitates, but they are meant for consumer AI clients such as Claude Desktop or ChatGPT. Their own page says third-party app development is not permitted, and it places real cash-on-delivery orders with no sandbox. For a learning project, a mock server was the right call, and it made me build the same shape of integration: an agent talking to someone else's tools over MCP.

Why MCP, and what Spring AI gives you

MCP has three kinds of things a server can expose, and Spring AI has an annotation for each. I ended up using all three.

Tools are actions the model can call. This is placeOrder, trimmed:

@McpTool(name = "placeOrder",
         description = "Place a Cash on Delivery order. Only call this once the customer "
                     + "has agreed to the dishes AND given a delivery address.")
public String placeOrder(
        @McpToolParam(description = "Restaurant name", required = true) String restaurantName,
        @McpToolParam(description = "Dishes with optional quantities, e.g. '2 Paneer Tikka, Dal Makhani'", required = true) String items,
        @McpToolParam(description = "Delivery address exactly as the customer gave it", required = true) String address) {
    // validate, total it up, save the order, return a plain-text confirmation
}

Resources are readable state. The order card in the UI gets its live data from one, so the MCP server needs no REST controllers at all:

@McpResource(uri = "order://{orderId}", name = "Order",
             description = "A placed order's items, total, address and live status",
             mimeType = "application/json")
public McpSchema.ReadResourceResult getOrder(String orderId) { /* ... */ }

Prompts are reusable templates. I added one, recommend_meal, mostly to see how it feels to consume one from the client side.

On the backend, connecting to the MCP server is configuration, and handing its tools to the model is one line:

spring.ai.mcp.client.streamable-http.connections.food-delivery.url=${mcp-server.base-url}
spring.ai.mcp.client.streamable-http.connections.food-delivery.endpoint=/mcp

ChatClient.builder(chatModel)
        .defaultSystem(SYSTEM_PROMPT)
        .defaultToolCallbacks(mcpToolCallbackProvider.getToolCallbacks())
        .build();

The MCP client fetches the tool list from the server at startup. Add a tool to the server, restart, and the model can use it. That part is genuinely nice.

Then the model started making things up

The plumbing came together quickly. Then I started using it, and a 7B model on a 4 GB laptop GPU showed me all the ways it could be confidently wrong. Here is the honest list.

1. It printed its tool calls instead of making them

Sometimes the model didn't call a tool at all. It wrote the call as plain text:

{"name": "searchDishes", "arguments": {"vegetarian": true, "maxPrice": 250}}

Spring AI only reacts to real, structured tool calls, so that text went straight to the customer as if it were the answer, and no tool ever ran. Other times the model skipped the tool and improvised an answer with no real data behind it.

The fix was a small class that looks for this pattern in the reply, runs the intended tool itself, and returns the real result. It then needed two follow-ups:

• MCP tool results are wrapped in a content-block list ([{"text":"..."}]), and the model sometimes echoed that raw wrapper back. Running the fix over its own output needed a second unwrap step, and my first version chained the two steps as alternatives instead of a pipeline, so the second never ran on the result of the first. Debug logging found that one; staring at the code hadn't.
• Asked to do two things at once, the model leaked two calls on separate lines. Parsing the reply as one JSON document silently failed, so the cleanup now works line by line.

2. It announced orders it never placed

This was the one that made me stop trusting replies. The model would say, in a perfectly formatted message, "Order placed! Order ID is ORD1001." No order existed. It had never called placeOrder. It had guessed the next ID. On another run it reused the ID of an order I had already cancelled.

I tightened the prompt more than once and it kept happening, because the model isn't disobeying an instruction; it is producing text that looks like the answer. So the check moved out of the model.

3. It invented dishes and prices

That is the naan story. Even with the prompt saying "look dishes up first", the model produced dishes and prices that weren't in the data, and it did so the same way on every run, because I had set the temperature to 0.

4. It said "cancelled" without cancelling

My favourite: "Order cancelled. Reason: [Reason returned by cancelOrder function]." That is a template placeholder, printed as if it were a real message. The order was still confirmed.

The fix: don't trust the reply, check the claim

I ended up with three layers.

1. Prompt rules (the system prompt). "Look up named dishes first", "never claim an order without calling placeOrder".
2. Tool-side validation (the MCP server). placeOrder refuses with no address or an unknown dish, and a cancel is refused once the order is out for delivery.
3. Reply guards (the backend). Four small classes, below.

The prompt helps, but it is the weakest layer. The tool-side checks are worth more than they look, because a tool that says "these dishes are not on this menu" is an honest signal the backend can also use. The reply guards are where the real safety comes from, and I put them in their own package so they are easy to find:

public interface Guardrail {
    String apply(GuardrailContext context, String reply);
}

Each guard checks one kind of claim against the real system:

• ToolCallLeakGuard repairs tool calls printed as text.
• OrderClaimGuard requires any order ID in a reply to be one the customer mentioned, or one created during this turn. Otherwise it asks the model once more with a direct instruction, and if that fails, it tells the customer nothing was ordered, using the tool's real reason.
• DishClaimGuard checks every dish and price against the real menu, which the MCP server exposes as a resource. A wrong claim is replaced with a plain correction that lists what is available.
• CancelGuard verifies that a requested cancellation actually happened, and calls the cancel tool itself if it didn't.

OrderClaimGuard, stripped down, is the whole idea in one place:

@Override
public String apply(GuardrailContext context, String reply) {
    if (!claimsUnplacedOrder(reply, context)) {
        return reply;
    }

    String retried = context.askAgainWithNote().apply(RETRY_NOTE);
    return claimsUnplacedOrder(retried, context) ? couldNotPlaceMessage() : retried;
}

A small chain runs them in order, and ChatController shrank to two jobs: call the model, then pass the reply through the chain.

Now the same request gets a straight answer. Asking for a dish that doesn't exist:

I can't offer 'Naan': not on any restaurant's menu.

(And once I actually added naan to the mock menu, it ordered fine: one butter chicken and two naan, Rs.449, cash on delivery.)

One honest caveat. These guards are heuristics over text. They catch invented dishes, prices, order IDs and cancellations. They do not catch every possible false sentence a model can produce, and I would not describe them as a guarantee.

Smaller lessons that cost me time

Spring Boot 4 uses Jackson 3, and the package changed. It is tools.jackson.*, not com.fasterxml.jackson.*. I had a Docker build failing quietly because I piped Maven's output through tail, which hid the non-zero exit code, and the old image kept running. Check exit codes, not just output.

Documentation summaries can be wrong. A summary of the docs gave me a package name for the MCP annotations that did not exist. What worked was opening the actual jar and reading the class signatures with javap. It took two minutes and saved an hour.

Java strings and regex. "\bcancel" in a Java string is a backspace character followed by "cancel", not a word boundary. It compiled, silently matched nothing, and the cancel guard did nothing. It needs "\\bcancel".

Containers hold on to old addresses. nginx resolves a hostname once and keeps the IP, so recreating the backend container gave 502s until I made it re-resolve through Docker's DNS. Separately, the Spring MCP client connects once at startup and does not reconnect, so restarting the MCP server alone breaks the backend until you restart that too.

Simplify the flow to help the model. I started with a shopping cart: add items, view cart, remove, place order. Every extra step was another chance for a 7B model to slip. I replaced it with one call: agree the dishes, give an address, placeOrder with everything in it, cash on delivery. That removed a whole class of two-step slips, though the model still needed the guards.

What I'd do differently

• Write the tests for the guards first. They are the part that most needs proof, and they don't need a slow model to test, because they are functions of a reply string. This project has no automated tests, and the guards are where I'd add them.
• Expect to verify anything that matters. If a model reply results in money moving, a booking, or a deletion, check the outcome in the system that did it, not in the wording of the reply.
• Treat a bigger model as the real fix. A 4 GB GPU makes larger local models impractical. A stronger model would need fewer guards. The guards are what let a small local model be useful in the meantime, but they are a workaround, not a substitute.

Try it

The code is here: github.com/sajeevkr2/food-delivery-agent-poc. It runs with docker compose up --build plus Ollama on the host; the README covers setup and has the full architecture diagram.

If you build something similar, the one habit I'd keep is small: after a model tells you it did something, go and look.