We trained our own AI model for one small job.

21 October 2026

The short answer

We took a small open model, trained it on 345 real small business websites, and scored it on 59 more that we had checked by hand. Trained, it never once told anyone to fix a problem the check had not found. The same model with careful instructions and an example made up 71. It runs live on our AI page, where you can try it on any website.

The job

Our free website check looks at a home page and reports on speed, accessibility, privacy, security and the domain. A report full of findings is useful to a developer and baffling to most business owners. What they want to know is what to fix first, why it matters to them, who does it and how big a job it is.

So the model has one job: read the report, pick the fixes that matter most, put them in order and explain each one plainly. It is not allowed to know anything the check did not find. A plan that invents a problem is worse than no plan.

Why not just ask a big AI tool?

We tried the obvious route first: the same small model, with no training, given clear instructions, the list of fixes it could choose from and a worked example. It wrote fluent, confident plans, and in 42 of the 59 test sites it added at least one fix for a problem the check never found. It also reworded fix titles 55 times, which makes plans hard to check.

Bigger models follow instructions better, but they cost more for every answer. For a narrow job that runs thousands of times, a small model that has learned the job is cheaper, faster and easier to keep honest.

How we built it

The training data. We ran the check on 444 real small business home pages: shops, cafes, clinics, trades and booking sites near 25 UK towns, found from public map data. We kept only the reports, never the pages themselves, and no personal details.

The answers. For each report we wrote the plan it should produce. A rules script drafts them, because it knows the fix for every finding the check can make. We then checked 60 of them by hand, line by line. Those 60 (59 after dropping one that was not a small business) became the test set the model never saw.

The training. We did not retrain the whole model. We trained a small add on, called an adapter, that adjusts how the model answers. The method is called LoRA. The adapter is a few megabytes, it trained on one of our own computers in about 70 minutes, and the cost was electricity.

Running it. The adapter sits on top of the base model on a cloud AI service, next to the rest of our website. The plan streams onto the page as it is written, so you can watch it work.

What it scored

On the 59 sites checked by hand, against the same model with careful instructions:

• Fixes for problems the check never found: none, against 71.
• The right fixes picked: 98%, against 48%.
• The same order as the plans checked by hand: 81%, against 3%.
• Plans with a wrong number in them: none, against 3%.

The live version runs on the full size base model rather than the compressed one we trained on. On a spot check of 10 sites it still made up nothing and picked 96% of the right fixes. It got the exact order on 6 of the 10, usually by swapping two small items.

What it cannot do

The plans we checked by hand were the rules drafts, and we left them unchanged because they were right. So the model learned our rules, and it does that very reliably. It has not yet learned anything a rules script could not do.

That is still worth having. It proves the whole pipe works: data, training, scoring, running it live and learning from use. And it is a base to grow from, toward plans that answer questions in free text and handle findings the rules have never seen.

How it keeps learning

Each plan ends with "Was this plan right?". When someone runs a check, the model keeps a copy of the report and the plan to learn from, with the website's address and name removed. Anyone can untick the box to opt out. When about 100 new plans have been rated, we train again. The new adapter only replaces the live one if it scores at least as well on the same 59 test sites.

What this means for your business

Most useful AI in a small business is not a chatbot. It is a narrow, repeated job: sorting enquiries, drafting replies from your own price list, checking forms. A small model trained on your own examples can often do that job more reliably than a big general one, and the data stays where you can account for it. The hard part is not the training. It is deciding exactly what a right answer looks like, and scoring against it honestly.

If you have a job like that, our AI work starts with working out whether it suits AI at all.

Where this comes from

• The scores: our own test of 59 websites, 20 October 2026. The method and the figures are on our AI page.
• The base model: Mistral 7B Instruct v0.2, released under the Apache 2.0 licence.
• The training method: LoRA: Low-Rank Adaptation of Large Language Models, Hu and others, 2021.
• Training on our own computer: MLX LM.
• Running the adapter: Workers AI LoRA adapters.
• Finding the websites: OpenStreetMap, map data under the Open Database Licence.