Crawler

HuutBot

HuutBot is the program Huut uses to read a website, so the support assistant set up for that site can answer visitors' questions from what the site actually says. It identifies itself in every request as:

HuutBot/1.0 (+https://huut.si/bot)

How it behaves

• It follows your robots.txt. Allow and Disallow rules, wildcards, and Crawl-delay are respected.
• It respects page-level signals. Pages marked noindex are skipped, and links marked nofollow are not followed.
• It stays on one site. It reads only the website it was pointed at, never other domains it links to.
• It is gentle. It makes a couple of requests at a time with a pause between them, and slows down further if your robots.txt asks it to.
• It only reads. It doesn't log in, fill in or submit forms, add anything to a cart, or change anything on your site.
• It refuses private networks. It will not fetch internal, local or cloud-metadata addresses.

Why it visited your site

Someone set up Huut for your website (usually you, or your team). Huut reads a whole site only after the person setting it up has proved they control it, by adding a code to the site's DNS, home page or a file on the site. Without that proof, Huut reads nothing beyond a single look at the home page.

Block it

Add this to your robots.txt. HuutBot will stop reading your site on its next visit:

User-agent: HuutBot
Disallow: /

Questions or removal

If you think your site was added without permission, or you want what Huut learned from it removed, email hello@huut.si with the website address and we'll remove it.