RekAiBot

RekAiBot is the web crawler operated by Rek.ai AB, Sweden. It reads the content of websites that use Rek.ai so that we can deliver relevant recommendations and search results to their visitors.

If you found this page from your server logs, the short version is this: we only crawl websites that belong to our customers, we visit at most one page per second, and you can stop us at any time through robots.txt or by contacting us.

Why is RekAiBot visiting my site?

RekAiBot only crawls websites owned by organisations that have an agreement with Rek.ai. It does not crawl the open web and it does not follow links to external sites.

If your organisation uses Rek.ai, the crawler is there because you or a colleague set it up. It reads your pages so that our recommendation and search features can suggest the right content to your visitors.

If your organisation does not use Rek.ai and you still see RekAiBot in your logs, something is wrong. Please contact us and we will look into it right away.

How to identify RekAiBot

Requests from RekAiBot carry this user agent string:

Mozilla/5.0 (compatible; RekAiBot/1.0; +https://rek.ai/bot)

All requests come from a fixed set of IP addresses:

91.106.198.7
91.123.202.104

Anyone can spoof a user agent string. If you want to be sure a request is from us, check the source IP against the list above.

How RekAiBot crawls

Rate. RekAiBot makes at most one request per second to a given website.

Content. The crawler fetches HTML pages and follows internal links on the site it has been set up for.

Frequency. How often a page is revisited depends on the setup for each customer site.

robots.txt. RekAiBot follows robots.txt rules that are addressed to it by name, using the token RekAiBot. It does not follow rules in the wildcard group User-agent: *. Many of our customers use the wildcard group to keep pages out of public search engines while still wanting those pages to appear in their own site search and recommendations. Addressing RekAiBot by name lets you control our crawler without changing how your site appears anywhere else.

How to control or stop RekAiBot

To block RekAiBot from your whole site, add this to your robots.txt:

User-agent: RekAiBot
Disallow: /

To keep RekAiBot out of a specific section:

User-agent: RekAiBot
Disallow: /internal/

Changes to robots.txt are picked up the next time we visit your site. If you need crawling stopped sooner, or if robots.txt is not an option for you, email us at bot@rek.ai and we will stop within one business day.

You can also block the IP addresses listed above in your firewall. Keep in mind that blocking RekAiBot in any of these ways will stop recommendations and search from updating on your site.

What we do with the content

Content fetched by RekAiBot is used only to power recommendations and search on the website it was collected from. We do not sell it, share it with third parties, or use it to train general purpose AI models.

For details on how we store and protect data, see our Security and Privacy Policy pages.

Contact

Questions or concerns about RekAiBot? Email bot@rek.ai.

If you are reporting unexpected traffic, please include the domain, the time of the requests, the source IP address and the user agent string you saw. That helps us find the cause quickly.