ALMbot, the Keply crawler
Who it is, what it reads, how it behaves and how to block it.
What it does
ALMbot builds a catalog of public websites: what a site is, where it is, what it is about. It helps answer people's questions by sending them to the original sources, with a link.
What it reads, and what it does not
ALMbot reads from each page only what it needs to index the site on Almagesto, the Keply search engine: the title, the headings, a description (the one the page declares or, if missing, its first paragraph), the images and the media with their addresses. It does not take the whole content and does not republish it. It does not use it to train models.
How it behaves
- It obeys robots.txt and Crawl-delay.
- One request at a time per site, with a pause between requests.
- It backs off when a site asks it to wait (429, 503) and comes back later.
- It respects a site's text-and-data-mining reservation (EU Directive 2019/790, art. 4).
How to recognise it
It identifies itself with this user-agent:
ALMbot/1.0 (+https://keply.it/en/almbot.html)
How to block it
Add to your robots.txt:
User-agent: ALMbot Disallow: /
ALMbot re-reads robots.txt before every round: the change applies from the next visit.
Contact
For a question or a report about ALMbot: almbot@keply.it