LinkosBot, and how to say no to it.

LinkosBot builds the Linkos backlink index that Theylink runs on. It reads public HTML pages and stores which page links to which, the anchor text, rel attributes and when we last saw the link. It doesn’t render JavaScript, doesn’t fetch images or media, and never logs in.

• User agentLinkosBot/1.0 (+https://linkos.bio/bot)
• What it never fetchesadult, gambling and known-spam hosts are refused at any depth and never followed; a page with a clearly adult title contributes nothing. LinkosBot reads HTML text only, never images or video.
• Politenessrobots.txt honoured, Crawl-Delay honoured, at most 2 concurrent requests and one request every 2 seconds per host by default; a 429 or 503 backs us off for minutes to hours, and a host that answers 403 five times is left alone for a week
• Rechecksweekly for every indexed page; daily for pages linking to tracked competitors
• Dead linksa link is called dead only after two failing checks at least 24 hours apart

Slow it down or stop it

Two lines in your robots.txt are enough. We pick them up within 48 hours and stop.

User-agent: LinkosBot
Disallow: /

To slow it instead: Crawl-delay: 10 under the same user agent.

Take a site off Theylink

robots.txt stops the crawl; a verified request also removes the site from every public page. Remove a domain takes a minute. To remove one email address from editor contacts, use contact opt-out.

Questions about the crawler: use the contact page