LinkosBot, and how to say no to it.

LinkosBot builds the Linkos backlink index that Theylink runs on. It reads public HTML pages and stores which page links to which, the anchor text, rel attributes and when we last saw the link. It doesn’t render JavaScript, doesn’t fetch images or media, and never logs in.

• User agentLinkosBot/1.0 (+https://linkos.bio/bot)
• Politenessrobots.txt honoured, Crawl-Delay honoured, at most 2 concurrent requests and 1 request per second per host by default
• Rechecksweekly for every indexed page; daily for pages linking to tracked competitors
• Dead linksa link is called dead only after two failing checks at least 24 hours apart

Slow it down or stop it

Two lines in your robots.txt are enough. We pick them up within 48 hours and stop.

User-agent: LinkosBot
Disallow: /

To slow it instead: Crawl-delay: 10 under the same user agent.

Take a site off Theylink

robots.txt stops the crawl; a verified request also removes the site from every public page. Remove a domain takes a minute. To remove one email address from editor contacts, use contact opt-out.

Questions about the crawler: [email protected]