What is AlphaBot?

AlphaBot is Ad-Bull’s web crawler and indexer. It discovers publicly available pages and stores lightweight search information so that users can find relevant content via our search.

What we generally store

• Page URL and host
• Page title and a short summary/snippet
• Primary page text used for search (trimmed)
• Language (if detectable)
• Timestamps such as first seen / last updated
• Optionally: references to visible images on the page (URL and alt text)

We aim to index publicly accessible content. We don’t index pages that clearly instruct crawlers not to.

Quick Infos

• Home
• Add your URL

It can take some days to index your site!

How to opt out (robots.txt)

AlphaBot follows robots.txt. Examples:

Place the file at /robots.txt - Changes are respected on the next visit.

Page-level controls (meta robots)

On individual pages you can signal indexing preferences:

Or target AlphaBot specifically:

The noarchive directive can ask crawlers not to keep cached copies/snippets.

Content concerns

If you’d like a URL to be removed or corrected:

• Use robots.txt or meta robots as shown above, or
• Contact us with the exact URL and a brief note.

Legitimate requests are handled promptly.

General behavior

• We respect standard webmaster signals (robots.txt, meta robots, canonical links, HTTP status).
• We prefer publicly accessible pages and sitemaps where available.
• We aim to be considerate and avoid unnecessary load on sites.