XunLMove cross-border work forward
中文

Website & SEO

robots.txt Checker

Read a website's robots.txt, HTTP status, basic rules and declared Sitemap addresses.

◈ Online Lookup · Public Information Only

Inputs & Settings

Target domain or URL will be submitted to the server for querying; do not enter links containing secret keys.

Results workbench

Fill in the details, then click Start Lookup.

↗

Results will be displayed here

Review details, copy or download, and then continue your work.

YOUR NEXT STEP

Turn a website check into a monitoring routine

Create a company account to manage sites and notifications. Pro supports 5 sites and 5-minute availability monitoring, with deep diagnostics available according to your plan and permissions.

Create company accountExplore ProContact Pan Chenxing

New companies receive a 14-day Pro trial, with no card required. Trials exclude the paid monthly credit grant.

Already have an account? Open your workspace →

How to use robots.txt Checker

  1. Enter Website URL and adjust the options as needed.
  2. Run the tool and inspect its results, evidence and limitations.
  3. Copy or download the result, then verify it against the actual website, source data or business records.

Try a concrete example

Enter https://www.xunl.net to inspect the file at the site root.

How to interpret results

robots.txt describes crawler preferences, not access control. A missing or malformed file needs interpretation alongside server responses; this basic view does not prove indexing.

Frequently asked questions

Where should robots.txt be placed?

Place in the root directory /robots.txt under the same protocol, host, and port; manage each subdomain separately.

Can Disallow prevent indexing?

Not guaranteed. Disabling indexing usually uses crawlable noindex; sensitive resources should use access control.

Can it verify whether rules allow a specific URL?

This tool reads and interprets files; complex wildcards, precedence, and specific crawler support should be verified against official rules.

robots.txt checker practical guide and common questions

What does a robots.txt checker do?

robots.txt publishes path rules for cooperating crawlers. Its scope depends on protocol, host and matching rules; it is not a password protecting private material.

Who is this for, and when is it useful?

Site owners changing crawler rules during launches or migrations.

How do I use this robots.txt checker?

Submit the site, inspect robots retrieval, applicable rules and Sitemap declarations; crawler restrictions are not access control.

How should I interpret the results?

Confirm reliable retrieval, then inspect the intended crawler and path. Disallow, noindex and authentication solve different problems; keep sensitive information behind access controls.

Where is the work performed, and which services receive data?

Our server requests robots.txt at the relevant protocol and host root and parses basic rules. It does not write rules to your website.