AI crawler and referral tracking


AI crawler and referral tracking brings three evidence sources together: robots.txt access checks, Cloudflare bot requests and GA4-attributed visits. Review each source separately to understand access, crawling and website traffic.

AI crawler and referral tracking brings three evidence sources together: robots.txt access checks, Cloudflare bot requests and GA4-attributed visits. Review each source separately to understand access, crawling and website traffic.

AI crawler and referral tracking brings three evidence sources together: robots.txt access checks, Cloudflare bot requests and GA4-attributed visits. Review each source separately to understand access, crawling and website traffic.

Trial dates at checkout. Payment details required.

Trial dates at checkout. Payment details required.

Separate paths for AI crawler requests and visitors referred from AI answers to a website.
Separate paths for AI crawler requests and visitors referred from AI answers to a website.
Separate paths for AI crawler requests and visitors referred from AI answers to a website.

Measure AI crawler activity and referrals

Agent Analytics

Connect crawler activity, referral data, and AI agent behavior


AI Crawler Insights

See whether important AI crawlers can access your site

Cloudflare AI Bot Tracking

Monitor how AI bots interact with your website

Illustrative walkthrough: review robots.txt rules, check OpenAI SearchBot readiness and identify technical crawl access issues. Sample data.

Illustrative walkthrough: review Cloudflare AI bot activity and filter by user agent, path, time range and request status. Sample data.

Measure AI crawler activity and referrals

Agent Analytics

Connect crawler activity, referral data, and AI agent behavior


AI Crawler Insights

See whether important AI crawlers can access your site

Cloudflare AI Bot Tracking

Monitor how AI bots interact with your website

Illustrative walkthrough: review robots.txt rules, check OpenAI SearchBot readiness and identify technical crawl access issues. Sample data.

Illustrative walkthrough: review Cloudflare AI bot activity and filter by user agent, path, time range and request status. Sample data.

Measure AI crawler activity and referrals

Agent Analytics

Connect crawler activity, referral data, and AI agent behavior


AI Crawler Insights

See whether important AI crawlers can access your site

Illustrative walkthrough: review robots.txt rules, check OpenAI SearchBot readiness and identify technical crawl access issues. Sample data.

Cloudflare AI Bot Tracking

Monitor how AI bots interact with your website

Illustrative walkthrough: review Cloudflare AI bot activity and filter by user agent, path, time range and request status. Sample data.

GA4 AI Referrals

Measure traffic coming from AI platforms

Illustrative walkthrough: review sessions, active users, engaged sessions and page views from AI referral sources. Sample data.

Connect GA4 to track sessions, active users, page views, events and available conversions from AI referral sources.

Choose the evidence that answers your question

Agent Analytics helps technical and marketing teams review access, crawler activity and attributed visits. Start with the question you need to resolve: can a crawler reach the page, what requests were recorded, or what visitors did after arriving through an AI referral?

Prepare the relevant data source

Identify the website and pages you want to investigate. For Cloudflare and GA4 reporting, use the appropriate zone or property and confirm you have permission to connect it. Check the available date range before comparing results across sources.

Review each layer

  1. Use crawler insights to inspect relevant robots.txt rules and technical access findings.

  2. Review Cloudflare bot activity by the available agent, path, time and request-status information.

  3. Review GA4-attributed AI referrals and available engagement or conversion data.

Example: deciding who acts next

Illustrative diagnostic examples with fictional data.

Evidence

Access check

Example finding

A search crawler is disallowed on a product path.

Next action

Ask the site owner to review the relevant rule.

Evidence

Cloudflare activity

Example finding

Requests to a priority page are blocked.

Next action

Inspect request status and the applicable firewall setting.

Evidence

GA4 referrals

Example finding

Sessions are attributed to perplexity.ai.

Next action

Review engagement and events; investigate landing pages in GA4.

Use these records to assign the right task: a technical owner can investigate an access rule, while a marketer can review attributed engagement and investigate landing pages in GA4. Review bot requests and GA4 sessions as separate measures, keeping source, date range and missing data visible.

For Generative Engine Optimisation reporting, combine these checks with saved answer observations where relevant. Follow the bot-log troubleshooting guide for access investigations and the AI referral attribution guide for visitor measurement.

Use the ChatGPT traffic and leads guide for Cardiff businesses to connect referral analysis with measurable website actions.

Bring an access or referral question and the data sources available to your team.

FAQ

Frequently Asked Questions

1. What is Agent Analytics?

2. What are AI crawler insights?

3. Why does AI visibility matter?

4. What is Cloudflare AI bot tracking?

5. Can NeuraCite show traffic from AI platforms?

6. Which AI platforms or bots can be monitored?

7. Who should use Agent Analytics?