# OpenAI Blocks Sites From ChatGPT Search via Aura Check

Website operators and technical SEO teams have spent the past two years configuring `robots.txt` directives and web application firewall exceptions to welcome OpenAI's official web crawlers. By explicitly whitelisting `GPTBot`, `OAI-SearchBot`, and `ChatGPT-User` across Cloudflare and origin server rules, publishers assumed their pages were guaranteed to remain accessible whenever a user or autonomous agent requested live information inside ChatGPT.

A newly documented pre-flight security layer inside OpenAI's browsing infrastructure proves that assumption wrong. Before ChatGPT live search or Codex Browser Use ever initiates an outbound HTTP connection to a target domain, OpenAI executes an internal policy evaluation against an undocumented backend endpoint called `Aura`. If that pre-flight check flags a URL or domain as restricted, the browsing agent aborts the connection inside OpenAI's own infrastructure before a single packet reaches the publisher's server.

Because the block occurs upstream of standard web crawling, affected websites see zero rejected requests, zero 403 status codes, and zero crawler hits in their origin access logs. Even more frustrating for enterprise IT administrators, an explicit block issued by the `Aura` service overrides local workspace allowlists and caches the denial locally for a full 24 hours, offering site owners no public lookup tool or appeal workflow.

For publishers, SaaS documentation teams, and e-commerce brands that rely on AI-assisted discovery, this silent pre-flight gate introduces a major blind spot in technical observability. Understanding how OpenAI separates crawler access from agentic browsing policy—and why conversational prompt context can accidentally trigger a 24-hour domain lockout—is now essential for diagnosing sudden drops in ChatGPT referral traffic.

## Fast Facts

- **Undocumented Pre-Flight Endpoint:** Autonomous browsing agents query `chatgpt.com/backend-api/aura/site_status` before attempting to fetch any external URL.
- **Mandatory 24-Hour Local Cache:** Once a domain or URL receives a block verdict from the `Aura` endpoint, the denial is cached locally for **24 hours**, preventing retry attempts even if the underlying trigger was temporary.
- **Three Public Crawlers Bypassed:** Standard allow rules in `robots.txt` for `GPTBot` (model training), `OAI-SearchBot` (search indexing), and `ChatGPT-User` (user-initiated fetches) never execute when `Aura` halts the request pre-flight.
- **Admin Allowlist Override:** While transient network timeouts to `aura/site_status` fail open (defaulting to allow access), an explicit block verdict strictly overrides custom domain allowlists configured by enterprise workspace administrators.
- **Prompt Context Contamination Risk:** Requests sent to the `Aura` status API transmit conversation context alongside the target URL, meaning sensitive or adversarial terms in a user's prior chat history can inadvertently trigger a site-wide browsing block against legitimate mainstream publishers.

## Why `robots.txt` and Server Logs Cannot Detect `Aura` Blocks

For decades, search engine diagnostics followed a predictable handshake. A bot identified itself via a `User-Agent` string and published IP range, checked `robots.txt`, requested the target URL, and recorded an HTTP response code in the publisher's CDN and origin server logs. If Googlebot or Bingbot stopped indexing a section of a site, an SEO engineer could inspect Cloudflare logs or Search Console crawl stats to pinpoint the exact firewall rule, DNS error, or 5xx timeout responsible.

OpenAI's agentic browsing architecture splits that workflow into two decoupled layers. The first layer consists of the documented crawlers—`GPTBot`, `OAI-SearchBot`, and `ChatGPT-User`—which respect `robots.txt` and publish verifiable IP ranges. The second layer is the agent safety harness governing live interactive browsing inside ChatGPT and Codex Browser Use.

| Diagnostic Layer | Standard Crawler (`OAI-SearchBot` / `GPTBot`) | User Fetch (`ChatGPT-User`) | Pre-Flight Policy Gate (`Aura` API) |
|---|---|---|---|
| **Execution Point** | Asynchronous background crawl | Live HTTP fetch after safety pass | Upstream check at `backend-api/aura/site_status` |
| **Honors `robots.txt`** | Yes (strictly respects directives) | Yes (checks before page retrieval) | Runs **before** any `robots.txt` lookup occurs |
| **Origin Log Footprint** | Visible IP & `User-Agent` in CDN logs | Visible IP & `User-Agent` in CDN logs | **Zero packets sent** to publisher CDN or origin |
| **Failure / Cache Behavior** | Retries on standard crawl schedule | Returns live HTTP status to session | Caches explicit block locally for **24 hours** |

When a user asks ChatGPT to summarize a product page, verify a pricing table, or inspect a documentation URL, the agent first submits the target link to `chatgpt.com/backend-api/aura/site_status`. If `Aura` returns a negative status, the agent immediately tells the user it cannot access the website. A site administrator who checks their server logs five minutes later will see no incoming request from an OpenAI IP address and may mistakenly conclude that their server is healthy or that the user mistyped the URL.

## How Prompt History Can Trigger False-Positive Site Lockouts

OpenAI's public help documentation notes that autonomous agents are restricted from visiting domains associated with illegal activity, malware, or compliance violations via a global blocklist. Every enterprise security team agrees that stopping coding and browsing agents from visiting command-and-control servers or phishing pages is mandatory.

The operational problem for legitimate businesses lies in *how* the `Aura` API evaluates requests. Network inspections show that calls to `chatgpt.com/backend-api/aura/site_status` do not merely pass a clean domain string in isolation; they include contextual metadata from the active chat session.

Consider what happens when a cybersecurity analyst, journalist, or software engineer uses ChatGPT or Codex to investigate a vulnerability, analyze a phishing incident, or debug a regex filter containing exploit strings—and then asks the agent in the same conversation thread to fetch a reference article from a mainstream technology publisher or SaaS documentation portal. Because the prompt history contains high-risk security keywords alongside the outbound URL check, the `Aura` classifier can flag the request as unsafe and cache a 24-hour block against that domain for the session or workspace.

Even more concerning for engineering teams using Codex Browser Use inside corporate environments is the hierarchy of authority. OpenAI engineers have confirmed that if the `Aura` API experiences a network outage or timeout, the client defaults to allowing access so developer workflows do not grind to a halt. However, when `Aura` returns an explicit block decision, that verdict is absolute: it overrides custom domain allowlists that enterprise administrators have explicitly configured for their own internal or partner web properties.

## Practical Implications for Publishers, DevOps, and SEO Teams

Because there is no "OpenAI Webmaster Tools" console where domain owners can inspect their `Aura` status or request a manual review, technical teams must shift from passive log monitoring to active synthetic testing. Waiting for customers to complain that ChatGPT claims your pricing page or developer docs are "inaccessible" means losing high-intent buyers at the exact moment they are comparing vendors.

### The 4-Step Enterprise Implementation Playbook

1. **Separate Clean-Session Synthetic Probes From Context-Heavy Chats:** When testing whether ChatGPT or Codex can access your domain, always run the check inside a brand-new, zero-history chat session using a neutral prompt (`"Fetch and summarize the main heading on https://example.com/pricing"`). If a clean session succeeds while an existing thread fails, your domain is not globally blocklisted—the prior thread triggered a context-sensitive `Aura` block and cached it for 24 hours.
2. **Correlate `ChatGPT-User` Log Drops With Referral Analytics:** Configure a daily log alert on your CDN (Cloudflare, Akamai, or Fastly) that tracks the ratio of `OAI-SearchBot` background crawls to `ChatGPT-User` live fetches. If `OAI-SearchBot` continues crawling normally while `ChatGPT-User` live requests suddenly drop to zero on specific URLs, investigate whether content on those pages (such as embedded security payloads, unfiltered user comments, or ad scripts) is tripping pre-flight safety heuristics.
3. **Host Clean Markdown Twins for Documentation and Pricing Pages:** Autonomous browsing agents and safety classifiers frequently choke on heavy client-side JavaScript bundles, third-party ad-tech trackers, and obfuscated scripts that resemble malware loaders. Publish clean `/llms.txt` indexes and lightweight Markdown mirrors (`/docs/page.md`) so agents can ingest structured text without triggering browser-safety false positives.
4. **Isolate Security and Red-Team Prompts in Dedicated Developer Workspaces:** Instruct engineering and security teams using Codex Browser Use never to mix exploit analysis, malware logs, or adversarial prompt testing in the same session or environment where they run automated browser tests against staging or production company domains, preventing accidental 24-hour local `Aura` lockouts.

## Next Steps