User-Agent Categories

Most client requests to Netlify’s edge servers contain a User-Agent header. This header is very valuable when analyzing and acting on traffic, but its values vary widely.

To help with this, Netlify categorizes all User-Agent header values into a taxonomy of categories that you can preview in the table below. In some cases, a sub-category is also provided.

This enables you to analyze and act on incoming requests using broad, meaningful categories instead of implementing your own logic for handling raw User-Agent values.

Why the User-Agent header is useful

Although the User-Agent header is optional and clients can set it to any string, it remains highly useful. Despite historical quirks and inconsistencies, it helps identify traffic patterns and tailor responses for different types of clients.

For example:

• Block or rate limit specific client types.
• Learn what URLs are frequently accessed by AI agents answering end-user questions.
• Investigate requests resulting in 404 errors: are they from human visitors, crawlers, or other automated bots?
• Understand traffic spikes and error patterns, which are often correlated with a specific client type.
• Return a pre-rendered page for client types that do not execute JavaScript.

All of these tasks are easier when relying on User-Agent categories instead of raw values.

Using categories on Netlify

The category (and sub-category, if any) is exposed in the request header Netlify-Agent-Category. You can use this header with any Netlify feature that supports header-based conditions:

1. Access the header via the request parameter passed to Netlify Functions and Edge Functions.
2. Configure an Edge Function to run only if the header matches specific values.
3. Add a rate limiting rule to throttle access to specific resources based on the header value (Enterprise-only feature).

The value of the header is:

<category>[;<optional sub-category>]

For example: ai-agent;user, tooling;monitoring, or other.

There is always one category, and either zero or one sub-category.

See the full mapping below for available values.

Usage with Edge Functions

Here is an example Edge Function invoked only if Netlify associates the incoming request’s raw User-Agent string with either the ai-agent or crawler category:

JavaScript

// Only runs if User-Agent category contains "ai-agent" or "crawler"
export default async (req) => {
  const userAgent = req.headers.get('user-agent');
  console.log(`The User-Agent of this AI agent/crawler is: ${userAgent}`);
};

export const config = {
  path: "/blog/*",
  header: { "netlify-agent-category": "(ai-agent|crawler)" }
};

TypeScript

import type { Config } from "@netlify/edge-functions";

// Only runs if User-Agent category contains "ai-agent" or "crawler"
export default async (req: Request) => {
  const userAgent = req.headers.get('user-agent');
  console.log(`The User-Agent of this AI agent/crawler is: ${userAgent}`);
};

export const config: Config = {
  path: "/blog/*",
  header: { "netlify-agent-category": "(ai-agent|crawler)" }
};

In this example:

1. The function is configured to run only when netlify-agent-category matches one of the specified values.
2. When the function runs, it accesses the raw User-Agent header and logs it.

Optimize Edge Functions with matching

Prefer using header matching over using conditional logic within a function’s code, so that Edge Functions will only run when necessary.

Although Edge Functions are fast and lightweight, invoking a function always adds some latency.

Testing header values

Here is how to test functionality that is dependent on the User-Agent category:

Testing deployed projects

The Netlify-Agent-Category header is only set when your project is deployed and served by Netlify. If a client provides this header when making a request, Netlify will override it.

To test different categories, set the User-Agent header appropriately (rather than attempting to set Netlify-Agent-Category directly).

You can set the User-Agent header easily using browser developer tools, common browser extensions, or command-line HTTP utilities.

Testing locally

When running your site locally with netlify dev or our Vite plugin, Netlify-Agent-Category is neither set nor overridden.

To test a Netlify Function or Edge Function that uses this header, set it directly to any value you want.

Available categories

Netlify organizes User-Agent request headers into the following categories to help you make sense of where requests to your site are coming from.

Category	Request header value	Description
Browser	browser	Mostly human visitors using common browsers.
AI Agent	ai-agent	AI chat services (e.g., ChatGPT) and other AI tools providing on-demand answers to questions. Typically not high-volume.
Page Preview	page-preview	Social networks, chat and misc. tools fetching links for in-app preview.
Crawler	crawler	Bulk indexing and scraping of content by self-declared bots (search engines, AI services, feed generators, etc.).
Tooling	tooling	Monitoring tools, custom scripts and various other automated programs.
Other	other	User-Agent request header was provided, but uncategorized. Mostly niche tools.
Not Provided	none	No User-Agent header provided. Usually bots.

Note: some vendors use multiple User-Agents, which may fall into different categories. For example, OpenAI uses one User-Agent for high-volume crawling and another for ad-hoc browsing by ChatGPT.

Browser

Value in request header: browser

This category is most correlated with normal browsing by human visitors across all device types (desktop, mobile, tablet).

Some automated tools report browser-like User-Agents. These may be legitimate testing or monitoring tools, or bots attempting to masquerade as browsers.

When this category is matched:

A User-Agent string that doesn’t match any other category and can be parsed as a known browser pattern is categorized as browser.

Sub-categories: none.

Is browser traffic always legitimate?

Netlify does not blindly trust a given User-Agent, and neither should you.

However, in most sites that receive constant user traffic - the vast majority of client requests reporting a browser User-Agent are legitimate clients.

AI Agent

Value in request header: ai-agent

AI services that browse the web ad-hoc to answer a specific user prompt or query.

For example, when a user asks ChatGPT “What are the newest attractions in New York City?”, the model may perform a web search and then fetch several pages for context.

Traffic in this category is usually low-volume and initiated directly by a human user’s request. It is generally not advisable to block or rate-limit these requests, as they indicate your content is being surfaced as relevant by AI systems.

This category does not include bulk crawlers operated by AI providers. High-volume crawlers behave very differently from ad-hoc browsing and are categorized separately under the Crawler category.

When this category is matched:

Based on known substrings associated with AI agent User-Agents. Examples include: ChatGPT-User/, Claude-User/.

Note that high-volume crawlers such as GPTBot/ are not matched here and fall under the Crawler category instead.

Sub-categories:

• user - The AI agent is browsing to answer an end-user’s prompt. This is the common case.
• search - The AI agent is performing web retrieval as part of a built-in dedicated search feature.

Page Preview

Value in request header: page-preview

When a URL is shared in a social network, messaging app, or embedding tool, that platform typically fetches the page once or a few times to extract metadata (OpenGraph, Twitter Card data, etc.) for the preview.

These ad-hoc preview fetches are distinct from high-volume crawling performed by the same companies.

When this category is matched:

Matching is based on well-known User-Agents used specifically for link preview fetching — not general crawling. Examples include: Slackbot, WhatsApp, Instagram, Discord, Google Web Preview.

Sub-categories:

• general - All User-Agents in this category fall under this sub-category.

Crawler

Value in request header: crawler

Bulk crawlers that scan your site for indexing or large-scale data collection.

Some crawling is beneficial (e.g., search engines keeping your content fresh in their index). Others, especially high-volume AI crawlers, may cause spikes that impact performance. Some crawlers are stable and well-behaved (like Googlebot and Bingbot), while others have inconsistent or aggressive behavior.

Your blocking decisions should depend on the crawler’s usefulness to your site, and its request volume.

When this category is matched:

Based on known substrings identifying a wide variety of crawlers across different purposes.

Sub-categories:

• general - Search-engine crawlers and other crawlers with generic patterns such as “crawl”, “spider”, or “bot”.
• ai - High-volume AI provider crawlers (e.g., from OpenAI, Anthropic, Perplexity). Note that some vendors (notably Google) do not use a separate User-Agent when crawling for AI purposes.
• social - Crawlers from social platforms (e.g., Meta services, Pinterest).
• seo - Crawlers from SEO analytics vendors. If you use an SEO service, you may see requests from the same vendor (but with a different User-Agent) appear in the Tooling category, used by that vendor to audit your site.

Tooling

Value in request header: tooling

A broad set of automated tools used for monitoring, testing, integrations, CI pipelines, headless browsing, custom scripts, and more. It also includes some third-party bots and scrapers that are not well-identified.

Some entries are explicit (e.g., monitoring vendors), while others are based on generic indicators (e.g., any User-Agent with the substring bot or api) unless they match a more specific category such as Crawler.

When this category is matched:

Netlify maintains curated lists of known substrings for each sub-category. Additionally, many generic User-Agent patterns fall here by default unless they match a higher-priority category.

Sub-categories:

• monitoring — Monitoring services such as Datadog, Pingdom, and other non-explicitly listed User-Agents having a generic substring like health or monitor. High request rates often indicate overly aggressive polling that you may have configured by mistake.
• netlify-service - Requests generated by Netlify’s internal services (typically low-volume).
• prerender - Headless browsers used by third-party prerendering services.
• custom-code - Default User-Agents from common programming languages, HTTP libraries, and CLI tools (e.g., curl, wget, python, node, go-http-client). These typically originate from either custom scripts you wrote or bots built by others.
• general - Tools and self-declared bots that do not match any other sub-category, including unmapped User-Agents containing substrings such as bot or api.

Other

Value in request header: other

User-Agents that do not map to any other category and are not known, common browsers.

The share of requests to your site falling in this category should be very low (under 1-2%).

Sub-categories: none.

Not Provided

Value in request header: none

Requests without a User-Agent header.

Though the User-Agent header is technically optional, common tools and client libraries typically add a default header. Thus, having no User-Agent provided may well signal an undesired bot.

Sub-categories: none.