> ## Documentation Index
> Fetch the complete documentation index at: https://tyk.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Managing Webcrawlers

> Configuring your Live Portal to restrit webcrawlers and bots

## Configure robots.txt (Built-in Solution)

From version 1.14.0, the Tyk Developer Portal includes built-in support for customizing the `robots.txt` file, which is the standard way to instruct search engines and other well-behaved crawlers about which parts of your site they should not access.

To configure this:

1. Log in to the Admin Portal
2. Navigate to **Settings > General**
3. Scroll down to the **robots.txt Settings** section
4. Edit the content to control crawler access

A restrictive `robots.txt` configuration would look like:

```
User-agent: *
Disallow: /
```

This instructs all crawlers to avoid indexing any part of your site. By default, the Portal already uses a restrictive `robots.txt` configuration.

## Implement Additional HTTP Headers

You can add custom response headers to further discourage crawling:

* `X-Robots-Tag: noindex, nofollow` - Similar to robots.txt but as an HTTP header
* `Cache-Control: no-store, no-cache, must-revalidate` - Prevents caching

These can be added in your proxy configuration or by customizing your portal theme.

## Best Practices

* Regularly check your server logs for unusual crawling patterns
* Consider using a CAPTCHA for registration forms to prevent automated sign-ups (not supported natively by Tyk Developer Portal at this time)
* Use JavaScript-based content rendering for sensitive information, as basic crawlers may not execute JavaScript

Remember that while these methods can deter most crawlers, they cannot provide absolute protection against determined scrapers that deliberately ignore `robots.txt` rules or use sophisticated techniques to mimic human behavior.