收录各大搜索引擎与商业爬虫的 User-Agent、抓取行为、渲染能力与来路 IP 数据,为站长提供「识别 - 分析 - 拦截」全流程参考。取名源自镇守门户的瑞兽,为网站守住爬虫访问的大门。
| User-Agent | 爬虫名称 | 类型 | 抓取频率 | 抓取深度 | 遵守 Robots | 是否活跃 |
|---|---|---|---|---|---|---|
| Cursor | Cursor | AI数据 | Unclear at this time. | — | 无视 | 停用 |
| Datenbank Crawler | Datenbank | AI数据 | Unclear at this time. | — | 无视 | 停用 |
| DeepSeekBot | DeepSeek | AI数据 | Unclear at this time. | — | 无视 | 停用 |
| Devin | Devin AI | AI数据 | Unclear at this time. | — | 遵守 | 停用 |
| Diffbot-User | Diffbot-User | AI数据 | Only when prompted by a user. | — | 遵守 | 停用 |
| Dot | Dot | AI数据 | Unclear at this time. | — | 无视 | 停用 |
| DoubaoBot | ByteDance | AI数据 | Unclear at this time. | — | 无视 | 停用 |
| Echobot Bot | Echobox | AI数据 | Unclear at this time. | — | 无视 | 停用 |
| EchoboxBot | EchoboxBot | AI数据 | Unclear at this time. | — | 无视 | 停用 |
| ElevenlabsBot | ElevenlabsBot | AI数据 | Unclear at this time. | — | 无视 | 停用 |
| ERNIEBot | ERNIEBot | AI数据 | Unclear at this time. | — | 无视 | 停用 |
| ExaUser | ExaUser | AI数据 | Unclear at this time. | — | 无视 | 停用 |
| Factset_spyderbot | Factset_spyderbot | AI数据 | No information provided. | — | 无视 | 停用 |
| FirecrawlAgent | Firecrawl that extracts web content and converts it into str | AI数据 | No information provided. | — | 遵守 | 停用 |
| FriendlyCrawler | Unknown | AI数据 | Unclear at this time. | — | 遵守 | 停用 |
| GeistHaus-PageFetcher | GeistHaus | AI数据 | Unclear at this time. | — | 无视 | 停用 |
| Gemini-Deep-Research | Gemini-Deep-Research | AI数据 | Unclear at this time. | — | 无视 | 停用 |
| Google-Agent | Google-Agent | AI数据 | Unclear at this time. | — | 遵守 | 停用 |
| Google-CloudVertexBot | Google-CloudVertexBot | AI数据 | No information. | — | 遵守 | 停用 |
| Google-Firebase | Google-Firebase | AI数据 | Unclear at this time. | — | 无视 | 停用 |
| Google-Gemini-CLI | Google-Gemini-CLI | AI数据 | Unclear at this time. | — | 无视 | 停用 |
| Google-NotebookLM | Google-NotebookLM | AI数据 | Unclear at this time. | — | 无视 | 停用 |
| GoogleAgent-Mariner | GoogleAgent-Mariner | AI数据 | Unclear at this time. | — | 无视 | 停用 |
| GoogleAgent-URLContext | GoogleAgent-URLContext | AI数据 | Unclear at this time. | — | 无视 | 停用 |
| GoogleOther-Image | GoogleOther-Image | AI数据 | No information. | — | 遵守 | 停用 |
| GoogleOther-Video | GoogleOther-Video | AI数据 | No information. | — | 遵守 | 停用 |
| Grok Bot | Grok Bot | AI数据 | Unclear at this time. | — | 无视 | 停用 |
| HarkBot | HarkBot | AI数据 | Unclear at this time. | — | 无视 | 停用 |
| HenkBot | HenkBot | AI数据 | Unclear at this time. | — | 无视 | 停用 |
| Hunyuan | Hunyuan | AI数据 | Unclear at this time. | — | 无视 | 停用 |