查看robots.txt

User-agent: *

Crawl-delay: 10 无论使用哪种用户代理, 都应该在两次下载请求之间给出10秒的抓取延

User-agent: Googlebot

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Googlebot-Image

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Baiduspider-news

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Baiduspider

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Baiduspider-image

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Sosospider

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: bingbot

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: 360Spider

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: HaosouSpider

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: yisouspider

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: YoudaoBot

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Sogou Orion spider

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Sogou News Spider

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Sogou blog

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Sogou spider2

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Sogou inst spider

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Sogou web spider

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: EasouSpider

Request-rate: 1/2 # load 1 page per 2 seconds

Crawl-delay: 10

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: MSNBot

Request-rate: 1/2 # load 1 page per 2 seconds

Crawl-delay: 10

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Bytespider

Disallow: /

User-Agent: *

Disallow: /

————————————————

版权声明:本文为CSDN博主「郝曈曈」的原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接及本声明。

原文链接:https://blog.csdn.net/HaoTongTongSEO/article/details/101425438

追加问题
    6 人参与回答
hello
hello · 公众号:囧少

学习

行星减速机 · 行星减速机品牌厂家

这谁能有心思看啊

y

留下足印

搜外用户
搜外用户 · 来搜外学习的一名普通用户

学习,Robots