查看robots.txt

User-agent: *

Crawl-delay: 10 无论使用哪种用户代理， 都应该在两次下载请求之间给出10秒的抓取延

User-agent: Googlebot

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Googlebot-Image

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Baiduspider-news

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Baiduspider

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Baiduspider-image

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Sosospider

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: bingbot

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: 360Spider

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: HaosouSpider

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: yisouspider

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: YoudaoBot

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Sogou Orion spider

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Sogou News Spider

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Sogou blog

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Sogou spider2

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Sogou inst spider

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Sogou web spider

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: EasouSpider

Request-rate: 1/2 # load 1 page per 2 seconds

Crawl-delay: 10

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: MSNBot

Request-rate: 1/2 # load 1 page per 2 seconds

Crawl-delay: 10

Disallow: /login

Disallow: /logout

Disallow: /resetpassword

Disallow: /terms

Disallow: /search

Disallow: /notifications

Disallow: /settings

Disallow: /inbox

Disallow: /admin_inbox

Disallow: /?guide

User-agent: Bytespider

Disallow: /

User-Agent: *

Disallow: /

————————————————

版权声明：本文为CSDN博主「郝曈曈」的原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接及本声明。

原文链接：https://blog.csdn.net/HaoTongTongSEO/article/details/101425438