各大搜索引擎蜘蛛IP段 以及怎么样判断蜘蛛真假
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><span style="color: black;">搜索引擎蜘蛛会<span style="color: black;">定时</span>来抓取网站内容，以便收录网页，站长们是非常欢迎蜘蛛来爬的，<span style="color: black;">然则</span>却有人经常伪造蜘蛛IP进行恶意采集内容，还有<span style="color: black;">非常多</span>DDOS攻击<span style="color: black;">亦</span>会伪<span style="color: black;">导致</span>蜘蛛的User Agent来攻击，我查看过一位站长<span style="color: black;">伴侣</span>被DDOS攻击的日志，里面大概20%的国外IP伪<span style="color: black;">导致</span>Googlebot来<span style="color: black;">拜访</span>，<span style="color: black;">这般</span>会对<span style="color: black;">咱们</span>的网站<span style="color: black;">导致</span>影响。<span style="color: black;">咱们</span>必须要揪出这些假蜘蛛，拒绝<span style="color: black;">她们</span><span style="color: black;">拜访</span>网页，<span style="color: black;">此时</span>候<span style="color: black;">咱们</span>只能<span style="color: black;">经过</span>IP来识别了，<span style="color: black;">由于</span>User-agent是<span style="color: black;">能够</span>伪造的，<span style="color: black;">仅有</span>IP<span style="color: black;">才可</span>准确判断。</span></p>
<div style="color: black; text-align: left; margin-bottom: 10px;"><img src="https://p3-sign.toutiaoimg.com/pgc-image/0327cd4b4593489cbe528df64b5205cc~noop.image?_iz=58558&from=article.pc_detail&lk3s=953192f4&x-expires=1725101053&x-signature=g1HfPYMRXo0VcR1EU9DtGS2g%2FpI%3D" style="width: 50%; margin-bottom: 20px;"></div>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><span style="color: black;">下面是站<span style="color: black;">成长</span>本营<span style="color: black;">近期</span>收集的各大搜索引擎蜘蛛IP数据，数据没办法<span style="color: black;">保准</span>完全准确，<span style="color: black;">由于</span>搜索引擎官方是不公开IP段的，且IP都会变，我对比<span style="color: black;">太多</span>个数据和自己核验，下面的数据还是挺<span style="color: black;">靠谱</span>的，有遗漏的下面还会有<span style="color: black;">处理</span><span style="color: black;">方法</span>。</span></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><span style="color: black;">百度蜘蛛IP段：</span></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">61.135.0.0/16</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">123.125.0.0/16</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">111.206.0.0/16</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">180.76.0.0/20</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">180.149.128.0/19</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">220.181.0.0/16</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">36.110.128.0/17</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">124.164.0.0/14</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">116.179.0.0/16</p>180.97.0.0/18
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><span style="color: black;">Google中国蜘蛛IP段：</span></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">203.208.32.0/19</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">203.208.60.0/24</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">74.125.0.0/16</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">72.14.192.0/18</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">64.233.160.0/19</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">66.249.64.0/19</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">66.102.0.0/20</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">64.68.64.0/19</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">66.249.90.0/24</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">66.249.91.0/24</p>66.249.92.0/24
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><span style="color: black;">搜狗蜘蛛IP段：</span></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">118.184.176.0/22</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">123.180.0.0/14</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">123.125.0.0/16</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">61.135.0.0/16</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">123.126.64.0/18</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">111.202.0.0/16</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">36.110.128.0/19</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">220.181.0.0/16</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">106.120.128.0/17</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">49.7.0.0/18</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">218.30.96.0/19</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">106.38.0.0/16</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">111.13.0.0/16</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">58.250.0.0/16</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">183.36.96.0/19</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">49.7.64.0/18</p>必应蜘蛛IP段：
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">103.25.156.0/24</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">111.221.16.0/20</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">157.56.0.0/16</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">199.30.16.0/20</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">65.52.0.0/14</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">13.64.0.0/11</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">157.55.0.0/16</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">207.46.0.0/16</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">40.64.0.0/10</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">52.224.0.0/11</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">40.76.0.0/14</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">40.80.0.0/12</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">52.160.0.0/11</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><span style="color: black;">360搜索蜘蛛IP段：（这是官方<span style="color: black;">供给</span>的，非常准确）</span></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">180.153.0.0/16</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">180.163.128.0/17</p>42.224.0.0/12
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><span style="color: black;">头条搜索蜘蛛IP段：</span></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">110.249.201.0/24</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">110.249.202.0/24</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">111.225.148.0/24</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">111.225.149.0/24</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">220.243.135.0/24</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">220.243.136.0/24</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">111.224.0.0/14</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">60.8.0.0/15</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">110.240.0.0/12</p>220.243.128.0/18
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><span style="color: black;">神马搜索蜘蛛IP段：</span></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">42.156.128.0/17</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">42.120.128.0/17</p>106.11.144.0/20
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><span style="color: black;">以上IP段可能<span style="color: black;">将来</span>还会新增，<span style="color: black;">况且</span>新增的IP段<span style="color: black;">咱们</span><span style="color: black;">亦</span><span style="color: black;">没法</span>获取到，<span style="color: black;">此时</span>候<span style="color: black;">咱们</span>就需要<span style="color: black;">另一</span>一种<span style="color: black;">方法</span>来核实了，那<span style="color: black;">便是</span>DNS反查，<span style="color: black;">咱们</span><span style="color: black;">能够</span><span style="color: black;">经过</span>反查来访的蜘蛛IP，查到对应的域名<span style="color: black;">位置</span><span style="color: black;">就可</span>核实。</span></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><span style="color: black;"><span style="color: black;">经过</span>DNS反查IP用的是一种代码指令，<span style="color: black;">区别</span>的操作平台<span style="color: black;">运用</span>的指令<span style="color: black;">亦</span>是<span style="color: black;">区别</span>的，<span style="color: black;">例如</span>：linux、windows、Mac OS这三个操作平台，<span style="color: black;">运用</span>的指令为host、nslookup、dig，<span style="color: black;">查找</span>的格式为：host ip（ip=ip<span style="color: black;">位置</span>）、nslookupip（ip=ip<span style="color: black;">位置</span>）、digip（ip=ip<span style="color: black;">位置</span>）。例如：host 111.222.XXX.XXX 或 nslookup 111.222.XXX.XXX 或 dig host 111.222.XXX.XXX，如果是用PHP<span style="color: black;">能够</span><span style="color: black;">运用</span> gethostbyaddr("IP")来获取，<span style="color: black;">区别</span>的蜘蛛IP对应的域名是不<span style="color: black;">同样</span>的，<span style="color: black;">咱们</span>只要核实是哪家的蜘蛛就OK了。</span></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><span style="color: black;">下面我把各大搜索引擎蜘蛛的域名分享下：</span></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">百度蜘蛛IP对应的域名（例子）：</p>baiduspider-116-179-32-216.crawl.baidu.com
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">谷歌蜘蛛IP对应的域名（例子）：</p>crawl-203-208-60-60.googlebot.com
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">必应蜘蛛IP对应的域名（例子）：</p>msnbot-40-77-167-2.search.msn.com
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">搜狗蜘蛛IP对应的域名（例子）：</p>sogouspider-123-126-113-79.crawl.sogou.com
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">头条蜘蛛IP对应的域名（例子）：</p>bytespider-111-225-149-71.crawl.bytedance.com
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">神马蜘蛛IP对应的域名（例子）：</p>shenmaspider-42-156-137-55.crawl.sm.cn
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><span style="color: black;">360搜索官方说是禁止反查的，<span style="color: black;">因此</span><span style="color: black;">咱们</span>只用官方给出的IP段判断就OK了。</span></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><span style="color: black;"><span style="color: black;">经过</span>以上IP反查<span style="color: black;">能够</span>得到域名，判断域名<span style="color: black;">是不是</span>是这些域名结尾就<span style="color: black;">能够</span>核实，<span style="color: black;">这般</span><span style="color: black;">咱们</span>在防御上就不会将搜索引擎蜘蛛误杀了。</span></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">以后还会继续分享有价值的站长<span style="color: black;">关联</span>资源，感谢您的浏览！</p>
你的见解真是独到，让我受益匪浅。 大势所趋，用于讽刺一些制作目的就是为了跟风玩梗，博取眼球的作品。 顶楼主，说得太好了！
页:
[1]