When updating Baidu’s robots.txt, repeatedly clicking “Detect and Update” may cause intermittent update failures—sometimes it works, sometimes it doesn’t. As a result, content that should not be indexed (and is prohibited in robots.txt) gets indexed and then deleted, which is quite common. What’s the issue here? It’s not due to excessive server load, but because the firewall incorrectly blacklists some Baiduspider IPs.
We won’t go into regular servers—most people know that those in Beijing, Shanghai, and Guangzhou are generally reliable. However, there are some special servers that the vast majority of webmasters may not be aware of. For example, Western Digital’s “Hong Kong/Taiwan Servers” are quite interesting: are they really located in Hong Kong or Taiwan? The data centers are actually in mainland China, so how can they be called Hong Kong/Taiwan servers? They use Hong Kong/Taiwan IPs to avoid ICP filing, while all data remains in mainland China.
What’s the problem with this? You’ll find that the website’s server uses CDN—even a single image you upload will return a “302 status code.” While this improves access speed, is it beneficial for SEO?
Large-scale websites typically use CDN acceleration, but some sites apply CDN not only to “devices” but also to spiders. What’s the final result? If the CDN nodes are unstable, this will be a fatal issue for website spiders.
Many large sites enable CDN to prevent attacks, and the consequences of not configuring “spider origin-pull” are predictable. Does your website use CDN? Log in to the Baidu Webmaster Platform to check if spiders can crawl the real IP address!
