PetalBot

1.1 What Is PetalBot

PetalBot is an automatic program of the Petal search engine. The function of PetalBot is to access both PC and mobile websites and establish an index database which enables users to search the content of your site in Petal search engine. You can identify crawling from Petal by analyzing the User-agent field.

1.2 What Is the User-Agent of PetalBot

Product name User-Agent Full User-Agent
PetalBot(PC) PetalBot Mozilla/5.0 (compatible;PetalBot;+https://aspiegel.com/petalbot)
PetalBot(mobile) PetalBot Mozilla/5.0 (Linux; Android 7.0;)
AppleWebKit/537.36 (KHTML, like Gecko)
Mobile Safari/537.36 (compatible; PetalBot;+https://aspiegel.com/petalbot)

1.3 How to Block PetalBot from Visiting Your Site

PetalBot complies with the Internet robots protocol. You can use the robots.txt file to completely prevent PetalBot from accessing your website, or to prevent PetalBot from accessing some files on your website.

Note: Banning PetalBot from accessing your site will make the pages on your site and all search engine services provided by Petal unsearchable in the Petal search engine.

Robots setup recommendations

You can set different crawling rules according to different user-agents of each product, and you can directly prevent the crawling of PetalBot. The following robots can prevent Petal crawling or conditional allow:

User-agent: PetalBot
Disallow: /
User-agent: PetalBot
Allow: /w/api/
Disallow: /trap/

1.4 How Much Pressure Does PetalBot Bring to a Web Server

In order to have a better retrieval results for the target resources, PetalBot needs to maintain a certain amount of crawling of your website. We try not to bring an unreasonable burden on the website, and we will make adjustments based on comprehensive factors such as server capacity, website quality, and website updates. If there is any unreasonable behavior of PetalBot’s visiting, please send your concerns to search@aspiegel.com.

1.5 How to Judge PetalBot Crawling

Given that there may be many fake PetalBot crawling, it is recommended that you apply reverse DNS lookup to check if the crawling source IP address belongs to PetalBot. The verification methods for different platforms may vary. The following three methods are for Linux, Windows and iOS.

On the Linux platform, you can use the host ip command to reverse the IP address and check if it is from PetalBot. PetalBot’s hostname is in the format petalbot.*.aspiegel.com, and it is impersonated if it is not petalbot.*.aspiegel.com.

$ host 114.119.167.220
220.167.119.114.in-addr.arpa domain name pointer petalbot-114-119-167-220.aspiegel.com

On the Windows platform or the IBM OS/2 platform, you can use the nslookup ip command to reverse the IP address and check if it is from PetalBot. Open the command processor and enter nslookup xxx.xxx.xxx.xxx (IP address) to resolve the IP address to check if it is PetalBot crawling. PetalBot’s hostname is in the format petalbot.*.aspiegel.com, and it is impersonated if it is not petalbot.*.aspiegel.com.

On the Mac OS platform, you can use the dig command to reverse the IP address and check if it is from PetalBot. Open the command processor and enter dig xxx.xxx (IP address) to resolve the IP address to check if it is PetalBot crawling. PetalBot’s hostname is in the format petalbot.*.aspiegel.com, and it is impersonated if it is not petalbot.*.aspiegel.com.

1.6 Why My Website Can Still Be Found in Petal When It Has Been Already Added to robots.txt

It takes time to update data in the search engine index database. Although PetalBot has stopped accessing the pages on your site, it may take several months to clear the page index information that has already been established in the Petal search engine database. Besides, please check if your robots configuration is correct.

If your rejection is very urgent, please make your request to search@aspiegel.com.