1.1 What Is PetalBot
PetalBot is an automatic program of the Petal search engine. The function of PetalBot is to access both PC and mobile websites and establish an index database which enables users to search the content of your site in Petal search engine. You can identify crawling from Petal by analyzing the User-agent field.
1.2 What Is the User-Agent of PetalBot
|Product name||User-Agent||Full User-Agent|
|PetalBot(mobile)||PetalBot||Mozilla/5.0 (Linux; Android 7.0;)
AppleWebKit/537.36 (KHTML, like Gecko)
Mobile Safari/537.36 (compatible; PetalBot;+https://aspiegel.com/petalbot)
1.3 How to Block PetalBot from Visiting Your Site
PetalBot complies with the Internet robots protocol. You can use the robots.txt file to completely prevent PetalBot from accessing your website, or to prevent PetalBot from accessing some files on your website.
Note: Banning PetalBot from accessing your site will make the pages on your site and all search engine services provided by Petal unsearchable in the Petal search engine.
Robots setup recommendations
You can set different crawling rules according to different user-agents of each product, and you can directly prevent the crawling of PetalBot. The following robots can prevent Petal crawling or conditional allow:
1.4 How Much Pressure Does PetalBot Bring to a Web Server
In order to have a better retrieval results for the target resources, PetalBot needs to maintain a certain amount of crawling of your website. We try not to bring an unreasonable burden on the website, and we will make adjustments based on comprehensive factors such as server capacity, website quality, and website updates. If there is any unreasonable behavior of PetalBot’s visiting, please send your concerns to firstname.lastname@example.org.
1.5 How to Judge PetalBot Crawling
You can verify if a web crawler accessing your server really is PetalBot.
To verify PetalBot as the caller:
1. Run a reverse DNS lookup on the accessing IP address from your logs, using the host command.
2. Verify that the domain name is in aspiegel.com.
3. Run a forward DNS lookup on the domain name retrieved in step 1 using the host command on the retrieved domain name. Verify that it is the same as the original accessing IP address from your logs.
|$ host 18.104.22.168
22.214.171.124.in-addr.arpa domain name pointer petalbot-114-119-160-240.aspiegel.com
$ host petalbot-114-119-160-240.aspiegel.com
petalbot-114-119-160-240.aspiegel.com has address 126.96.36.199
1.6 Why My Website Can Still Be Found in Petal When It Has Been Already Added to robots.txt
It takes time to update data in the search engine index database. Although PetalBot has stopped accessing the pages on your site, it may take several months to clear the page index information that has already been established in the Petal search engine database. Besides, please check if your robots configuration is correct.
If your rejection is very urgent, please make your request to email@example.com.