Block various marketing crawlers: Amazonbot, MJ12bot

Today we checked the network logs and found that there are a large number of crawlers, such as: VoilaBot, EtaoSpider, SemrushBot, Amazonbot, etc., especially Amazonbot, which crawls web pages endlessly. They do not provide any search services. We read the instructions on the Internet: ( KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)", they are only providing Alexa improved services for searches on our website It has no effect. Continuous crawling is a waste of resources. Therefore, in order to make it easier to observe network logs, we decided to block it. Many of the following crawlers can be blocked. At present, there is no benefit. They are all marketing. I summarized it, Can be copied directly to robots.txt for use：

User-agent: VoilaBot
Disallow: /

User-agent: EtaoSpider
Disallow: /


User-agent: MJ12bot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: rogerbot
Disallow:/

User-agent: dotbot
Disallow:/

User-agent: AhrefsBot
Disallow: /

User-agent: Alexibot
Disallow: /

User-agent: SurveyBot
Disallow: /

User-agent: Xenus
Disallow: /

User-agent: Xenus Link Sleuth 1.1c
Disallow: /

User-agent: similar-web
Disallow: /

User-agent: Amazonbot
Disallow: /