This website was made with the help of this documentations. If your crawler is respectful, you are not at risk. If you don't honor robots.txt, well maybe you should.
The dragon sits here