Why do testers review robots.txt during information gathering?

Problem Statement

Robots guidelines sometimes include sensitive or obsolete paths that hint at admin or backup areas. It is a hint, not an access control.

SolutionRead Only

curl -s https://example.com/robots.txt

This question appears in the following practice sets: