The old robots.txt was invalid because "User-agent: *" was found twice in robots.txt. Using the API will generate correct robots.txt. |
||
---|---|---|
.. | ||
docs_controller_spec.rb | ||
robots_txt_controller_spec.rb |
The old robots.txt was invalid because "User-agent: *" was found twice in robots.txt. Using the API will generate correct robots.txt. |
||
---|---|---|
.. | ||
docs_controller_spec.rb | ||
robots_txt_controller_spec.rb |