In connection with Sitemaps, we have also built out first-party support for a custom robots.txt file!
Below is quick walk-through of the web interface, but as always, your robots.txt file can be managed via the API.
By default, the sitemap index file (found at /sitemaps/index.xml) is included at the top of the robots.txt file. You can optionally select to suppress the sitemap, though that’s not recommended. And you can supply any content you want to be returned back to bots.
At Revenue HQ, we allow all bots to follow all links, except to our /api/ directory. You can see our robots.txt file at https://revenuehq.com/robots.txt.
