A robots.txt file can be used to allow or disallow several bots from visiting a site. It tells a search engine to specify a way to interact with the indexing of your website’s contents.

In a robots.txt file, the following syntaxes are used:

User-agent: *” means this applicable to all robots.

“Disallow: /” tells the robot that it should not visit any pages on the site.
Allow: / will tell robots.txt to check all content.
Disallow: /sample-directory/ will disallow robot to visit the listed directory.
Disallow: /private_file.html will block specific page
Disallow: /images/example.jpg will block specific images
Disallow: /*.gif$ will block .gif file type from access, similarly other extension types can be added here.

You can specify the sitemap rule in the last of your robots.txt file. If you want, for example, http://www.example.com/sitemap.xml.gz

This support article will explain how you can implement a robot.txt file on your Cloudways server.

To connect to your server using SSH please follow this article.

Once you are connected to SSH/SFTP, you can upload your existing robots.txt to the public_html of your application folder.

If the robots.txt file is not uploaded or it is not there, you can create a new robots.txt file under the webroot of your application directory by typing the following command:

Now open the file by typing vi command and append the entries as described in the below screenshot.

You can modify the entries as per your need and save the file, press Esc and then type:

You can also use a windows based text editor like Notepad to create a file robots.txt and then upload it to the public_html of your application folder.

To find out how to connect SFTP to your server, please visit this article.

That’s it! We hope this article was helpful. If you need any help, then feel free to search your query on Cloudways Support Center or contact us via chat (Need a Hand > Send us a Message). Alternatively, you can also create a support ticket.