A robots.txt file can be used to allow or disallow several bots from visiting a site. It tells a search engine to specify a way to interact with the indexing of your website’s contents.
In a robots.txt file, the following syntaxes are used:
User-agent: *” means this applicable to all robots.
“Disallow: /” tells the robot that it should not visit any pages on the site.
Allow: / will tell robots.txt to check all content.
Disallow: /sample-directory/ will disallow robot to visit the listed directory.
Disallow: /private_file.html will block specific page
Disallow: /images/example.jpg will block specific images
Disallow: /*.gif$ will block .gif file type from access, similarly other extension types can be added here.
You can specify the sitemap rule in the last of your robots.txt file. If you want, for example, http://www.example.com/sitemap.xml.gz
This support article will explain how you can implement a robot.txt file on your Cloudways server.
How to Setup robots.txt for Your Website
Step 1: Access Your Server via SSH
To connect to your server using SSH please follow this article.
Once you are connected to SSH/SFTP, you can upload your existing robots.txt to the public_html of your application folder.
Step 2: Creating a robots.txt File
If the robots.txt file is not uploaded or it is not there, you can create a new robots.txt file under the webroot of your application directory by typing the following command:
Now open the file by typing vi command and append the entries as described in the below screenshot.
You can modify the entries as per your need and save the file, press Esc and then type:
Adding a robots.txt File via SFTP
You can also use a windows based text editor like Notepad to create a file robots.txt and then upload it to the public_html of your application folder.
To find out how to connect SFTP to your server, please visit this article.