Is a robots.txt file needed in phpBB?

[halilesen]

robots.txt is a file where you can tell search engines and other bots which parts of your website should be crawled and which parts should not.

phpBB does not need a robots.txt file by default. This is because phpBB has an exceptional permissions system. If you don't allow the robots to see your profile page, they will never see it. However, even if you tell the robots not to crawl the profile page with the robots.txt instruction, there is a chance that the bots will not obey it. So the phpBB permission system is much stronger and healthier than robots.txt instructions. I recommend you to check both forum permissions and group permissions for bots.

However, in some cases it can be useful to have a robots.txt file. I think it is useful to use it for both in SEO analysis's and add sitemap. Let's create a robots.txt file for phpBB forum. I assume phpBB is installed in root.

Create a file named robots.txt to add to root. Add it inside:

Code: Select all

User-agent: *
Allow: /

Sitemap: https://phpbbgroup.com/sitemap.xml

Here I have allowed the entire site to be crawled. Even if I didn't add this, it's already the default. phpBB's permissions system already works, so it won't make a difference. However, my purpose here was to draw attention to the sitemap. Replace the sitemap URL with your phpBB board sitemap URL and save.

[halilesen]

You can also add to the robots.txt file to block some bots in your phpBB board. For example:

Code: Select all

User-agent: CCBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Elisabot
User-agent: GPTBot
User-agent: UptimeRobot
Disallow: /

Add to top and save. There are bots that don't care about the robots.txt file, but there are some that do.