Blog Privacy: Don't add custom rules to wpcom robots.txt if blog_public=0

[dereksmart]

In #35803, these rules were added to robots.txt regardless of the environment. It cluttered up the robots.txt and was confusing on simple, for sites that are discouraging search engines already.

This change implements custom behavior based on the environment

• wpcom: different
• core/woa: same as before

I think we can revert the wpcom test change too D161949-code

Proposed changes:

• Do not output extra robots.txt rules in wpcom simple environment when blog_public=0

Other information:

• Have you written new tests for your changes, if applicable?
• Have you checked the E2E test CI results, and verified that your changes do not break them?
• Have you tested your changes on WordPress.com, if applicable (if so, you'll see a generated comment below with a script to run)?

Jetpack product discussion

p1726689440830569/1726685246.860939-slack-C02AVAR9B

Does this pull request change what data or activity we track or use?

No

Testing instructions:

Full instructions in Field Guide here for both Atomic and WPCOM testing PCYsg-Osp-p2#simple-testing

In WPCOM simple:

• jetpack build plugins/mu-wpcom-plugin
• jetpack rsync mu-wpcom-plugin {yoursandbox/to/mu-plugins/jetpack-mu-wpcom-plugin/production}
• Point a simple site to your sandbox
• Comment out the domo_arigato_mr_sandboxo hook (in wpcom sandbox)
• Visit the site's /robots.txt

Toggle these settings and test individually:

• Discourage search engines from indexing this site: robots.txt should just have User-agent: * Disallow: /
• Prevent third-party sharing for dsmartsandbox.wordpress.com: Should show the list of AI user agents in robots.txt.

Test also on a WoA site using the Jetpack Beta plugin

[github-actions]

Are you an Automattician? Please test your changes on all WordPress.com environments to help mitigate accidental explosions.

• To test on WoA, go to the Plugins menu on a WordPress.com Simple site. Click on the "Upload" button and follow the upgrade flow to be able to upload, install, and activate the Jetpack Beta plugin. Once the plugin is active, go to Jetpack > Jetpack Beta, select your plugin, and enable the fix/blog-privacy-cluttering-wpcom-robots-txt branch.

  • For jetpack-mu-wpcom changes, also add define( 'JETPACK_MU_WPCOM_LOAD_VIA_BETA_PLUGIN', true ); to your wp-config.php file.

• To test on Simple, run the following command on your sandbox:

  bin/jetpack-downloader test jetpack-mu-wpcom-plugin fix/blog-privacy-cluttering-wpcom-robots-txt
  

Interested in more tips and information?

• In your local development environment, use the jetpack rsync command to sync your changes to a WoA dev blog.
• Read more about our development workflow here: PCYsg-eg0-p2
• Figure out when your changes will be shipped to customers here: PCYsg-eg5-p2

[github-actions]

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

• ✅ Include a description of your PR changes.
  
• ✅ Add a "[Status]" label (In Progress, Needs Team Review, ...).
  
• ✅ Add testing instructions.
  
• ✅ Specify whether this PR includes any changes to data or privacy.
  
• ✅ Add changelog entries to affected projects
  

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖

---

The e2e test report can be found here. Please note that it can take a few minutes after the e2e tests checks are complete for the report to be available.

---

Follow this PR Review Process:

1. Ensure all required checks appearing at the bottom of this PR are passing.
2. Choose a review path based on your changes:
   • A. Team Review: add the "[Status] Needs Team Review" label
     • For most changes, including minor cross-team impacts.
     • Example: Updating a team-specific component or a small change to a shared library.
   • B. Crew Review: add the "[Status] Needs Review" label
     • For significant changes to core functionality.
     • Example: Major updates to a shared library or complex features.
   • C. Both: Start with Team, then request Crew
     • For complex changes or when you need extra confidence.
     • Example: Refactor affecting multiple systems.
3. Get at least one approval before merging.

Still unsure? Reach out in #jetpack-developers for guidance!