Issue #2364343 by damien_vancouver, criz, ksenzee, Neograph734, joegraduate, k_zoltan, droplet, pounard, jp.stacey, ciss, corbacho, TravisJohnston: Fix robots.txt to allow search engines access to CSS, JavaScript and image files

Author: DavidRothstein

CHANGELOG.txt

@@ -32,6 +32,8 @@ Drupal 7.50, xxxx-xx-xx (development version)
   tests within a project.
 - Made run-tests.sh exit with a failure code when there are test fails or
   problems running the script.
+- Fixed robots.txt to allow search engines to access CSS, JavaScript and image
+  files.
 
 Drupal 7.44, 2016-06-15
 -----------------------

robots.txt

@@ -15,6 +15,39 @@
 
 User-agent: *
 Crawl-delay: 10
+# CSS, JS, Images
+Allow: /misc/*.css$
+Allow: /misc/*.css?
+Allow: /misc/*.js$
+Allow: /misc/*.js?
+Allow: /misc/*.gif
+Allow: /misc/*.jpg
+Allow: /misc/*.jpeg
+Allow: /misc/*.png
+Allow: /modules/*.css$
+Allow: /modules/*.css?
+Allow: /modules/*.js$
+Allow: /modules/*.js?
+Allow: /modules/*.gif
+Allow: /modules/*.jpg
+Allow: /modules/*.jpeg
+Allow: /modules/*.png
+Allow: /profiles/*.css$
+Allow: /profiles/*.css?
+Allow: /profiles/*.js$
+Allow: /profiles/*.js?
+Allow: /profiles/*.gif
+Allow: /profiles/*.jpg
+Allow: /profiles/*.jpeg
+Allow: /profiles/*.png
+Allow: /themes/*.css$
+Allow: /themes/*.css?
+Allow: /themes/*.js$
+Allow: /themes/*.js?
+Allow: /themes/*.gif
+Allow: /themes/*.jpg
+Allow: /themes/*.jpeg
+Allow: /themes/*.png
 # Directories
 Disallow: /includes/
 Disallow: /misc/