WordPress Robots.txt advice from Robots.txt is a way to tell a search engine which pages it’s allowed to spider, to “see”, and which pages it cannot “see”. Because of that, robots.txt differs from meta name=”robots” tags, which tell search engines on those individual pages, whether they can include them in their index or not. The difference is subtle, but important. Because of that, the suggested robots.txt in the codex is wrong. Let me explain:
Google sometimes lists URLs that it’s not allowed to spider, because it’s blocked by robots.txt, because a lot of links point to a URL. A good example of this is a search for [RTL Nieuws] (disclosure: RTL is a client of mine). rtlnieuws.nl 301 redirects to the news section of rtl.nl. But… rtlnieuws.nl/robots.txt exists… And has the following content:
WordPress is web software you can use to create a beautiful website or blog. We like to say that WordPress is both free and priceless at the same time.
The core software is built by hundreds of community volunteers, and when you’re ready for more there are thousands of plugins and themes available to transform your site into almost anything you can imagine. Over 60 million people have chosen WordPress to power the place on the web they call “home” — we’d
Adding this snippet to the functions.php of your wordpress theme will create a new metabox within your post editing screen with a select menu listing all PDF files. Adding the second snippet in your wordpress template in the location you wish to display the files URL.
Another year is coming to a close, and it’s time to look back and reflect on what we’ve accomplished in the past twelve months. The WordPress community is stronger than ever, and some of the accomplishments of the past year are definitely worth remembering. Software Releases We had two major releases of the WordPress web…