Chapter 4: Technical SEO
Not all pages should be indexed. For instance, sign-in pages, secure forms, and others don’t provide any real SEO value in the first place and can eat into crawl budget. So, to prevent a website's pages from being indexed by Google and removed from search results, you can take several measures.
Use robots.txt. Add a robots.txt file to your website's root directory with the following content:
User-agent. *
Disallow. /
This will block all search engine crawlers from accessing your website.
Add a meta tag. In the <head> section of your HTML pages, add the following meta tag.
html
<meta name="robots" content="noindex, nofollow">
This tells search engines not to index the page and not to follow any links on the page.
Use the X-Robots-Tag HTTP header. You can set the X-Robots-Tag HTTP header to "noindex" to prevent search engines from indexing a page. In your server configuration, you can add the following rule: Header set X-Robots-Tag "noindex" and this will prevent search engines from indexing the page.
Password-protect your pages. If you want to keep your pages private, you can password-protect them. This way, only users with the correct login credentials can access the pages, and search engines won't be able to index them.
Use Google Search Console. If you want to remove a page from Google's search results, you can use the Google Search Console. Go to the "Removals" tool and submit the URL of the page you want to remove. This will temporarily remove the page from Google's search results for about six months.
Remember that these methods may not guarantee complete removal from search engines, as other websites may still link to your pages and search engines may have already indexed your content.