• Home
  • /
  • Blog
  • /
  • When and How Will Google Index My Website?

questions and answersThis post is part of the weekly Q&A section. Just use the contact form if you want to submit a question.

Rick Regan asks:

I have a two-part question about how Google indexes sites:

1) Is Google supposed to index an entire site at once or does it do so incrementally? My blog (on my own domain) appears to be getting indexed incrementally, to the point where it’s taken a month to index all 30+ of my pages. Most of those pages were present before I manually submitted my URL to Google. I’m wondering if I’m doing something wrong or not doing something I should be doing. Does this have anything to do with my blog being new, or having no external links pointing to it?

2) Does Google eventually drop noindex and 404 pages? I have archive pages that got indexed but I have since added to them the “noindex” tag. I also deleted an empty category that Google now gets a 404 on. Will Google eventually remove those pages as it re-crawls my site?

I will answer to each question individually.

1) Most of the times Google will index a new website gradually, yes. That is at least what I have observed with most of my websites. The speed at which Google will index all your internal pages will depend on different factors though.

If you get some very trusted and relevant backlinks, and on top of that you also have a very efficient internal link structure, all your internal pages will get indexed fast. If, on the other hand, you have very few backlinks and a poor link structure, it might take a while before you get to see all your pages indexed.

Keep in mind that using the manual URL submission to Google will have a small impact upon the speed and breadth of your indexation. In fact many people recommend that if you want to get a site indexed fast you should NOT use that feature, and rather focus on getting some trusted backlinks to your site.

2) Yes, eventually Google will fix those issues. New sites don’t get crawled very often, so that is certainly the reason. As soon as Google finds out about the “noindex” tag, for example, it will remove those pages from their index.


Tags


You may also like

WP Engine Review

Kinsta Review

WPX Hosting Review

About the author 

Daniel

Daniel Scocco is a programmer and entrepreneur located in São Paulo, Brazil. His first company, Online Profits, builds and manages websites in different niches. His second company, Kubic, specializes in developing mobile apps for the iOS and Android platforms.

  1. Hey,

    That’s a nice piece of info….

    Is there a specific time frame that Google follows for refreshing the Google’s search positioning… If yes, please let me know about it.

  2. I also submit sitemaps to Google, Yahoo and Live.I also use Google Webmaster Tools. I have some anecdotal evidence that when you have errors on your site like duplicate titles and meta-descriptions that this seems to have an adverse affect on the pages that are indexed.

  3. since last one year i build this web site,and also verified by webmaster,after this i can’t understand after the expiry one year my web is not working properly,being an affiliate with click bank,having nickname and with other sites,my web site makes not a single dollar,what is the short falls in my site i could not under stand.please help to make this site in full functioned

  4. My site is already indexed by google. When I serach using site:mysiteurl, its ok, but when I google link:mysiteurl it does not show eventhough I have done alot of backlinking. Why?

  5. in this month i lost half of my google index and that was suck, i don’t know what happen …
    is google filtering the site content? and deleting the duplicate index?

  6. Thanks alot, this article is definately worth taking note of.

    Socially bookmarked to bring awareness 🙂

    Karl

  7. yes dear this is true .
    Good backlinks from google trusted sites always will get your site indexed more quickly. We have had performed many experiements to find out what gives the best approach.

  8. Good backlinks from google trusted sites always will get your site indexed more quickly. We have had performed many experiements to find out what gives the best approach.
    (1) manual site submission – can be slow, google will typically send the bot round within day to check that the site has indexable content, and returning a 200.Then over a period of weeks gradually more pages are crawled and indexed.
    (2) Links on High PR / High volume sites ( we believe that google puts more priorty on traffic that is flowing to the site rather than page rank). Typically you get index inclusion after a couple of days.
    (3) Now here is the 24 carret nugget, Site registration period – pay a bit extra and get your domain registered for 10 years and you will get almost immediate indexing. Google have it in their patent – just think about it – with yearly domains around $3 for bulk registrations, google sandbox yearlings but mark 10 year regies as more likely to web sites with conviction behind them rather than spam / squeeze sites. We registered WebTrafficAgents.com and on the second day were at page 4 of 27Million results.

  9. Now Google put the new site in the “Sandbox”. Hence it takes much time to index. Better to use the XML sitemap for gradual indexing.

  10. Daniel,
    i have a new blog and hope that google will index the domain soon before i put more post. Yes, i am working on link building. Thanks for your articles. Useful for me.
    Good 2 know you
    Aiko

  11. Very Good Post.

    I have a number of websites that I have created and had indexed quite quickly.

    I use a few different methods to do this. Submit content to sites that Google loves like ezinearticles and digg.

    I also submit sitemaps to Google, Yahoo and Live.

    I also use Google Webmaster Tools. I have some anecdotal evidence that when you have errors on your site like duplicate titles and meta-descriptions that this seems to have an adverse affect on the pages that are indexed.

    You can also use Google Webmaster Tools to see how often Google crawls your site.

  12. On a somewhat related note, I’m really confounded about the disparity of the search results between Google, Yahoo, and Live (MSN). I submit sitemaps to each of these and they are crawling my site, but the return can be radically different.

    For example, if I use the image search feature of each to search for “site:www.2dolphins.com logo” I get:

    Live – 112 results

    Google – 72 results

    Yahoo – 22 results

    And the funny thing is that there’s very little overlap – not many of the results from Yahoo also show up in the Live image search results.

  13. Yes, #2 is usually just a matter of time before it all gets cleaned up. You just have to be patient because there’s nothing you can do to speed up the process. Consider it a lesson learned I guess!

  14. @Cristian: Webmaster tools says that tool is for “urgent” removal (mine was not urgent – just wanted to remove duplicate content). They recommend noindex or robots.txt otherwise.

    @Calvin: I have a similar problem. The number of URLs Webmaster tools says it’s indexed from my sitemap doesn’t match the URLs returned in a “site: ” search for my site. I don’t know why.

  15. Wouldn’t a site map help? If you loaded a sitemap xml from day one, wouldn’t that help Google easily find all your pages?

  16. Daniel,

    A number of my sites seem to have dropped off Google. Even when I use “site:mysubdomain.mydomain.com”, Google says it can’t find anything.

    But when I look at Google Webmaster Tools, it said my sites are in the index.

    Any idea what’s going on?

  17. I’m glad I stopped by today. I’ve always been curious about the “noindex” tag. I’ve been meaning to research this, but I’ve procrastinated this forever. Maybe I knew I would stop by Daily Blog Tips and find the answer. Thanks!

  18. I think the best way to let google index your sites quickly is submit your web pages to some social bookmark sites, like digg, mixx, reddit, and then you can get a lot of traffic.

  19. @Matej, yeah social bookmarks that pass link juice are a good option to get indexed.

  20. Daniel, thanks for answering my question.

    In the time that has passed since I asked the question Google did eventually index all my pages (well, all but one for some reason) . And yes, the noindexes and 404s eventually got removed as you said they would.

    I have a few backlinks now and I see from my stats that Google crawls my site more frequently now. Most of my posts seem to get indexed within an hour of posting.

  21. Usually my sites get indexed in a couple of hours. I usually submit my site on Digg with the exact keywords. It helps me to get traffic ( google-digg-my site ) and I get indexed really fast.

Comments are closed.

{"email":"Email address invalid","url":"Website address invalid","required":"Required field missing"}