This post is part of the weekly Q&A section. Just use the contact form if you want to submit a question.
Rick Regan asks:
I have a two-part question about how Google indexes sites:
1) Is Google supposed to index an entire site at once or does it do so incrementally? My blog (on my own domain) appears to be getting indexed incrementally, to the point where it’s taken a month to index all 30+ of my pages. Most of those pages were present before I manually submitted my URL to Google. I’m wondering if I’m doing something wrong or not doing something I should be doing. Does this have anything to do with my blog being new, or having no external links pointing to it?
2) Does Google eventually drop noindex and 404 pages? I have archive pages that got indexed but I have since added to them the “noindex” tag. I also deleted an empty category that Google now gets a 404 on. Will Google eventually remove those pages as it re-crawls my site?
I will answer to each question individually.
1) Most of the times Google will index a new website gradually, yes. That is at least what I have observed with most of my websites. The speed at which Google will index all your internal pages will depend on different factors though.
If you get some very trusted and relevant backlinks, and on top of that you also have a very efficient internal link structure, all your internal pages will get indexed fast. If, on the other hand, you have very few backlinks and a poor link structure, it might take a while before you get to see all your pages indexed.
Keep in mind that using the manual URL submission to Google will have a small impact upon the speed and breadth of your indexation. In fact many people recommend that if you want to get a site indexed fast you should NOT use that feature, and rather focus on getting some trusted backlinks to your site.
2) Yes, eventually Google will fix those issues. New sites don’t get crawled very often, so that is certainly the reason. As soon as Google finds out about the “noindex” tag, for example, it will remove those pages from their index.
Also make your site relevant to the user’s search terms and thereby Google – get lots of back links and if you’re feeling behind the competition check them out. I use back link checker – just Google it. Scopes out where my competition is getting their back links and I have found it a good way to find relevent directories to sign up for – and thereby get links from.
Posting on blogs doesn’t hurt either 🙂
I am really new to this internet marketing game I have to say I really like this site and I am using it to try and get higher rankings with my word press blog right now it seems as though I can’t even get indexed hoping to get that moving with uh what do you call it link juice lol
thanks for everyones comments
Getting a good ranking on google takes along time. I know how easy it is to get discourages when huge websites or even incompetent sites rank better and easier than yours. Just keep adding a reasonable amount of backlinks everyday and make sure you add to the site everyday with new, interesting content.
goodluck
Thanks for your answers. I have been having the same questions recently as my blog was getting listed but some pages of my website were not. The information that I got from everyone really helped me to understand just exactly how search engine works.
Thanks again
Chris
@Jacklin – thanks, I may well do, that’s a very kind offer!
My website is about how to negotiate a salary increase, but it’s appearing very difficult to rank highly over some of the articles from “About” etc.
I’ve been working on backlinks and now have used a social bookmarking service to get me sorted, though that has only just finished so I’d imagine it will take google a few weeks to recognize the backlinks.
I have read that .gov and .edu backlinks are extremely valuable also but most seem to be nofollow so I’m not getting recognition from them.
Yahoo also seems to have about 90 backlinks recognized for me now, whereas google have none…..
I have been waiting 2 weeks for my meta description to be updated, today I have added 3 monre pages to my site and I am about to submit my first sitemap to google.
I am interested to know if Google will visit my site really fast as it’s my first site map submission?
This is a very interesting post and I am off to work on some more backlinks.
I did notice that no-one has mentions .edu backlinks as I was under the impression that Google likes these as they are from an authoritative domain extension. I have added about 50 in the past few weeks and I am hoping they have some impact too!
Thanks for the info.
My site was getting indexed within minutes after i had made a post, but for a week my server was down and i was not in town. when i came back, and started my site again. the google just takes days to index any new posts and i have lost all my traffic.
I got index in google in 1 week after i added sitemap & robot txt, but how i am suppose to know on which page my website is showing in search result on my keyword? any suggestion?
I am using google webmaster tools, sitemap.xml, bookmarked on social web sites but I found that my sitemap.xml is not listing all blog posts hence not getting indexed in google too
Help, I launched my new site and submitted it to google. Its in wordpress and I have used the settings to let everyone see it. For some reason google wont index it, am I doing something wrong?
Hey Sean,
If you want to get in touch with me, please contact me on facebook buddy.
Just make a search for “Jacklin Octave”
It’s easy to understand social bookmarking sites.
@Mike, you don’t have to pay for indexing tho. Just let google and other search engines do it for you, but you have to register in their webmaster tools.
Please, free to contact the owner of that blog, or should me an email for more details.
Take care!
Jack
Interesting read – can anyone give me some more insight into social bookmarking sites? I’ve heard delicious and digg mentioned in posts but I’m not aware of how they work or why I would use them.
Thanks in advance,
You have to watch out for companies that will overcharge your for “indexing service”. Some companies will take advantage of consumers/businesses and charge up to $300 dollars for this service.
starcomsystems.net is a prime example.
There are a plethora of ways to get indexed. You can actually use social bookmarking sites, google webmaster tools, or the quickest way is to submit your website to all the ping sites.
Think about it, every time you make a change or a post, just ping it and you will see how gradually your traffic grows.
And the secret is by doing what i’m doing right now….LOL
Find out!!!
Take care
Jacklin
Very nice blog.
I am also optimizing a site. I use ezinearticles, digg, mixx, forums, press releases, etc to bring my site up in search. My site is improving on daily bases.
I hope one day it gets on first page of the site.
@Bryan, you are frustrated with just one web site. Imagine how frustrated the Google Bot can get, he has billions of site to crawl. He cannot run, so be patient.
There is really no fast way to do this. If you go too fast then Google will think you just bought all those links and might even ban your site. Slow organic link building is the best way to go.
thanks Ed, it really sounds frustrating though because now the WMT again shows no brandnew links for my site for 3 days. My blogs and directory submissions didnt give any favorable results and so were my blogs and forums participations. So these does not automatically give us links or google just missed those links during certain crawls? I want some clarifications regarding when a link will be found or listed in WM or in any link checker tools? thnks
Hello,
In Google’s point of view, which one it valued the most those links from directory listings, blogs, articles, or comments…Id try to do each of these but it seems directory listings works for me most of the time if i checked in webmastertools. I know making blogs/articles is a long process but im learning it now and hoping to post a better one someday. I just want to know what works the most so that i can concentrate on that to help boost my site’s ranking. My site is about a month old now but it’s still has “0” ranking and I don’t gaining enough links everyday, do you know some ways to gain links really fast? I mean in legal way possible because its bad to called spammers and it wont be good for our site. thanks
Definitely you need to wait. Google has to crawl billions of pages daily. Keep that in mind.