Advertisement

Why new topic here is indexed by Google in very few minutes?

Started by May 04, 2011 07:41 AM
8 comments, last by owl 13 years, 6 months ago
Being aware again, my new topic is indexed by Google search engine in very few minutes.
That's to say, after just a while after I started a new topic, I can find it in Google search result.

What's the magic? Only because this forum is active? Or any other magic?

https://www.kbasm.com -- My personal website

https://github.com/wqking/eventpp  eventpp -- C++ library for event dispatcher and callback list

https://github.com/cpgf/cpgf  cpgf library -- free C++ open source library for reflection, serialization, script binding, callbacks, and meta data for OpenGL Box2D, SFML and Irrlicht.

Activity would be a good bet. If I was writing the Googlebot, I would prioritise sites for reindexing based on how frequently they appear to have updated content. Of course, determining what is genuinely updated content as opposed to style changes, new ads or people trying to game the system would be the hard part...
Advertisement
The problem is how the bot finds out what's new.
Indexing whole site in every few minutes is not realistic.

https://www.kbasm.com -- My personal website

https://github.com/wqking/eventpp  eventpp -- C++ library for event dispatcher and callback list

https://github.com/cpgf/cpgf  cpgf library -- free C++ open source library for reflection, serialization, script binding, callbacks, and meta data for OpenGL Box2D, SFML and Irrlicht.

And yet they manage. Either Google have enough resources to make it realistic, or they're using a smarter algorithm. Or both. The relevance of their search engine depends on having timely information when people go to search.

And yet they manage. Either Google have enough resources to make it realistic, or they're using a smarter algorithm. Or both. The relevance of their search engine depends on having timely information when people go to search.


Maybe they subscribe to feeds?
I'm fairly certain the google spider updates site information based off of how frequently it is updated. When you first register your site (or it registers your site for you if you are popular) it will update probably twice a day. If it finds that a large amount of activity happened, it might up it to four times a day and continue increasing/decreasing based off activity. As this forum is really active and probably searched more than most places, it probably updates more than just every 5 minutes.

I'm pretty sure it has an explanation if you go to the site registration page on Google.
Advertisement

I'm fairly certain the google spider updates site information based off of how frequently it is updated. When you first register your site (or it registers your site for you if you are popular) it will update probably twice a day. If it finds that a large amount of activity happened, it might up it to four times a day and continue increasing/decreasing based off activity. As this forum is really active and probably searched more than most places, it probably updates more than just every 5 minutes.

I'm pretty sure it has an explanation if you go to the site registration page on Google.



Yes I agree. You have a very insightful post my friend.
yes Ive noticed this since a couple of years, PITA in some respects eg you ask a question & then you google for further info 10 secs later & your question comes to the top

I just checked www.opengl.org (whos online) & heres what I got

Search Spider Last Activity Location Google 2 seconds ago Reading a post
Forum: High-level APIs (e.g. Inventor, Performer, Optimizer)
Thread: Open Inventor MSN 4 seconds ago Reading a post
Forum: Items of Importance to the OpenGL Community
Thread: The ARB announced OpenGL 3.0 and GLSL 1.30 today Yahoo 27 seconds ago Viewing Active Topics Google 4 minutes 18 seconds ago Reading a post
Forum: OpenGL coding: beginners
Thread: glColorPointer Yahoo 8 minutes 31 seconds ago Reading a post
Forum: User Hardware, Software, & Gaming Help
Thread: Do I have OpenGL on my laptop... how to know?
With regards to search engines, Google and Yahoo tend to stay pretty high on the list of latest page views. They tend to pull a page every few seconds.

Stupid or naive Googlebots could consume a lot of traffic and bring sites to their knees, and Google has a lot of smart programmers. I'm pretty sure they've figured out efficient ways to monitor sites while not angering the site owners for hurting load.
Glad to know that those websites (GDnet, Opengl, etc) don't need to do magic on SE.

Any way no matter how smart they are, the bots from the search engines will consume a lot a lot of traffic every day...

I really like the fact that new information is indexed so quickly. Because now I heavily use Google "site:gamedev.net" to search this site instead of the search function here.

https://www.kbasm.com -- My personal website

https://github.com/wqking/eventpp  eventpp -- C++ library for event dispatcher and callback list

https://github.com/cpgf/cpgf  cpgf library -- free C++ open source library for reflection, serialization, script binding, callbacks, and meta data for OpenGL Box2D, SFML and Irrlicht.

This topic is closed to new replies.

Advertisement