Well, I had a school project that required me and 3 team members to create a web crawler and compute various statistics from sites. Naturally, I crawled GameDev.net. Seeing as I'm sitting in history class and have little motivation to listen, I'm gonna post a few of the results. I was mostly surprised by the fact that gamedev has the lowest word to picture ratio. Each page must reference a lot of little images, I'm guessing.

Anyway, we crawled 10,910 pages on GameDev.net and 41,363 pages on amazon.com, just to give you an idea of how many pages we crawled. And "the" is the most common word on GameDev.net, appearing 162,578 times across the 10,910 pages.

Well, I had a school project that required me and 3 team members to create a web crawler and compute various statistics from sites. Naturally, I crawled GameDev.net. Seeing as I'm sitting in history class and have little motivation to listen, I'm gonna post a few of the results. I was mostly surprised by the fact that gamedev has the lowest word to picture ratio. Each page must reference a lot of little images, I'm guessing.

Anyway, we crawled 10,910 pages on GameDev.net and 41,363 pages on amazon.com, just to give you an idea of how many pages we crawled. And "the" is the most common word on GameDev.net, appearing 162,578 times across the 10,910 pages.

All of this data seems obvious to me. The word "the" is probably the most commonly used word in the English language. Gamedev.net obviously would score lower than the other sites for words per image. I mean look at GDNet's user interface compared to those other sites. Obviously we are going to have fewer misspellings than blogs or Wikipedia since the unintelligent and uneducated masses generally contribute to those sites a lot more. This all seems intuitively apparent to me.

Heh, pretty cool thanks for sharing

Did you actually restrict the crawler to wikipedia and cnn's english sites, or did you have non-english dictionaries for things?

Again, I congratulate you on leaping directly to conclusions completely unsupported by your statistics. *golf clap*

Did you actually restrict the crawler to wikipedia and cnn's english sites, or did you have non-english dictionaries for things?

Of course. I should've put "en." on the wikipedia label [s]and "www." on the cnn label[/s] [edit: I just realized my team member did not use "www." on cnn.com when he ran the crawler, so yes, it did visit cnn's multilingual pages, so cnn's spelling-ness is gonna be off, but then again, see my note about the dictionary below; I did run it on the English part of Wikipedia, that much I know 'cause I did that one myself]

Again, I congratulate you on leaping directly to conclusions completely unsupported by your statistics. *golf clap*

I didn't really come to any conclusions...

@SteveDeFacto: This wasn't really meant to be a mind blowing post. I'm just bored in history class and figured I'd share some results.

[edit]

I should also mention our dictionary was pretty lame (given to us by our professor), as it didn't contain the words "a", "I," or anything with an apostrophe... why, I'm not sure.

My apologies cornstalks, I saw SDF in the summary and thought he made both posts.

My apologies cornstalks, I saw SDF in the summary and thought he made both posts.

Basically after seeing my username you automatically made the assumption that the information must be wrong without even thoroughly reading or trying to understand it? This clearly proves you are an ignorant troll trying to do nothing more than provoke a response from me. I think I'll just ignore you from now on...

[quote name='Telastyn' timestamp='1330485444' post='4917607']
My apologies cornstalks [for accusing you of jumping to conclusions], I saw SDF in the summary and thought he made both posts. [the OP's post with the stat's, and SDF's post with the conclusions].

Basically after seeing my username you automatically made the assumption that the information must be wrong ..... I think I'll just ignore you from now on...[/quote]No, he "saw" that you posted stats and made conclusions from them, instead of Cornstalks posting stats and then you drawing conclusions from them... So he thought the OP was drawing questionable conclusions from their stats... instead of you drawing questionable conclusions from the OP's stats.
Ignoring troll posts instead of replying to them is a good personal policy though... If you don't have anything nice to say, don't start an 8-page flame war over it.

Can you find out which are the most misspelled words on Wikipedia?

[quote name='SteveDeFacto' timestamp='1330487098' post='4917610']
[quote name='Telastyn' timestamp='1330485444' post='4917607']
My apologies cornstalks [for accusing you of jumping to conclusions], I saw SDF in the summary and thought he made both posts. [the OP's post with the stat's, and SDF's post with the conclusions].

Basically after seeing my username you automatically made the assumption that the information must be wrong ..... I think I'll just ignore you from now on...[/quote]No, he "saw" that you posted stats and made conclusions from them, instead of Cornstalks posting stats and then you drawing conclusions from them... So he thought the OP was drawing questionable conclusions from their stats... instead of you drawing questionable conclusions from the OP's stats.
Ignoring troll posts instead of replying to them is a good personal policy though... If you don't have anything nice to say, don't start an 8-page flame war over it.
[/quote]

My point was that he didn't even pay attention to who posted it. As soon as he saw my name he wrote a reply without putting much thought in to it.