Hi, I''m reading Prima''s Multiplayer Game Programming, and I''ve finished my own socket class. I would like to create a small app that retrieves information from a webpage. For example, I know that if I want to view the html code for a certain website, i connect to that website''s IP and port 80, and I send this string "GET /HTTP/1.0\n\n", and I receive the html. But I receive the html for the main page (index.html), how do I get the html for any other page (x1.html, x2.html... etc)? Also, I want this to be a two way communication system, so that I can fill in forms in my app, and the app will send those to the website, so this has to 2-way. Where can I get a list of HTTP commands then? On a diferent topic, can someone explain how proxies work? Thanks for your patience,

[Hugo Ferreira][Positronic Dreams]
I Am Digerati.

http://www.faqs.org/rfcs/rfc1945.html

http://www.faqs.org/rfcs/rfc2616.html

Or read about WinInet SDK at MSDN - there is a better way to this rather then using socket connection.

download Ethereal and look at the http protocol data while you surf the web. You will learn everything you ever wanted to know about HTTP...

lisen to fingh - cause it is the easyest way !

quote:

---

Original post by pentium3id
Hi,
I'm reading Prima's Multiplayer Game Programming, and I've finished my own socket class.
I would like to create a small app that retrieves information from a webpage.
For example, I know that if I want to view the html code for a certain website, i connect to that website's IP and port 80, and I send this string "GET /HTTP/1.0\n\n", and I receive the html.
But I receive the html for the main page (index.html), how do I get the html for any other page (x1.html, x2.html... etc)?
Also, I want this to be a two way communication system, so that I can fill in forms in my app, and the app will send those to the website, so this has to 2-way.
Where can I get a list of HTTP commands then?

On a diferent topic, can someone explain how proxies work?

Thanks for your patience,

[Hugo Ferreira][Positronic Dreams]
I Am Digerati.

---

When you send GET /HTTP/1.0\n\n you get the default page. In order to get x2.html you have to specificly request it. so you'd send something like:

GET http://www.somehost.com/path/x2.html HTTP/1.0\r\n


try this site http://www.jmarshall.com/easy/http/ for more information.



[edited by - prh99 on July 14, 2003 10:37:31 AM]

I also recommend reading the RFCs and using a packet sniffer like ethereal. Just to get you started though, here's a rough outline of what you want to do:

the GET command works like this:

GET file HTTP/1.0\r\n\r\n

where file is the file/page you want returned. eg:

to get http://www.host.com/ you connect to www.host.com and send:
GET / HTTP/1.0\r\n\r\n

to get http://www.host.com/path/page1.html you send:
GET /path/page1.html HTTP/1.0\r\n\r\n

with HTTP/1.0, the server will send back the file in this format:

HTTP/1.0 200 OK\r\n
headers\r\n
\r\n
file data

(the first line will be different for errors eg. 404). The headers will tell you how to deal with the response. one important header is Connection. If it says:

Connection: Close

then the server will close the connetion when all the data has been sent. If it says:

Connection: Keep-Alive

it will keep the connection and let you send further commands when the page is sent. You can tell when all the data has been sent by using the Content-Length header. eg:

Connection: Keep-Alive\r\n
Content-Length: 9\r\n
\r\n
blablabla

You usually specify the Connection mode when you send the GET request by putting your own Connection header after the first \r\n eg:

GET / HTTP/1.0\r\n
Connection: Close\r\n\r\n

HTTP proxies act like normal web servers. You send requests, but you put the whole URL in as the file. eg. if you want to get http://www.host.com/page.html using proxy.whatever.com, you connect to proxy.whatever.com (often port 8080) and send:

GET http://www.host.com/page.html HTTP/1.0\r\n
Connection: Close\r\n\r\n

It'll now send www.host.com's response, probaby with a few of it's own informational headers chucked in.

As for sending data back, web forms usually use one of two methods: GET and POST. Take a look at the HTML for a form. It will have a URL as it's "action". For example say the action is /submit.cgi With the GET method, clicking the submit button will send something like this:

GET /submit.cgi?name=Joe&address=none&tel=123456 HTTP/1.0\r\n
(headers)\r\n
\r\n

i.e. it embeds the form data in the URL. The webserver will handle this and pass everything after the ? to submit.cgi

With the POST method, the POST command is used:

POST /submit.cgi HTTP/1.0\r\n
Content-Length: 32
(headers)\r\n
\r\n
name=Joe&address=none&tel=123456



[edited by - frankd on July 17, 2003 5:51:12 PM]

quote:

---

Original post by pentium3id
Also, I want this to be a two way communication system, so that I can fill in forms in my app, and the app will send those to the website, so this has to 2-way.
Where can I get a list of HTTP commands then?

---

You can send data both ways with HTTP, however, since HTTP is a request-response protocol, HTTP supports only half dublex, i.e. you can send only one way at a time.

In your case, where you want to post some data from a form back to the server, this is no problem. However, in cases where you use the HTTP protocol and you want to be able to pass back and forth data whenever you need, as for example in a game, you can use a trick to simulate full duplex, i.e. sending and recieving data at the same time. Of course, HTTP is not ideal choice for this, but there are cases where HTTP is the only available protocol as on some J2ME devices for example and then you can use this trick as I am in multiplayer game on mobile phones where HTTP is the only available protocol.

In this case, you can initiate a connection to the server (i.e. you send a request from the client) and there server handles the request and generates a unique ID of some sort to pass in its response back to the client. (This is because HTTP is a stateless protocol and you need to keep track of who owns which connection). However, an important thing here is that the server starts its response but actually never finishes it. You use chuncked transfer encoding in the response which means that you can pass chunck by chunck of data to the client in a loop and never finish the response. This way you have now 1 connection that can pass data from server to client whenever you need too.

But we also need to be able to pass data from client to server whenever you need to. This is done by taking the ID the server passed back in the beginning of its chuncked response and initiate one more request from client to server, passing along the ID so the server knows who this is coming from. By setting a high/long content-length before sending the request, the server will continue to read data in as it comes, and client can just pass data on as it wants. (Or you could also use chuncked transfer encoding here too).

Now you have 2 connections which will simulate full duplex over HTTP, one to send data from client to server and one to send from server to client.

This is of course a little bit off topic of what you asked since you just wanted to pass data from a form or something, but anyway, this is a trick you can use if you need too.

quote:

---

On a diferent topic, can someone explain how proxies work?

---

There different types of proxies and they can all do different tasks, some which are the same.

However, one of the most common things about the proxy, compared to a normal firewall, is that the client actually connects to the proxy server, while the proxy server connects to the destination host instead. So the client talk to the proxy and the proxy talks to the host for the client, passing data between this two connections.

One thing the proxy can do then, for example, if it knows the protocol that is used, it can intefere and filter out data it does not want to come in as for example certain parts of a HTML page or even simple words in an email.

But the basic thing is, when using a proxy, is that you connect to the proxy and the proxy connects to the destination so there is never a real connetion between the client and the destination.

Simple said that is