Networking sub-forum is generally quiet about now, so better here.

I have been slowly creeping upwards in the networking part of my project.

The problem I currently have is illustrated in this diagram ... a lot of times my English fails me so I'm guessing the diagram might complement my communications inadequacies. All works fine from stages 1 to 5 in real-time only if I send a saved file from a previous run-time (not the current file just received from client, saved to server HD) back to client. But fails if i try to the send same file received (from the current server run-time) and saved to HD. Is this a limitation of client server - is it the way it is meant to work? or am I doing something wrong?

I would like to send current file just saved in the same server run-time, because the current file would be used to generate more info and data that would be added on (enhancing the puzzle game) ... before it is read,uploaded and sent back to client. many Thanks

[attachment=35493:client server update.png]

This sounds like it's probably a bug in your code. There is nothing inherently impossible about doing what you describe.

Can you post the server code involved?

Here it is, many thanks

Included is the client code, just in case you need to check

In the server code, if FILE_RECEIVED (from client) and FILE_TO_SEND are different, it works (server returns the file to client), if they are the same (that is, file_to_send is pointing to file_received) the server doesn't return the file

I've cleaned code of debug prints, so it should be very readable now. The Android code before the send button code can be ignored, as it is less relevant

server code

public class FileServer {    
      public final static String FILE_RECEIVED ="C:\\root_folder\\QCD_070075c.jpg";
      public final static String FILE_TO_SEND = "C:\\root_folder\\QCD_050050z.jpg"; 
  
    public static void main(String[] args) throws IOException {
        int filesize=2373620;                    //temporarilly hardcoded but large enough
        long start = System.currentTimeMillis();       
        int bytesRead;     
        int current = 0;   
       //===============================================
        FileInputStream fis = null;
        BufferedInputStream bis = null;
        OutputStream os = null;           
        ServerSocket servsock = null;
        Socket sock = null;            
        //==============================================
        InetAddress IP=InetAddress.getLocalHost();    
        servsock = new ServerSocket(57925);      
        System.out.println("IP "+IP.getHostAddress()+"  ***%% :"+servsock.getLocalPort());  
                   
        while (true) {  
            System.out.println("receiving frm client and then send bk...");           
            System.out.println("555_999 Waiting...");           
            sock = servsock.accept();
            System.out.println("Accepted connection : " + sock);
                             
            byte [] mybytearray  = new byte [filesize];
            InputStream is = sock.getInputStream();
            FileOutputStream fos = new FileOutputStream(FILE_RECEIVED); // destination path and name of file
            BufferedOutputStream bos = new BufferedOutputStream(fos);

   //=================== read integer from client ==========
            InputStreamReader isr = new InputStreamReader(is);
            BufferedReader br = new BufferedReader(isr);
            String number = br.readLine();
            System.out.println("integer received from client is "+String.valueOf(number));
   //=======================================================         
            bytesRead = is.read(mybytearray,0,mybytearray.length);
            current = bytesRead;
                                         
            do {
               bytesRead =  is.read(mybytearray, current, (mybytearray.length-current));
               if(bytesRead >= 0) current += bytesRead;
            } while(current < Integer.valueOf(number));

            bos.write(mybytearray, 0 , current);
            bos.flush();
            long end = System.currentTimeMillis();
            System.out.println("AT SERVER: bytesRead  "+bytesRead+"  current  "+current);
            bos.close();

//======================== preparing to Send Data to Client  ==============================              
            // send file
            File myFile = new File (FILE_TO_SEND);
            mybytearray  = new byte [(int)myFile.length()];
            fis = new FileInputStream(myFile);
            bis = new BufferedInputStream(fis);
            bis.read(mybytearray,0,mybytearray.length);
            os = sock.getOutputStream();
  //=========================== send integer to client ===============                   
           OutputStreamWriter osw = new OutputStreamWriter(os);
            BufferedWriter bw = new BufferedWriter(osw);
            number = Integer.toString(mybytearray.length);
            String sendMessage = number + "\n";
            bw.write(sendMessage);
            bw.flush();   
  //========================== send file to client ===================                
            System.out.println("Sending " + FILE_TO_SEND + "(" + mybytearray.length + " bytes)");
            os.write(mybytearray,0,mybytearray.length);
            os.flush();
            System.out.println("number  "+number);
            System.out.println("Done.");
 //           servsock.close();
 //           sock.close(); 
/**/       }
    } 
}

client code

public class FSendfileActivity extends Activity {
    private Socket sock;
    private static final int SELECT_PICTURE = 1;
    private String serverIP = "192.168.1.4";    
    private String selectedImagePath;
    private ImageView img;
    final static String qcd = "qcd";
    String ImageDir2Client;      
    FileOutputStream fos = null;
    BufferedOutputStream bos = null;
//====================
    public  static String   FILE_TO_RECEIVED=null;
    int bytesRead = -1;
    int current = 0; 
     
    public final static int FILE_SIZE = 2373620;
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.fsendfile);
 
	    ImageDir2Client = Environment.getExternalStorageDirectory().getAbsolutePath();
	    FILE_TO_RECEIVED = ImageDir2Client + "/clientRootFolder/fromServer000018ccc.jpg";  
	   	     
	    Log.v(qcd, "setup complete ^^^^^ ...");

        img = (ImageView) findViewById(R.id.ivPic);
         
       ((Button) findViewById(R.id.bBrowse)).setOnClickListener(new OnClickListener() {
           public void onClick(View arg0) {
              
                Intent intent = new Intent();
                intent.setType("image/*");
                intent.setAction(Intent.ACTION_GET_CONTENT);
                startActivityForResult( Intent.createChooser( intent, "Select Picture" ), SELECT_PICTURE );
            }
       });
              
       Button send = (Button) findViewById(R.id.bSend);
       final TextView status = (TextView) findViewById(R.id.tvStatus);
       
       send.setOnClickListener(new View.OnClickListener() {
           @Override                     
           public void onClick(View arg0) {
           	new Thread(new Runnable() {                  
                   @Override        
                   public void run() {            
                       try {                                     
                       	    sock = new Socket();     
                            connection(sock,  serverIP, 57925);
                       	
                            File myFile = new File (selectedImagePath); 
                            byte [] mybytearray  = new byte [(int)myFile.length()];
                            FileInputStream fis = new FileInputStream(myFile);
                            BufferedInputStream bis = new BufferedInputStream(fis);
                            bis.read(mybytearray,0,mybytearray.length);
                            OutputStream os = sock.getOutputStream();
            //=========================== send integer to server ===============                   
                            OutputStreamWriter osw = new OutputStreamWriter(os);
                            BufferedWriter bw = new BufferedWriter(osw);
                            String number = Integer.toString(mybytearray.length);
                            String sendMessage = number + "\n";
                            Log.v(qcd, "mybytearray.length    "+mybytearray.length);
                            bw.write(sendMessage);   // send size integer here
                            bw.flush(); 
            //==================================================================                
                          
                        os.write(mybytearray,0,mybytearray.length); // send file
                        os.flush();  
                                          
    //=================  client receiving data ==============================
                 
                        // receive file
                        mybytearray  = new byte [FILE_SIZE];                       
                        InputStream is = sock.getInputStream();                   
                       fos = new FileOutputStream(FILE_TO_RECEIVED);                
                        bos = new BufferedOutputStream(fos);
        //=================== read integer from client ==========                    
                        InputStreamReader isr = new InputStreamReader(is);                   
                       BufferedReader br = new BufferedReader(isr);                
                        number = br.readLine();                 
        //=======================================================         
                        bytesRead = is.read(mybytearray,0,mybytearray.length);     
                       current = bytesRead;
                        do {
                           bytesRead = is.read(mybytearray, current, (mybytearray.length-current));
                           if(bytesRead >= 0) current += bytesRead;
                        } while(current < Integer.valueOf(number));
 
                        bos.write(mybytearray, 0 , current);
                        bos.flush();
                        System.out.println("File " + FILE_TO_RECEIVED
                            + " downloaded (" + current + " bytes read)");
                                                         
   /* */                  sock.close();
                    } catch (UnknownHostException e) {
                        e.printStackTrace();
                    } catch (IOException e) {
                        e.printStackTrace();
                    }  finally{
                        try{
         //                   sock.close();
                        }
                        catch(Exception e){
                            e.printStackTrace();
                        }
                    } 
            	} 
	          }).start();
	        }
	    });
        
    }       
    public static void connection(Socket s,  String serverIP, int port) {
        try {   
        	Log.v(qcd, " before connecting ****...");
        	s.connect(new InetSocketAddress(serverIP, port), 120000);
        	Log.v(qcd, " socket connection DONE!! ");
        } catch (UnknownHostException e) {
            e.printStackTrace();
            Log.v(qcd, " Unknown host..."+e);
        } catch (IOException e) {
            e.printStackTrace();
            Log.v(qcd, " Failed to connect...   "+e);
        }
    }
    public void onActivityResult(int requestCode, int resultCode, Intent data) {
        if (resultCode == RESULT_OK) {
            if (requestCode == SELECT_PICTURE) {
                Uri selectedImageUri = data.getData();
                selectedImagePath = getPath(selectedImageUri);
                TextView path = (TextView) findViewById(R.id.tvPath);
                path.setText("Image Path : " + selectedImagePath);
                img.setImageURI(selectedImageUri);
                Log.v(qcd, selectedImagePath);
            }     
        }
    }

    public String getPath(Uri uri) {
        String[] projection = { MediaStore.Images.Media.DATA };
        Cursor cursor = managedQuery(uri, projection, null, null, null);
        int column_index = cursor.getColumnIndexOrThrow(MediaStore.Images.Media.DATA);
        cursor.moveToFirst();
        return cursor.getString(column_index);
    }
}

What is the failure? Error / exception - on which side and what does it say? File received is truncated or incorrect - describe? Transfer too slow for "real time" requirement - how long? Something else?

My first guess is that a file handle remains open from the initial transfer, I don't see any close() calls on the file related objects in the client.

Consider wrapping the file handling code in a "try with resources" block on both the client and server - this makes it easier to get right. You can also do this with the socket if the connection is a once off.

What is the failure? Error / exception - on which side and what does it say? File received is truncated or incorrect - describe? Transfer too slow for "real time" requirement - how long? Something else?

It doesn't really crash in that sense, but through debug prints i know it hangs indefinitely in the server code, at the line after receiving the integer file size (before receiving the actual file. I cleaned other debug prints for readability). It hung indefinitely after this print

InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String number = br.readLine();
System.out.println("integer received from client is "+String.valueOf(number));

My first guess is that a file handle remains open from the initial transfer, I don't see any close() calls on the file related objects in the client. Consider wrapping the file handling code in a "try with resources" block on both the client and server - this makes it easier to get right. You can also do this with the socket if the connection is a once off.

Will try these suggestions now and update (edit) the thread after wards. Some close() calls had previously resulted in connection close() peer to peer exception, so had to remove. But I will close some few objects now and run

To take out file io as source of problems, you could save in a buffer, and send that back.

Using buffered IO, and then using the socket again assumes the buffered io doesn't eat more input data than exactly that one line, which is kind of dangerous.

Reading a decimal number from bytes isn't very difficult, modulo checking everything to death:

int number = 0;
int length = 0;
while (true) {
    byte k = read_byte(is); // Needs IO exception handling
    if (k == '\n') break;
    // check that k >= '0' and k <= '9'
    number = number * 10 + k - '0';
    // check that number stays in safe ranges
    length = length + 1;
    // check that length stays sane
}
// check that length > 0;

Didn't try to complile it, it may need a cast or an explicit conversion to a digit value.

Wow! this client server thing is really getting very crazy, so much inconsistencies in the output. One moment an half way improvement to what I already had is working fine, the next, with little/trivial or no changes at all and the read(byte[] b, int off, int len) blocks continuously, even the previously back steps that had no problems at all is blocked. Almost as deadly and mysterious as the image processing nightmarish coding i encountered several months ago. [ a 'block' is defined in the line in red below and also more fully in the documentation below]

It all has to do with this code (in the server code)

          bytesRead = is.read(mybytearray,0,mybytearray.length);
          current = bytesRead;
         
          do {
             bytesRead =  is.read(mybytearray, current, (mybytearray.length-current));
             if(bytesRead >= 0) current += bytesRead;
          } while(bytesRead > -1);

I barely understand the code and I dont really understand why it blocks here. I read read(byte[] b, int off, int len) doc over and over again. Why do the examples i google online all have the line current = bytesRead; Without that line i might have understood the code better. But I wouldn't mind even not understanding it .... as long as it works! And thats thing - it is not consistent. Sometimes the code moves closer to working, in that if i hard-code the correct number of the currently saved bytes in the do-while loop -> ex. while(current < 248843);, it works, but then sometimes even this and the previously working stage blocks indefinitely

read(byte[] b, int off, int len) documentation

int java.io.InputStream.read(byte[] b, int off, int len) throws IOException

• read

  
  public int read(byte[] b,
                  int off,
                  int len)
           throws IOException

  Reads up to len bytes of data from the input stream into an array of bytes. An attempt is made to read as many as len bytes, but a smaller number may be read. The number of bytes actually read is returned as an integer.

  This method blocks until input data is available, end of file is detected, or an exception is thrown.

  If len is zero, then no bytes are read and 0 is returned; otherwise, there is an attempt to read at least one byte. If no byte is available because the stream is at end of file, the value -1 is returned; otherwise, at least one byte is read and stored into b.

  The first byte read is stored into element b[off], the next one into b[off+1], and so on. The number of bytes read is, at most, equal to len. Let k be the number of bytes actually read; these bytes will be stored in elements b[off] through b[off+k-1], leaving elements b[off+k] through b[off+len-1] unaffected.

  In every case, elements b[0] through b[off] and elements b[off+len] through b[b.length-1] are unaffected.

  The read(b, off, len) method for class InputStream simply calls the method read() repeatedly. If the first such call results in an IOException, that exception is returned from the call to the read(b, off, len) method. If any subsequent call to read() results in a IOException, the exception is caught and treated as if it were end of file; the bytes read up to that point are stored into b and the number of bytes read before the exception occurred is returned. The default implementation of this method blocks until the requested amount of input data len has been read, end of file is detected, or an exception is thrown. Subclasses are encouraged to provide a more efficient implementation of this method.

  Parameters: b - the buffer into which the data is read. off - the start offset in array b at which the data is written. len - the maximum number of bytes to read. Returns: the total number of bytes read into the buffer, or -1 if there is no more data because the end of the stream has been reached. Throws: IOException - If the first byte cannot be read for any reason other than end of file, or if the input stream has been closed, or if some other I/O error occurs. NullPointerException - If b is null. IndexOutOfBoundsException - If off is negative, len is negative, or len is greater than b.length - off See Also: read()

This is bordering insanity, seems like I spent all day chasing my own tail (like I have one).

There isn't an exception or a stacktrace to read even when I place a try-catch around the code, but it just blocks indefinitely. The doc says

• This method blocks until input data is available, end of file is detected, or an exception is thrown.

When I place print out all over the place, i see that - the problem starts with the first read(), it reads way less than half the file size sent from the client. I dont know why. And then the second read() blocks forever. This happens only when I try to read the same file save at the current server run time. As described in the first post - when I read a previously saved file even at the current server in real time, it work fine , but then even this also blocks at times. If you think what i'm writing deosn't make any sense, you're very correct because it really makes no sense.

At one stage, things was so chaotic I had to revert back to a backup for both server and client code to get going again!

dunno if anyone has some knowledge or experience with this and can help with a FIX/hint/solution. Many thanks

To take out file io as source of problems, you could save in a buffer, and send that back.

Using buffered IO, and then using the socket again assumes the buffered io doesn't eat more input data than exactly that one line, which is kind of dangerous.

Reading a decimal number from bytes isn't very difficult, modulo checking everything to death:

int number = 0;
int length = 0;
while (true) {
    byte k = read_byte(is); // Needs IO exception handling
    if (k == '\n') break;
    // check that k >= '0' and k <= '9'
    number = number * 10 + k - '0';
    // check that number stays in safe ranges
    length = length + 1;
    // check that length stays sane
}
// check that length > 0;

Didn't try to complile it, it may need a cast or an explicit conversion to a digit value.

Thanks Alberth. Didn't see your post in good time as this problem locked me in my own little world for a long while. So with this way, will I have to write a little function -> read_byte()?

I guess by embedding a Java inbuilt code that reads byte and returns an int for each byte, into read_byte(). So at the completion of the loop within read_byte, the total is returned to 'k', correct? Since the other data I am sending between client and server code is an image file, would this still work? As I shouldn't be able to test for the numbers between 0-9

it reads way less than half the file size sent from the client. I dont know why.

This one is easy. Remember the network code splits your big data block into a lot of small packets and sends each of them to the other side?

These packets do not arrive all at the same time, so if you read at the other side, and a bunch arrives, the kernel gives you what it received so far, up to the length you requested. Normally, it is less than what you asked for, unless you start asking for a single byte or so. In any case, your local processing is always faster than the network, so at some point you'll run out of received bytes, and you have to wait again for more.

You may think the stream is "<number>" "\n" "<data>" but to the application it is one long stream of bytes without any separation markers whatsoever. Data doesn't arrive in blocks as you wrote it. Big blocks get split, smaller blocks may get merged before sending. It is totally feasible that the receiving end gets "<number>\n<d" and "ata>" or it might get "<numb", "er", ">\n<dat", and "a>", or some other random split on the stream in small packets.

As for not getting more after that, until you proof otherwise, my guess is that your buffered io wrappers for reading the number are eating some of your file data. Don't read from different levels of io layers from a single source, it's not reliable. Lower layers may optimize performance by reading more than needed at higher level etc.

And then the second read() blocks forever. This happens only when I try to read the same file save at the current server run time. As described in the first post - when I read a previously saved file even at the current server in real time, it work fine , but then even this also blocks at times. If you think what i'm writing deosn't make any sense, you're very correct because it really makes no sense.

After reading the InputStream.read description, and your buffer loop reader, it starts to make sense, but it's a few steps before you arrive there.

The first thing what you must do is to stop using code that you do not fully understand. If you don't entirely understand your "read buffer" loop code, you're coding the higher levels on top of "I think it might do X", for some unknown value of X. Together with the inherently random nature of network IO wrt timing, speed, and bandwidth, you get what you experience, seemingly random behavior that works and then breaks again.

The only way forward is to exactly know what X is. You do that by writing your own "read buffer" loop from the InputStream.read documentation. That way, you know the purpose of each and every statement, you know exactly what to expect from that low level function, and you can stack the higher level code on top of known functionality.

So as homework, I'd suggest you write your own "read data into buffer" loop, using only the description of InputStream.read. Afterwards, compare notes, where does your implementation work differently from the $random website one? My guess is you'll do the "-1" case differently, in particular, after receiving all data that you need.

The second thing you must do is to stop thinking in server-only and client-only. It seems to me you're coding a server, and you;re coding a client, then run them both and hope it works. In networking, failure and blocking can have 3 causes. A) Server code can be wrong, B) Client code can be wrong, C) Either side can get blocked or die or whatever, due to things that the other side does or does not do. The 3rd item is what is causing your blocking.

To get a handle on this, you can draw the interactions between the server and the client with a message sequence chart

https://en.wikipedia.org/wiki/Message_Sequence_Chart

That page has a lot more stuff than you need though. Basically, you draw two vertical lines next to each other, one is labeled "server" and the other is labeleled "client". Time increases vertically down. With arrows from one line to the other, you can show what messages (what data) you send from one end to the other. I'd start after accepting the connection, and up to and including closing the connection (since it's a relevant event that you send by closing the socket, and that you receive with the -1 return value from read).

You'll end up with some horizontal arrows from the client to the server (server receiving length and the file), and then some arrows from client to server (client receiving length and the file), and then 'close connection' arrows at the end. Technically, the arrows are not horizontal (network transfer takes time), but you're not interested in precise timing, just order of messages being exchanged.

If all is well, at this point, you understand that the "-1" case in the google code waits for closing the connection. Relate that to the diagram, and you'll find how things block indefinitely.

So with this way, will I have to write a little function -> read_byte()?

It's the elementary InputStream.read function, except for the closed-connection case. I had hoped you'd understand that, but I guessed wrong.

Note this is also $random website code, which above I said you shouldn't use without understanding. Analyze it, and understand how and why it works. Then rebuild it. Know X, instead of guessing what X might be.

Keep in mind nothing in networking is going to be "real time" really. You're basically sending and receiving packages through the mail and are responsible for making sure you send and receive all the pieces and also handle all the code related to waiting for them.

As an example if your server is sending a file to a client then the client is going to have to communicate with the server so that both parties know they are in the state of about to transfer a file, then you'd have to client sit there and wait(a specified maximum amount of time usually) for the pieces of the file to be received.

Usually you'll want to code with the mindset of worst case scenario. You might get a really tiny file in 10k pieces, it might take multiple seconds to transfer, the other end could disconnect partway through sending. Network code is very error prone without a lot of safeguards simply because there are so many things that can go wrong.

Moving this to Networking. Please don't post in other forums just because one is quiet; it's not meant to be a real-time chat system but a categorised forum.