Advertisement

How to unblock a server wait (in code) without stopping the server run

Started by April 09, 2017 03:52 AM
15 comments, last by hplus0603 7 years, 7 months ago

A few things that I'm not clear about. The first is serious, the second is a mystery and the third is an ideal state that i would like my code to run at

1. Read() function

public int read(byte[] b, int off, int len) throws IOException

  • This method blocks until input data is available, end of file is detected, or an exception is thrown.


Read() can block for any reason, it might be because faults in server code, user send no data at all or sends corrupted data.

If server code is well written and clear any bugs, the other two could still cause a block. And if you have many players chances are that someone somewhere will inadvertently do something stupid every so often that would cause a server block. I called this one serious (at the first line) because anytime there is a block, I've had to restart server. But it is meant to run continuously. And if the read block assumes data is still coming so it waits and blocks forever, disrupting and blocking other players.

The do-while/read() code is below, what check (code) can I add to the code that would timeout the block, without causing an exception and without needing to restart the server.


          bytesRead = is.read(mybytearray,0,mybytearray.length);
           current = bytesRead;
   
          do {
             bytesRead =  is.read(mybytearray, current, (mybytearray.length-current));
             if(bytesRead >= 0) current += bytesRead;
        } while(current < Integer.valueOf(number));         
 

2. second issue is a bit of a mystery.

In a thread, the code is supposed to run sequentially. But a line not reach can cause a issue at an earlier line. Again this happened specifically with blocking. Not closing an object in a latter code line (not reached yet) can cause a block in an earlier read() line (more often in server code)

3. My last post in the previous thread I mentioned all was now working fine. Well that was until I decided to send and receive jpeg file integers and text file with data continuously between client and server without closing either. (One cycle here means a client->server->client data transfer). It runs guaranteed in the first cycle. By the third cycle (sometimes second cycle) the application is forced stopped at the end of the run.

I don't know what causes it. The forced stopped produced no crash dump. If it did I don't know where the exception is dumped (definitely not on the console or logcat), so I can't find clues as to what the cause is. But I do need to fix this

In trying to blindly fix this (in the client code), i'm guessing some objects had been closed in the wrong order or not closed at all. But all my attempts to tweak things so far has either resulted in an exception or a block. In fact this 3rd issue is the main reason I was aware of the second issue above

How do I solve this - as in call sequences, close order, object life-time freed?

I suppose the first and third issues are the core issues i need to solve. The second is like a 3b to 3

Any ideas how I can solve the first and third problems

Many Thanks for any help

server code


public class FileServer {                 
    public final static  String   FILE_RECEIVED ="C:\\rootFolder\\NEW_070075n.jpg";
   
  public static  void main(String[] args) throws IOException {
   
      int bytesRead;     
      int current = 0;   
     //===============================================
      FileInputStream fis = null;
      BufferedInputStream bis = null;
      OutputStream os = null;           
      ServerSocket servsock = null;
      Socket sock = null;            
      //==============================================
      InetAddress IP=InetAddress.getLocalHost();  
      servsock = new ServerSocket(57925);      
      System.out.println("IP "+IP.getHostAddress()+"  ***%% :"+servsock.getLocalPort());  
                 
      while (true) {  
          System.out.println("555_999 Waiting...");           
          sock = servsock.accept();
          System.out.println("Accepted connection : " + sock);
                           
          
          InputStream is = sock.getInputStream();
          FileOutputStream fos = new FileOutputStream(FILE_RECEIVED); 
          BufferedOutputStream bos = new BufferedOutputStream(fos);
 //=================== read integer from client ==========
          InputStreamReader isr = new InputStreamReader(is);
          BufferedReader br = new BufferedReader(isr);
          String number = br.readLine();
          System.out.println("integer received from client is "+String.valueOf(number));
           byte [] mybytearray  = new byte [Integer.valueOf(number)];
 //=======================================================         
          bytesRead = is.read(mybytearray,0,mybytearray.length);
           current = bytesRead;
          System.out.println("1  bytesRead  "+bytesRead+"  mybytearray.length  "+mybytearray.length);
         
          do {
             bytesRead =  is.read(mybytearray, current, (mybytearray.length-current));
             if(bytesRead >= 0) current += bytesRead;
          } while(current < Integer.valueOf(number));         

	   
          bos.write(mybytearray, 0 , current);
          bos.flush();
          bos.close();
// ======================== write to-be-rendered data to text file ====================== 
                          
          File pathPlusfile = new File("C:/rootFolder/renderData4.txt");
	  appendToFile( pathPlusfile, "Hello, home joel888", 20999  );
                              
//======================== preparing to Send Data to Client  ==============================              
/**/  //==================   Send Data in text file to Client  ============================================              
          // send file   
//          File myFile = new File (FILE_TO_SEND);     
//          mybytearray  = new byte [(int)myFile.length()];
//          fis = new FileInputStream(myFile);
//========================= send jpeg image file to client ===============================		  
          mybytearray  = new byte [(int)pathPlusfile.length()];
          fis = new FileInputStream(pathPlusfile);
          bis = new BufferedInputStream(fis);
          bis.read(mybytearray,0,mybytearray.length);
          os = sock.getOutputStream();
//=========================== send integer to client ===============                   
          OutputStreamWriter osw = new OutputStreamWriter(os);
          BufferedWriter bw = new BufferedWriter(osw);
          number = Integer.toString(mybytearray.length);
          String sendMessage = number + "\n";
          bw.write(sendMessage);
          bw.flush();   
//========================== send file to client ===================                
          os.write(mybytearray,0,mybytearray.length);
          os.flush();
          System.out.println("number  "+number);
          System.out.println("Done.");
          bis.close();
          fis.close();  
          os.close();
          fos.close();        
          is.close(); 

          closeFile(  );               
//           servsock.close();
//           sock.close(); 
     }
  }
  BufferedReader bufferedReader = null;
  String stringObjectData = "";
  public int  numFromFile = 0;

     static BufferedWriter  bufferedWriter = null;
    public static  void appendToFile( File myPathPlusFile, String S, int num  ){
 	   
  	   try{
  		  bufferedWriter = new BufferedWriter(new FileWriter(myPathPlusFile, true));
//  		  bufferedWriter.write( S );   
//  	      bufferedWriter.newLine();
  		  bufferedWriter.append( S ); 
  		  bufferedWriter.append( "     " ); 
  		  bufferedWriter.append( Integer.toString(num) ); 
  	      bufferedWriter.newLine();
  	      bufferedWriter.flush();
  	   }
  	   catch (IOException e){
  	      e.printStackTrace();
  	   }
    	//  closeFile(  );
     }
    
	 public static  void closeFile(  ){
 	   try{
 //	      bufferedWriter.flush();
 	      bufferedWriter.close();
 	   }
 	   catch (IOException e)
 	   {
 	      // TODO Auto-generated catch block
 	      e.printStackTrace();
 	   }
	}
}


Client code


public class FSendfileActivity extends Activity {
    private static final int SELECT_PICTURE = 1;
    private Socket sock;
    private String serverIP = "192.168.1.4";    
    private String selectedImagePath;
    private ImageView img;
    final static String qcd = "qcd";
    String ImageDir2Client;      
    FileOutputStream fos = null;
    BufferedOutputStream bos = null;
//====================
    public  static String   FILE_TO_RECEIVED=null;
    int bytesRead = -1;
    int current = 0;                 
    
    public final static int FILE_SIZE = 2373620;
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.fsendfile);    

	    ImageDir2Client = Environment.getExternalStorageDirectory().getAbsolutePath();                    
	    FILE_TO_RECEIVED = ImageDir2Client + "/Server root/fromServer000019eee.txt";   
	   

        img  = (ImageView) findViewById(R.id.ivPic);  
         
       ((Button) findViewById(R.id.bBrowse)).setOnClickListener(new OnClickListener() {
           public void onClick(View arg0) {
              
                Intent intent = new Intent();
                intent.setType("image/*");
                intent.setAction(Intent.ACTION_GET_CONTENT);
                startActivityForResult( Intent.createChooser( intent, "Select Picture" ), SELECT_PICTURE );
            }
       });
              
       Button send = (Button) findViewById(R.id.bSend);
       final TextView status = (TextView) findViewById(R.id.tvStatus);
       
       send.setOnClickListener(new View.OnClickListener() {
           @Override                     
           public void onClick(View arg0) {
           	new Thread(new Runnable() {                  
                   @Override        
                   public void run() {            
                       try {                                     
                       	sock = new Socket();     
                       	connection(sock,  serverIP, 57925);
                       	
                            File myFile = new File (selectedImagePath); 
                            byte [] mybytearray  = new byte [(int)myFile.length()];
                            FileInputStream fis = new FileInputStream(myFile);
                            BufferedInputStream bis = new BufferedInputStream(fis);
                            bis.read(mybytearray,0,mybytearray.length);
                            OutputStream os = sock.getOutputStream();
            //=========================== send integer to server ===============                   
                            OutputStreamWriter osw = new OutputStreamWriter(os);
                            BufferedWriter bw = new BufferedWriter(osw);
                            String number = Integer.toString(mybytearray.length);
                            String sendMessage = number + "\n";
                          
                            bw.write(sendMessage);   // send size integer here
                            bw.flush(); 
            //==================================================================                
                            os.write(mybytearray,0,mybytearray.length); // send file
                            os.flush();  
                                          
    //=================  client receiving data ==============================
 /* */            
                        // receive file
                        mybytearray  = new byte [FILE_SIZE];
                        InputStream is = sock.getInputStream();
                       fos = new FileOutputStream(FILE_TO_RECEIVED);
                        bos = new BufferedOutputStream(fos);
        //=================== read integer from client ==========
        
                        InputStreamReader isr = new InputStreamReader(is);
                       BufferedReader br = new BufferedReader(isr);
                        number = br.readLine();
        //=======================================================         
                        bytesRead = is.read(mybytearray,0,mybytearray.length);
                       current = bytesRead;

                        do {
                           bytesRead = is.read(mybytearray, current, (mybytearray.length-current));
                           if(bytesRead >= 0) current += bytesRead;
                        } while(current < Integer.valueOf(number));
                                 
                        bos.write(mybytearray, 0 , current);
                        bos.flush();
                                                             
                        br.close();  
                        bis.close();       
              //***          bw.close();
                        fos.close();          //STILL EXPERIMENTING WHICH ORDER/MISSING CLOSE()
              //***          bos.close();     //WOULD FIX THIRD PROBLEM MENTIONED IN ORIGINAL POST
              //*          isr.close();
              //          is.close();
              //          os.close();
                        osw.close();
                        fis.close();
   /* */                sock.close();
                    } catch (UnknownHostException e) {
                        e.printStackTrace();
                    } catch (IOException e) {
                        e.printStackTrace();
                    }  finally{
                        try{
         //                   sock.close();
                        }
                        catch(Exception e){
                            e.printStackTrace();
                        }
                    } 
            	}
	          }).start();
	        }
	    });
        
    }       
    public static void connection(Socket s,  String serverIP, int port) {
        try {   
        	Log.v(qcd, " before connecting ****...");
        	s.connect(new InetSocketAddress(serverIP, port), 120000);
        	Log.v(qcd, " socket connection DONE!! ");
        } catch (UnknownHostException e) {
            e.printStackTrace();
            Log.v(qcd, " Unknown host..."+e);
        } catch (IOException e) {
            e.printStackTrace();
            Log.v(qcd, " Failed to connect...   "+e);
        }
    }
    public void onActivityResult(int requestCode, int resultCode, Intent data) {
        if (resultCode == RESULT_OK) {
            if (requestCode == SELECT_PICTURE) {
            	img.setImageURI(null);
                Uri selectedImageUri = data.getData();
                selectedImagePath = getPath(selectedImageUri);
                TextView path = (TextView) findViewById(R.id.tvPath);
                path.setText("Image Path : " + selectedImagePath);
                img.setImageURI(selectedImageUri);
                Log.v(qcd, selectedImagePath);
            }     
        }
    }

    public String getPath(Uri uri) {
        String[] projection = { MediaStore.Images.Media.DATA };
        Cursor cursor = managedQuery(uri, projection, null, null, null);
        int column_index = cursor
                .getColumnIndexOrThrow(MediaStore.Images.Media.DATA);
        cursor.moveToFirst();
        return cursor.getString(column_index);
    } 
}



can't help being grumpy...

Just need to let some steam out, so my head doesn't explode...


The do-while/read() code is below, what check (code) can I add to the code that would timeout the block, without causing an exception and without needing to restart the server.

You can use non-blocking IO. Java has another API for this, "NIO". Google for "java non-blocking server" for examples and explanations.

You may be able to get away with configuring the socket with a timeout, Google for "java socket SO_TIMEOUT".

I think the NIO API above was introduced to solve this problem without threading or setting timeouts on individuals sockets, though it is more complex again.

In a thread, the code is supposed to run sequentially. But a line not reach can cause a issue at an earlier line. Again this happened specifically with blocking. Not closing an object in a latter code line (not reached yet) can cause a block in an earlier read() line (more often in server code)

Threads can have strange behaviour if you don't know what you are doing, in particular the order of operations visible across threads. Without going into too much detail, to make single threaded code fast, the compiler and CPU are allowed to do lots of surprising things, such as execute the code in a different order than you specified. Both are constrained in that the result of such an execution must not be distringuishable from what would have run if you executed each statement in the order you specified. This optimization breaks down if one thread looks at work that another thread is doing, it can observe these unexpected orderings of actions. Since nobody wants to lose the single threaded optimisations, the solution is that CPU and compilers have special features (memory barriers, sychronisation primitives) which allow you to control how threads view each other's actions.

Long story short, multi-threading is hard. I'm not sure this is 100% related to the issue you describe, but I think trying to mix it in when you're struggling with sockets seems a bad idea

My last post in the previous thread I mentioned all was now working fine. Well that was until I decided to send and receive jpeg file integers and text file with data continuously between client and server without closing either. It runs guaranteed. By the third cycle (sometimes second cycle) the application is forced stopped at the end of the run.

Which "Application", the client or the server? Is there no error message at all? What about the process exit status?

Can you reproduce this behaviour running in a debugger? If all else fails, you could add print statements to try help find the code that is running just before the crash.

Advertisement

Just in general, I think you should look into extracting more helper functions in your code, ones that do a simple thing well.

Here is an example of what that might look like with your current code:


public class FileServer {                 
 
    public static  void main(String[] args) throws IOException {
        ServerSocket server = new ServerSocket(57925);
        System.out.println("Server at " + InetAddress.getLocalHost().getHostAddress() + " is listening on port " + server.getLocalPort()); 
        while (true) {
            System.out.println("Waiting for connection...");
            try (Socket client = server.accept()) {
                System.out.println("Client at " + client);
                processConnection(client);
            } catch (IOException e) {
                System.out.println("An error occurred processing client " + client);
                e.printStackTrace();
            }
        } 
    }
 
    private static void processConnection(Socket clientSocket) throws IOException {
        byte [] clientData = readFromClient(clientSocket);
        writeToFile(someFilename, clietnData);
        byte [] serverData = anyOtherServerSideProcessing(clientData);
        writeToClient(clientSocket, data);
    }
 
    private static byte [] readFromClient(Socket clientSocket) throws IOException {
        try (InputStream fromClient = clientSocket.getInputStream()) {
            int size = readInteger(fromClient);
            return readBytes(fromClient, size);
        }
    }
 
    private static void writeToClient(Socket clientSocket, byte [] data) throws IOException {
        try (OutputStream toClient = clientSocket.getOutputStream()) {
            sendInteger(toClient, data.length);
            sendBytes(toClient, data);
        }
    }
 
    private static void writeToFile(String filename, byte [] data) throws IOException {
        try (FileOutputStream fos = new FileOutputStream(filename)) {
            fos.write(data, 0, data.length);
        }
    }
 
}

You can use non-blocking IO. Java has another API for this, "NIO". Google for "java non-blocking server" for examples and explanations. You may be able to get away with configuring the socket with a timeout, Google for "java socket SO_TIMEOUT". I think the NIO API above was introduced to solve this problem without threading or setting timeouts on individuals sockets, though it is more complex again.

Is this similar to the second part of this SO answer? I saw it earlier on but didn't know how to integrate it into my code

Which "Application", the client or the server? Is there no error message at all?

Sorry i should have mentioned in original post. Its the client application that was forced stopped on the Android handset. Which is why I had concentrated my diagnosis on the client code

What about the process exit status? Can you reproduce this behaviour running in a debugger?

I couldn't reproduce in the debugger, though It runs to the end, frustratingly no exception dumb. Which makes it more difficult to know what the cause is

If all else fails, you could add print statements to try help find the code that is running just before the crash.

Hmm... i did put lots of prints statements in the code, but its just occurring to me now I could improve that a little bit more, because on the second or third cycle of run the text file dumped to the client looses the data written to it.

[Just in case, i should mention that this is not quite the same previous as the problem in the previous thread. In the previous thread the data never gets written to the text file at all even in the first cycle]

Just in general, I think you should look into extracting more helper functions in your code, ones that do a simple thing well. Here is an example of what that might look like with your current code:

Agreed ... better code structure, would even makes me see better. Starting on that now. Many thanks

can't help being grumpy...

Just need to let some steam out, so my head doesn't explode...

The traditional way to make sure sockets don't block the server is to use the select() call to figure out which sockets are available to read, and then call recv() (or recvfrom()) on each of those sockets. recv() is guaranteed to not block and return the available data if select() says it's readable. This is why recv() may return less than the full buffer you request -- it's guaranteed to not block if there is at least one byte!

The original Java sockets API didn't give very good access to these system calls, therefore you have to use the "nio" classes to build properly non-blocking servers in Java.

As far as "a line not reach can cause a issue at an earlier line" this is simply not true. (Well, for certain compute/memory workloads, the compiler or even CPU may in fact re-order operations, but I doubt this is your problem.)

If you run the same function in more than one thread, then one thread may get to line N+1 when the second thread is at line N.
If you run the code in a loop, then the second time through the loop may of course get to line N after the first time of the loop gets to line N+1.
If you believe line N+1 is not reached, then run your code in a debugger, put a breakpoint on line N+1, and see whether it is reached.

In general, though, based on the questions you're asking, I do not recommend you write network code at this point. There are too many other things you are not yet sure of, that get in the way of debugging the networking-specific bits. Networking, in itself, is hard, because it deals with distributed systems and very carefully constructed system call semantics. However, to use those calls correctly, you need to understand all of the underlying assumptions and conventions. It's pretty clear that you currently do not, and do not yet have the abiltity to isolate separate pieces of code to investigate and debug how they perform. I recommend you work on single-user programs for a bit longer until you learn all the necessary underlying skills and methods that will help you later understand networking. Based on my experience, this will actually be the faster path to understanding and success for you.
enum Bool { True, False, FileNotFound };

When I re-read my first post I see I hadn't read it throu a second time as my construction had been appalling, good you and ripoff could still manage to see the question. Have edited it properly however

In general, though, based on the questions you're asking, I do not recommend you write network code at this point. There are too many other things you are not yet sure of, that get in the way of debugging the networking-specific bits. Networking, in itself, is hard, because it deals with distributed systems and very carefully constructed system call semantics. However, to use those calls correctly, you need to understand all of the underlying assumptions and conventions. It's pretty clear that you currently do not, and do not yet have the abiltity to isolate separate pieces of code to investigate and debug how they perform. I recommend you work on single-user programs for a bit longer until you learn all the necessary underlying skills and methods that will help you later understand networking. Based on my experience, this will actually be the faster path to understanding and success for you.

This is tough to swallow though I know you must be correct. Its Sad to accept because I though I was almost there. But when I begin to read 'use NIO classes to create non-blocking server', I knew I was in deep shit as this equivalent to starting all over again. And as you mentioned I reckon it would even be way tougher than what I've experienced so far.

This shock will be difficult to recover from but when I do (if I do at all) , I will have to re-design the whole game so as to drop to networking part of it.

I'm just still surprised though that based on its practical advantage NIO tcp examples don't flood the internet more than the simplistic and impractical type of examples that I ended up following

Sad but thanks for the info

can't help being grumpy...

Just need to let some steam out, so my head doesn't explode...

Advertisement

It's a tough pill to swallow, but we've all been there at some stage in our development careers I think. Just use an existing library, and get good at coding most other things and from that experience you will gain enough knowledge to get through this barrier.

"Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety." --Benjamin Franklin

Just use an existing library, and get good at coding most other things and from that experience you will gain enough knowledge to get through this barrier.

Could you suggest one - a proper free package with simple APIs and with a tutorial for beginners?

can't help being grumpy...

Just need to let some steam out, so my head doesn't explode...

I agree that you need to nail down more general Java programming concepts before proceeding, and to work on code quality to make it easier to find and understand bugs. Here's a short code review to get you started on the sort of things you should consider:

  • You don't initialise `int bytesRead` but you initialise everything else. Be consistent (and ideally initialise everything).
  • You create an InputStreamReader around the InputStream, and then a BufferedReader around that, but then you go back to reading from the InputStream directly later. I'm pretty sure you were warned not to do this in a previous thread, because some of your data can get 'lost' inside the BufferedReader.
  • What happens if bytesRead is less than zero? What should happen?
  • Your client thread has a 'finally' block that does absolutely nothing. However exceptions thrown from that thread are probably what crashes your Android app.
  • Create separate functions for coherent blocks of functionality. For some reason your server code has all the reading and writing code mashed inline in one big loop, but you've factored out 'closeFile' when it's essentially a one-line operation. A more natural division here would be one function to read the data, one to write the file, one to read the file, one to send it. Then you don't end up with atrocities like 12 close() calls in a row, because you wouldn't have all of those objects in one place.
  • All of the above is doubly important when you're trying to wedge all that into an inline thread definition. It's almost impossible to debug code like that. Instead of `onClick` creating this anonymous class, you should probably move all that into a new class in a new file.
  • Clean it up. Put in consistent indenting, remove unused code, remove unused comments.

I agree that you need to nail down more general Java programming concepts before proceeding, and to work on code quality to make it easier to find and understand bugs. Here's a short code review to get you started on the sort of things you should consider:

  • You don't initialise `int bytesRead` but you initialise everything else. Be consistent (and ideally initialise everything).
  • You create an InputStreamReader around the InputStream, and then a BufferedReader around that, but then you go back to reading from the InputStream directly later. I'm pretty sure you were warned not to do this in a previous thread, because some of your data can get 'lost' inside the BufferedReader.
  • What happens if bytesRead is less than zero? What should happen?
  • Your client thread has a 'finally' block that does absolutely nothing. However exceptions thrown from that thread are probably what crashes your Android app.
  • Create separate functions for coherent blocks of functionality. For some reason your server code has all the reading and writing code mashed inline in one big loop, but you've factored out 'closeFile' when it's essentially a one-line operation. A more natural division here would be one function to read the data, one to write the file, one to read the file, one to send it. Then you don't end up with atrocities like 12 close() calls in a row, because you wouldn't have all of those objects in one place.
  • All of the above is doubly important when you're trying to wedge all that into an inline thread definition. It's almost impossible to debug code like that. Instead of `onClick` creating this anonymous class, you should probably move all that into a new class in a new file.
  • Clean it up. Put in consistent indenting, remove unused code, remove unused comments.

Thanks. This restructuring will certainly help overall. Particularly in the new setup i'm thinking about (explained below)

After hplus advice my perspective and target is changing. As you will see from previous posts, my original aim for this particular thread was to make my code non-blocking (or block-proof). But since I've been told I've got to use NIO classes (which is even harder than what i've done already and worse still, to me- it's like starting all over again), I've been in a state in a state of reflection since (to put it mildly). I'm thinking now, that in the short term I will redesign my game to exclude network coding but instead setup a learning network coding framework on my Android devices

For learning purposes I need to test some stuff exclusively on Android devices. To this end I've got some few questions::

1. Is it possible to use IP 127.0.0.1 to run a client->server loopback on the same android device (not on emulator)?

2. Could both Client and Server run in the same application on same device? ( asking because most devices at the moment can run only one app at time. And since client and server can't run as separate app but have to run in the same application)

3. Could port numbers be used with mobile devices, if so, do I have to retrieve local host and port number for a mobile device the same as is done for PC server?

4. Unrelated to above three ques, and just for curiosity sake - Why is it that despite the practical advantage of NIO api for tcp, their examples are not so common on the internet? In fact every time I every time I've searched I've ended up with the simplistic and impractical type of examples that I ended up following

can't help being grumpy...

Just need to let some steam out, so my head doesn't explode...

This topic is closed to new replies.

Advertisement