Hello.

I have a big file that cannot be all loaded in memory but I want to read chunks of it on request. I have multiple requests (some of them "in the same time") of users that must load some data that I have stored for them. Users send their requests over UDP and I wondered if I should read/write the chunks of data in the file concurrently on some threads or Queue the requests and read/write them in sequence.

Thanks.

You didn't mention the platform.

Look up asynchronous I/O functions suitable for your system.

Asynchronous calls avoid the overhead of creating and managing threads, the OS can reorder them based on best physical access patterns (reading them as the heads travel across the disk, rather than in the order made), and can notify you the moment the data is available.

At this point you're dealing with many of the problems of a pretty full featured database application: multiple threads, data not fitting into memory, simultaneous reads and writes. You may want to give some serious thought to actually using a database rather than ad hoc file reads.

Remember to forget not to look into memory-mapped files.
Nothing much more to say beyond what has been said.


L. Spiro