This topic originally in General Programming: http://www.gamedev.net/community/forums/topic.asp?topic_id=174744 Is there any way to reserve pages in linux without actually committing (eg using up memory or disk). Something equivalent to Windowses VirtualAlloc(#, #, MEM_RESERVE, #). I want to reserve large chunks of ram, preventing other program components like OpenGL''s glTexImage2D() and sbrk() from accessing these pages, because those specific virtual pages might be required by my code at a later date.

malloc. *grin* seriously.

but, why? what do you think this will gain you, at all?

Ok, the problem here is that I''m actually _writing_ a generic malloc. I''m trying to out-do LEA malloc, and my allocator seems to be less wasteful than LEA (mainly in heavily OO applications), although it''s about 1/2 as fast.

The important thing for my allocator is that I can ask for specific pages. (There is a lot of lee-way here, but the main requirement is that the blocks I ask for are aligned to large power-of-two boundaries.

An example:

I want to ask the OS to make sure no other program component attempts to use pages between 0x20100000 and 0x20200000. This means things such as dynamically loaded libraries, glTexImage2D and sbrk() will be forced to look for other virtual pages. The allocator may require any one of these pages at any time. When the allocator is done with that range, I want to ask the OS to no longer bother about keeping other program components away.

This is possible in windows, but I''m beginning to doubt that linux is as powerful in this area.

I''m having to resort to testing each page to see if something else is using it, and if it is, wasting many KB.

Anyone know a very fast way to ask if a virtual page is committed (eg in RAM or paged) with *nix?

Well, under linux, the VM does absolutely nothing about the fact that you allocated a pile of memory until you try to access it. (in which case, you hit a page fault, the VM sees that this is new memory and gives you a page of physical memory to use at that virtual address)

Now, as I understand it, what you''re trying to do is this:

Create a kind of memory pool with minimal memory impact. And you want to do this by informing the kernel which memory pages within this memory block can be outright discarded under VM pressure (Out of memory, for instance), and which will need to actually be swapped out, correct?

Interesting... very interesting... I''ve never seen this sort of thing before, I''m gonna go look into it. Curiosity, I suppose.

Actually, thinking about that, you might have just struck on a point I overlooked before, but may just work.

If I just mmap the entire area instead of reserving, ignore committing, and (munmap and immediately mmap) the area to effectively decommit the page.

There''s only two problems with this that I can see, and one is very very unlikely:

in between the munmap and the immediate mmap, another thread gains control and maps the pages. Highly unlikely, but it probably needs to be fixed if this malloc is going to be worthy of use.

The other, not so important problem is that access to pages that in the windows version would be reserved will not result in an error, rather, the memory will just be committed. But this isn''t really an issue as the program must be buggy in itself to cause this.

Now to see whether munmap''ing pages inside a mmap''ed region is legal (crosedb lfingerrs).

Sorry, don''t think it is legal. besides, you lose any performance gains you would ahve made by doing that.

I think I know exactly what kind of feature you''re looking for, even if you haven''t nailed it yet. ;-) I asked some smart guys about it, and I''m poking around some VM docs.

But in linux terms what you''re doing isn''t reserving a memory range, then allocating parts of it at a time like I assume you''re doing in windows.

What you''d be doing is allocating a large block, and then telling the kernel that the parts of it are no longer needed by the application, while still keeping the memory region. Now, the thing to realize is that this is going to be on page boundaries, which are all 4k under linux. (er, currently, sortof) I think you see that already, however.

Yeah, I''m definately using page boundaries and stuff.

I assume that you mean mmap and munmap are either slow or wasteful. I haven''t done any testing on them yet.

I''ve just got a very slow version working in linux because I''m having trouble detecting whether pages are in use (basically just trapping page faults).

BTW, thanks for all your help, CJ, much appreciated.

bingo, okay.

first, going along with my own thought train here. the method you use to free pages within the mmaped region is called madvise. google it or check man pages. MADV_WONTNEED is the flag you''d use, I believe.

also, in rereading some things. Are you just trying to be able to keep a specific memory range? and not intending this to be used for performance reasons? why?

If it is about just having a fast mempool type thing that''s memory efficient, then never mind.

The algorithm I wrote choses addresses for chunks (malloc''d blocks) within these regions such that after a long time, these regions are still wasting typically less than 2% of the space used. (Ie 98% of the bytes in the pages committed by the algorithm are actually part of the chunks).
But the algorithm doesn''t just pick the next page(s) in the region to use for malloc(), the chunk can actually turn up virtually anywhere in the "reserved" region, and the algorithm fully expects that page to be available.

Often the chunk span page boundaries, even if the chunk is much smaller than a page.

The need for reserving is something similar to the typical way that malloc uses a heap, and expects that the page just after the deepest chunk in the heap is still available. If it isn''t, those mallocs typically lose some space efficiency. But my version''s a lot more complex in that it expects pages quite a while after, and even sometimes before, to be available.

madvise looks to be pretty much exactly what I need, thanks again. Strange, I''m sure I looked at all the mman.h functions.

Using madvise with MADV_FREE looks more promising, as it looks like MADV_DONTNEED will still page the data to disk.

Dammit, a bug has crawled under the LCD display of my calculator and gotten squished, and there''s no way to remove it without destroying the display .

I think I''m gonna post another thread asking about querying page status under *nix.