Hi, I have a terrain engine that is using a lot of textures, including texture arrays and indexing. I'm using currently full size RAW textures in order to index into texture arrays. RAW textures are relatively large in size as they are full size data. My question is as follows:

Let's talk about 8 bit indexing, so 0-255 only. And let's assume I need as large size as possible so I'm using 16384 size textures:

RAW file in size 16384^2 8-bit depth = 256MB

DXT1 file in size 16384^2 8-bit per channel on 3 channels without mipmapping = 128MB

The cost of course is accuracy, understood.

But, assuming I don't need all the 0-255 range but I can get along with say 50 indices or so. Would it be fine to use DXT1 to get an index?

What I mean specifically is that let's say I need to be able to specify 50 different indices, would it be OK to encode the values into DDS DXT1 texture as 0, 5, 10, 15, 20 etc etc until it'll get to 220.

And in shader, sample that texture and translate the value to an accurate index. Can I do that? The motivation is of course memory size, with 128MB I can create 3 "index textures" instead of 768 for the same with RAW accurate data.

I guess I don't fully understand what you are trying to achieve...if all you are after is simple integral index..why would you need more than a single channel texture ? Are you use separate indices for array textures ? If you are using indices, then the associated geometry must have a way to specify a index into this 'index buffer' which is just another level of indirection. If this is the case why not just store the actual index that you are trying to look up instead? All in all DXT is lossy so chances are you will not get the same index back that was written into the texture prior to compression.

It's worth looking into how BC1 (AKA DXT1) works -- for each block of 4x4 pixels, you get to define two colors in RGB 565 (red/blue is 0-31, green is 0-63). Let's call those two colors A and B. You then get two more colors:
 C = ⅓A + ⅔B,  and
 D = ⅔A + ⅓B.
That gives you a palette of 4 colors that can be used within that block of pixel. Then, for each of the 16 pixels in that block, you get to specify an index into that palette (2 bits per pixel).

So for each group of 4x4 pixels, you can definitely have 2 unique different indices, and might also be able to represent 2 more if you're lucky/careful.
If you only needed 4 indices, you could reliably use this to have 4 different representable index values at all times... or you could represent between 2-4 different index values (out of a total set of 256) per 4x4 pixel area.

You're probably be more interested in BC4 - this format only stores a single channel, like your original data. Again, it operates on 4x4 pixel blocks. You get to pick two 8-bit values per block (0-255). Let's again call those two values A and B. You then get to pick from two schemes to define a palette with 8 entries in it.
Per block you can pick the values of A/B and to use this palette:
A, B, (1/7)A+(6/7)B, (2/7)A (5/7)B, (3/7)A+(4/7)B, (4/7)A+(3/7)B, (5/7)A+(2/7)B, (6/7)A+(1/7)B.
Or this palette:
A, B, 0, 255, ⅘A+⅕B, ⅗A+⅖B, ⅖A+⅗B, ⅕A+⅘B.
Then, each pixel in the block stores a 3-bit index into the chosen palette.

If you only needed 8 indices total, you could reliably use this to have 8 different representable index values at all times... or you could represent between 2-8 different index values (out of a total set of 256) per 4x4 pixel area.

Alternatively, you could just store more than one value per pixel in your texture. e.g. instead of 16384x16384 with an 8 bit value per pixel, you could make your texture be 16384x8192 with two 4bit values per pixel (stored as a single 8bit value), or 8192x8192 with four 2bit value per pixel (stored as a single 8bit value). If you bind your texture to your shader as an integer format (instead of UNORM) then you can read out the 8 bit integer directly and then extract the individual sub-values (e.g. int a = t&0xf, b = (t>>4)&0xf;)

Thanx a lot Hodgman!! I'll dive into those details to find what I need!