I would bother only if I need cross-platform communication, or multi-platform support. i.e. bitpacking, byte streaming, things that gets passed through the network layer. Endian conversion would automatically convert to a set endianness, through the use a serialiser.

As a rule of thumb, run your data through a proper serialiser, so not packing structs as binary data, or bitfields, enums, which might be optimised differently, or have different width. Rather, serialise each component individually, and for bitstreams, taking advantage of compression (serialise a bool a s single bit rather than a byte, serialise long ints, enums within a fixed range, ect...).

As for optimisations, that's not really where the gains are made, although they can be in some edge cases (memory alignment rules, for particles, GPU commands, ect...).