Yes, it is how the physical hardware works, and how it has worked for several decades.
It does not need to be DirectX, but the days of the display being a block of CPU available memory ended back around 1987. EGA displays were the last to do it, VGA displays emulated it for compatability. It has been about giving instructions to the card over the bus ever since.
I still recall $B0000 and $B8000 as the old addresses, and on VGA using a hardware mapped address at $A0000, but it was an eternity ago in computer time, back in 16-bit systems and memory pages rather than fully addressable flat memory models. Those days are long gone for PC development, good riddance.