Try to abstain from such extremes as "use C++ without pointers". You don't want to use C++ without integers either, do you.
Any kind of extreme is bad. There is no single undisputably correct "always do X" rule. C++ can do without raw pointers to a great extent, and in many cases this is preferrable (it's safer, and sometimes more idiomatic). On the other hand, there is nothing inherently wrong with pointers, and in many situations they are just the right thing.
Similar is true for malloc/free vs new/delete. Generally, you want new/delete because that is the "correct" thing which properly calls constructors and destructors. Then again, you preferrably don't want to use new/delete directly in modern C++, but rather something like e.g. make_unique. Usually, but not necessarily. I'm still using new and delete all the time. Yes, some people will frown at you, but let them frown. Try to make your code reasonable, not a religion. When it's reasonable (and safe) to use a raw pointer, well, then do it.
On malloc, again, usually you don't want to use it. But... sometimes you just want a block of memory where you'll write some stuff into (or where the operating system will place some string or a bunch of structures, or whatever), and you don't care about constructors and destructors being called (since they're trivial, or since there simply is no such thing for a raw block of memory). In that case, it's mighty fine to just use malloc instead (on my compiler, GCC 6.2, this makes a difference of 50kB in executable size!). It's just not what you want to do by default. You do it when you deem it justified.
Also, please note that "I realize references are pointers" is a misconception. You should not think of them that way. Although references may in some situations be implemented using what's effectively a pointer, they are by no means pointers. A reference is an alias, i.e. the same identical variable under a different name, and it has much stricter rules than a pointer. It must be properly initialized (with a valid object) and it cannot be changed to refer to something different later. In my experience, compilers are much, much better at optimizing function calls with references than with pointers. Most of the time, calling a function that takes a reference to some local variable generates exactly the same code as if you copied the function body's code into the place where you made the call, and text-renamed the variables. Except it's much more readable and better structured, which is a really good thing. Maybe one day, it will be that way with pointers, too. But in my experience, we're not there yet.
It is a bit difficult to create a null reference (intentionally at least... unintentionally it's impossible!) or an invalid reference, and it is hard to have a dangling reference too (actually... with lambda captures that is no longer true, it's pretty easy to have dangling references with these...). So, all in all references are always "better", right? They're not, because sometimes you just need what a pointer can do, and what a reference can't do. Again, no extremes. Use what's right for the situation, not what some rule tells you.