Not really a problem, it's more about having a deep understanding of what you're doing. Decisions need to be based on that knowledge, and those fundamental assumptions are not part of the posts.
Smart pointers transmit not just the address of an object, but also the semantics of who owns the object. They are part of the design of the lifetime of objects.
For most tools and libraries, the library itself doesn't actually own the objects nor do they do anything with their lifetimes. Objects are created outside of the system, they are passed to the system for use, then at a later point objects are destroyed outside the system. In this scenario the tool or library interface should not use smart pointers. That doesn't mean the user's code shouldn't use it, only that it doesn't belong in the library's API.
For some tools and libraries, they DO own the objects, serving both as the source that generates them and the sink that destroys them. In this scenario it also usually isn't appropriate because the ownership of the object cannot change, the source is also the sink. The only potential problem is holding a reference after it has been destroyed, which usually indicates a software defect rather than the need for a ref-counted smart pointer in the design.
Typically it is a bad idea to mix memory across APIs or libraries. As a simple example, one might be using the OS's standard heap, another might be using a debug heap, and still another may be expecting customized heaps for memory pools. Libraries might accept an allocator object when they're constructed so memory can rely on the proper pool, but even that tends to be troublesome when they come from outside sources because every organization tends to have and use slightly different functionality.
As a result, when it comes to libraries the most universally consistent method is to only expose raw pointers or handles. Either they accept raw pointers which the code guarantees have a suitable lifetime, or the library itself manages the memory and exposes either pointers or handles to the outside world. Neither of those scenarios is a smart pointer. The ownership semantics that are the primary benefit of smart pointers don't exist in the scenario, in no case do the objects shift ownership.