Presumably the other end already caches the fact that the request was declined, to avoid having to decline a flood of persistent requests.
I think the only thing you can do is not decline the request, or wait until some kind of time-out.
You should make sure to cancel/destroy any previous invitations.
Perhaps turn WiFi off and on again on both devices might reset the decline-cache?
Note that the core WiFi code doesn't know what to do when the invitation is declined; check out the comment on line 1707:
https://android.googlesource.com/platform/frameworks/base/+/b267554/wifi/java/android/net/wifi/p2p/WifiP2pService.javaOverall, I think WiFi Direct is not a great technology. It might help with connectivity when two people want to share pictures and are in the middle of the jungle with no access point or cell phone tower nearby, but otherwise, using regular IP networking over regular WiFi is much better.