Can you monitor traffic on the interface which is connected to your server and see if you see any incoming TCP connection request from the server?
Kindly monitor the interface with both configuration type to see the difference.
One with allow subnet configuration and another with explicit neighbor configuration (Without Passive and with passive)