Hi,
Is there anything that could be blocking incoming packets to 224.0.0.5 on the 3850 interface.
I would suggest to debug ospf hello packets on the 3850
debug ip ospf [hello | adj]
Could you share the debug output.
From the MX monitor interface output, all the parameters seem to be matching [hello, dead, mask], router-id ok. IMO, the 3850 ospf wilcard does not need to be changed as the mask is same as seen in the monitor traffic.
Also, cisco nsf is enabled on the interface but would not expect this to be doing anything at this stage.
Cheers,
Ashvin