Thanks for the comments !
A "push pin" fastener has two parts in my mind. The Helm manuals calls it a push pin, but it's not. The second head just give you something easy to get a hold of. Hence I just had to pull harder. I used an flat blade offset screwdriver, which work fairly well.