It is also worth checking if you have wires the switch correctly.
Depending on how the switch connections are configured, the relay may be always on, never on, or turn on and off with the switch. Because your talking to us, we know it's not the third one.
A bit of explanation about the switch. Logically you're looking for a 2 terminal device that has no connection normally, but had a connection when you press the button.
However, that switch has 4 terminals. It is possible that you have chosen the wrong two. In the orientation shown on your image, you have it wired assuming that the left two terminals are only connected when the button is pressed.
One common configuration for this type of switch is that the two terminals on the left are always connected, as are the two on the right. In this case you need to have the wires on the left and right sides. You could try that and see if the problem is resolved.
If the problem remains unresolved, use your wiring as a switch. Disconnect one end of one wire from the switch and note which LED turns on with the power applied. Then reconnect that wire, but to the left of the existing wire. When power is applied the other LED should turn on. If this works then you have the switch wired incorrectly, or maybe the switch is broken.
If you still haven't seen both LEDs on, then you have either wired the relay incorrectly, the voltage to the relay is insufficient to switch it, or maybe the relay is broken.
I realise a text description may be hard to follow, but I can't draw you anything right now.