Switch for multiple serial signal input sources

Hi, I'm working on a low budget project.
I'm kind of stuck and will appreciate your help and ideas.
I have a controller which supposed to take a serial a-synch Digital signal from an input source (such as card reader), check if the card is authorized then send the OK signal if it is authorized.

Now I have 4 readers working with the same controller and would like to set up a switching mechanism, where the first reader that send the serialized data will take the channel to the controller until it is done, and in case there is a collision (two readers hit the controller at the same time) then they can both be rejected and the users have to re-try and resend the data again. In case the first input sourrce beats the 2nd input source to the controller, then the 1st signal will be processed while the 2nd signal will be rejected and must be resent.

This is a low budget project so expensive microcontrollers will not do.

Thanks much