Need help with this voltage regulator circuit simulation

Duke,
by "an IC missing", do you mean an InitialConditoin (.IC statement)?

 

Duke: The vreg.zip contains the symbol (TL431.asy) for the IC. If you put the ASY-file in the same directory as the ASC-file, LTSPICE finds the symbol and displays it:

You also put TL431.lib in the same directory. LTSPICE finds the files and simulates.

The simulation runs in no time at all on my PC. But there's no regulation at all. Vout follows Vin. I think that's no wonder because the TL431 is imho operated completely wrong. The ref-input receives a voltage proportional too V1 (divided by R3, R4). Therefore the voltage at the cathode of U2 follows the input voltage instead of being kept steady. Since this voltage controls the gate of M1, the output can't but follow V1.

To the OP: Have a loook at the TL431 datasheet . Figure 27 on page 30 shows how to use the TL431 plus a bipolar Transistor as a linear regulator. I think a bipolar transistor is the better choice over a Mosfet here.

 

So now your circuit is no longer a voltage regulator but an overvoltage protection?

Why do you need an overvoltage protector when you already have a regulated output? Anyway. Have a look at this site. Circuit "c" is an overvoltage protection. It operates by triggering the triac in case of an overvoltage. The triac will short circuit the power supply, blowing the fuse and protecting the rest of the circuit.

If you want a non-destructive protection, you can try figure 12 from this datsheet. Set the shunt regulator's output voltage a bit above the nominal regulated input voltage. The shunt regulator will be inactive because Vin=Vout < Vset.
If Vin rises above Vset, the shunt transistor will become active, bypassing current from Vout to ground. This in turn should cause the current limiter in the first voltage regulator to reduce the output voltage in an azzempt to reduce the current. Without a fuse, however, this fault condition can be active for a long time, causing both the bypass transistor and the voltage regulator to become hot and eventually to become defect.

 

I added a load (500mA) and the circuit works fine with an output between 9.8 and 11 V.
Show the complete schematic including load. Do not check "step the load current source" in the simulation commands menu.

 

The circuit will work, but it is less than ideal.
The average input voltage of U1 is 17.5V. The output voltage being 12V meand that 5.5V are lost across U1. Together with the current fo 2A this makes 2A*5.5V=11W power lost across the regulator - lost and converted to heat. You will need a good heatsink.
It is better to reduce the input voltage. The 1085 operates down to 1V ionput-output differential voltage. Therefore 13V at the input is enough. Add some margin and design the transformer+rectifier for 14V input voltage to the 1085. The power lost in the regulator then is only ~4W.

You also don't need capacitors C1---C4.

 

You can use the circuit in this way, but you'll have to live with the heavy losses of the 5V regulators (~12.5V*Iload per regulator). And you still haev C1...C4 in place. Remove them.

Consider using a PC power supply. You can find rather inexpensice ones or maybe you can get hold of a used one? These supplies are switched mode supplies which means they have considerably smaller losses than a linear regulator.