As already mentioned, an LED without a current limiting resistor is not a suitable load for the charger circuit.
You really need a lower resistance trimpot, but the 100k one might work. You might disconnect the trimpot from the circuit, and see if you can set it to values around 2k to 3k ohms. Sometimes pots jump from some small value to zero, and small for a 100k pot may be higher than the desired set resistance.
You can put a resistor network around the 100k pot to give a useful adjustment range. You might give us a list of resistors that you have and we can suggest what to use.
The trimpot adjusts the maximum output voltage at a moderate charging current. It does not adjust the current limit. The current limit is fixed, and determined by R1 and the characteristics of Q1. ( Down to some non-zero output voltage. Pull the output too low, and the current limit goes up to whatever the LM317 will provide.)
The symptom of poor voltage regulation between the no-load and loaded conditions may be the result of providing too large a bleeder resistance on the LM317 output. This may not really matter in the application, as a few mA of "float" current may not harm the battery.
See the "Minimum Load Current" spec on the data sheet. This current, plus a bit more, is usually absorbed by a bleeder resistance that also forms a voltage divider for the Adjustment Pin of the LM317.
The 470 ohm value of R3, and thus of R2 and R4, may be too large for the LM317 to regulate with no external load. The LM317 idle current may be as high as 5mA. The data sheet application circuits all suggest a 240 ohm value for R3. There is a reason that this value is not higher. The LM317 idle current varies from one unit to another, with temperature, and possibly the phase of the moon.
It is possible that when you adjust the output with no load, you are adjusting the load resistance to give the correct output voltage based on the LM317 idle output current and the total resistance of R2 + R3 + R4. At this point, the voltage across R3 can be much higher than the 1.25V Reference Voltage of the LM317.
Increasing the external load current will then pull down the output voltage until the LM317 begins to regulate, when the load current finally exceeds the idle current. At this voltage, the output becomes much stiffer, so the output voltage drops very little with further increases in load current.
The answer may be to just adjust the output voltage with some load, say a 1k resistor.
As already suggested, you should add a series resistor to the base of Q1. Even 100 ohms might save the transistor.
If you connect a fully discharged battery, the current limit should function, and the LED, D1, should light, if perhaps dimly. There should be only about 2.6mA to drive this LED. And if the charger output is pulled too low, the LED may not light, and the current limit will be higher, as well. Possibly over 3A, until the LM317 warms-up.
This charger circuit will drain the battery if the power source fails. Often a diode is used to isolate the battery, with the regulator output voltage increased to compensate for the diode drop.
I hope that some of this makes sense to you. I'm sure you can make this work. Just keep on asking questions.
Ted