Yep, that's probably a safe assumption.
Note that from the original 6V, 1.25 will be dropped across a resistor, about another 2V across the LM317, leaving a maximum of 2.75V for the load. If your LED required more than 2.75V then it's not going to operate at the required current.
Two of these batteries in series would work, but note that the efficiency of linear current sources can be very low.
Did I not point you toward some switchmode LED drivers that would be a better choice? Ah yes, I did, but
these would work better with a 6V battery.