Your hearing's sensitivity to loudness is logarithmic so you can hear a pin drop but also hear something that is extremely loud.
The linear potentiometer is not a logarithmic volume control. If you turn it down from max to half then the audio level will be reduced only a little, but a log volume control will turn down the audio level quite a lot. If you turn up the linear pot from minimum to just a little higher then the audio level will suddenly turn on fairly high, but a log volume control will turn on the sound level only a little.