I've solved the drift. My DC offset resistors were too high. The ADC wants little to no impedance on it's input. I read this can be solved by using an op-amp, but reducing the resistors in the voltage divider was enough that after constant readings, I experience 1/1000th of a volt drop. So, that is perfectly acceptable.
With my measurements, I am also recording the time in ns. I will use this after the readings to calculate frequency using my own algorithm, and it will not slow down my readings at all.