An article I came across suggested that we should pick the largest and smallest story in the prioritised backlog and relatively estimate the remaining stories. This makes sense. However, I was wondering how this works from sprint to sprint? For example, in the next sprint the largest story might be 5 points compared to the 8 point story in the previous sprint and similarly the smallest story might be 1 instead of the 3 from the last sprint. This seems like it can cause the velocity to fluctuate from sprint to sprint.
I feel like I'm missing something fundamental in the way to do this correctly. Looking for the experts to clarify.