Generally you assume that you have no error (or at least very very little error) in the x-position, and larger error in the y-direction. This means that its super unlikely a point will be to the left/right of a point on your model. It’s much more likely to be directly above/below, so you fit based off of the y-error.
For cases where you actually do have uncertainty in the x position as well, there exist other fitting methods, like Orthogonal Distance Regression that do more like what you’re asking about.
Latest Answers