That’s a truly hard problem.
On short distance where you can see landmarks you could use triangulation. But on long distance where you don’t see the land you estimate your relative position by measuring how fast you go and in which direction of current are not too strong. But you will need to regularly correct it using absolute positioning strategies: using stars and time.
With the position of the sun or of the stars, you can relatively measure the latitude, usually using a sextant. It does not depends on time because the Earth “vertical axis” stays mostly parallel to itself in time (a little bit of precession but it’s negligible).
With the time you can know how much Earth rotated on itself, and therefore deduce where the sun or stars should be depending on your longitude.
Now, the trick is to keep track of time reliably on a moving ship. That’s actually the main driving factor for the invention of pocket watches and pendulum-less clocks.
Latest Answers