So in real life – fish and sharks are vertebrates (have spines) which evolved in the sea. Never evolved legs. Just like a snake slithering side to side the motion of the whole body made sense so the tails that move side to side – so vertical tails.
Sea mammals (whales, dolphins, walruses, otters etc….) come from walking land animals that went back to the sea for food. Ended up swimming so much they they re-evolved tails. Just like it is easier for you to kick up and down when you swim instead of side to side they evolved an up-down motion to move thru water – thus a vertical tail.
If your fantasy world has evolution, and the mermaids evolved from humans, then they should probably have sideways tails. However, the would also then not have scales, as their is no reason for mammals with hair to re-evolve scales (mammals already have great water-shedding hair, check out the otters).
If your fantasy world is where insane wizards combined a fish and human to magically make a mermaid -then the tail with be vertical, as that is how a fishes spine would line up with a humans when you patch them together.
Latest Answers