Your brain compares the differences in what your ears hear in order to determine the direction a sound comes from. It doesn’t only compare which sound is louder but also which sound arrives sooner, and the differences of the frequencies of each sound. High or low frequencies will arrive differently to the further away ear for example because it is blocked by your head. The shape of your ears also alters the characteristics of arriving sound depending which direction it comes from.
Earbuds send sound straight into your head, bypassing all these differences, so your brain correctly determines that this would only happen from a sound that is completely centered, neither left or right or forward/back.
Your brain works to interpret where sound is coming from in 3D space based on several factors including, but not limited to, the delay in sound hitting one ear before the other. When you have equal sound coming from directly inside both ears your brain thinks the sound is being emitted from the center of your head.
You normally use both of your ears to locate the sound source, usually by turning the head and observing changes. Sound on one side of the head reaches the other ear with a delay, and it gets muffled by the shape of the ear when coming from behind. If it is beamed directly into one ear these mechanisms do not work and the brain is confused.
There are signal processors that adapt stereo steams meant for loudspeakers to headphones. One of the operations they do is mix a delayed copy of hard panned sounds into the opposite channel. This creates a more natural panning, but can’t be done by default for mono compatibility and encoding processes that downmix to mono.
Latest Answers