However far away they are in light years, that’s when you’re seeing them. For example, a star 8 light years away looks to you how it did 8 years ago. Most stars you see in the night sky without a telescope are between 5 and 1000 light years away. Much further than that and they get too dim to see without a telescope. But if you see images of distant galaxies taken from Hubble, etc., you’re seeing what they looked like millions or billions of years ago, because they’re millions or billions of light years away.
You see *everything* as it was at a certain point in the past, because even though light is the fastest thing there is, it still has a finite speed. If you stood six feet away from me, you’d be seeing me as I was 0.000000006 seconds ago. You see the Moon as it was 1.5 seconds ago. When you look at the Sun, you’re seeing it as it was eight minutes ago.
And the nearest star, Alpha Centauri, is 4.4 light-years away, so you’re seeing it as it was 4.4 years ago. You asked if we were seeing stars from “millions of years ago,” but that would require that the stars would be millions of light-years away. None of the stars you can “see” in the sky are that far away; some of them are a dozen or so light-years away, some are hundreds, and some are a couple of thousand.
Anything further than that is too far away for you to see with the naked eye. The nearest large galaxy is 2 million light-years away, which means that we are indeed receiving light that was sent 2 million years ago, but it’s nearly impossible to see that galaxy with the naked eye. With the best telescopes we can see galaxies that are tens of *billions* of light-years away, and we are seeing them as they were near the beginning of the universe. But the “night sky” as you commonly see it doesn’t go back that far.
We’re seeing what they looked like when the light left the star. So, if a star is 50 light years away, it took the light 50 years to reach us and thus you’re seeing what it looked like 50 years ago. If it’s millions of light-years away, we’d be seeing what it looked like millions of years ago.
The sun (our sun) is approximately 8 light-minutes away. When you look at the sun, the light that hits your eye left the sun 8 minutes ago. When you see the top of the sun rise above the horizon, it was actually above the horizon 8 minutes earlier.
When people (WRT astronomy) talk about being able to look into the past, this is what they’re talking about. The further away a star is, the longer ago, and therefore, closer to when the Big Bang happened, the incoming image is from.
This is all going to be muddied when we add in relativity and gravitational pull on light and the expanding universe, but it’s just an ELI5 answer.
The night sky is a time machine. Because light travels at a fixed pace, the further away the object is, be it a star, a galaxy, or whatever, the *older* the photons are. This is wonderful because it means you can see into the past and even observe things that don’t exist anymore. The first generation of galaxies at the furthest reaches of the sky, as an example.
You can’t see everything, unfortunately. There are photons that’ve passed before we were even a species. We can’t see the Big Bang. We can, however, see the implications of it and observe the general development of the universe at different time periods.
Latest Answers