It’s a muscle, working it out in a controlled fashion makes it stronger and more efficient for all the time you aren’t intentionally working it harder.
You are effectively making it’s work 90% of the time easier by working it harder 10% of the time, instead of making it harder 100% of the time.
Take professional athletes for example, they often have a resting heart rate anywhere from 40-60 beats per minute at rest. The average resting heart rate for a moderately or less-active person is usually around 60-100 depending on multiple factors. Take sleep for example, assuming 8 hours that means a normal persons heart will beat maybe 28,800 times during sleep (assuming 60bpm though it’s probably lower because you’re sleeping), an athlete’s heart will beat only about 19,200 times during sleep (assuming 40bmp, though again probably less during sleep).
Factor that in to the amount of time you are at rest vs exercise over your lifetime, even with hours of exercise regularly the athletes heart is going to come out with fewer beats over time.