Efficiency (ie hp per cylinder liters) is only one of the aspects of the problem.
By going to a bigger motor you get a more flexible motor that can handle well low rpm without the need for downshifting. That’s why these cars sometimes had only a 4 speed automatic gearbox, you don’t care about being on the right gear.
Also the quest for higher efficiency (using turbo) also degrades reliability, for example the old Volvos are uber solid because they achieve low HP compared to their motor.
Lastly, carmakers and customers didn’t care because fuel was super cheap.
Latest Answers