Because moore’s law reached its limit and we can no longer increase capacity by simply reducing the transistor size because we got to a point where we ran into weird quantum mechanics effects.
Now we have to actually redesign the way things work to find places for optimisation, but that often makes things prohibitively expensive so improvements that actually make it into publicly available products are few and far between.
Back in the Pentium days, you’d release a Pentium III with a 1GHz clock instead of the previous 500MHz clock and you’d get a 100% improvement in speed across the board, no matter what.
Now that we reached the 3-4GHz barrier we instead add more cores, more cache, dedicated chips for certain tasks and whatnot and not only the gains are not as significant to begin with, only applications specifically designed to take advantage of those changes are afected. Worst case scenario, applications that do not take these changes into consideration actually perform WORSE.
Latest Answers