Subcompact fuel efficiency trends

This post on a proposal to buy up "clunkers" as a fiscal stimulus and at the same time address some environmental concerns had me looking at trends in fuel efficiency. I obtained the data from Appendix F of Light-Duty Automotive Technology and Fuel Economy Trends: 1975 Through 2007. I was surprised to find that since 1996 fuel efficiency had declined and that fuel efficiency of 1982 subcompacts were no better than those of today. Of course, fuel efficiency is not just a function of model year but also depends on how well the older cars have been maintained. As such, a plan to buyback "clunkers" because they are more polluting based on some model year cutoff is perhaps not the best way to approach it.

Of course, more analysis of other car classess needs to be done but since my Saturn SL2 is considered a subcompact I just went with this. What this does not show is the average weight of the vehicles in this class which has been trending upward just as fuel efficiency has been trending down. I suspect that the increasing weight of the vehicles in this class accounts for the lower fuel efficiency. This is probably explained by the disappearance of the 2-door and hatcback versions of the Honda Civic that were ubiquitous in the early 80s. With fuel prices at $4.00 a gallon we're starting to see a comeback of these 2-door cars like the Toyota Matrix

Eyeballing Appendix F for midsized cars, fuel efficiency for this class of vehicles has actually increased. From 1975 to 2007, it looks to be from around 11mpg to 24 mpg. Fuel efficiency reached 21 mpg around 1985 and seems to have hit a wall since then.

CAFE standards were enacted in 1975 which is when the data starts and if we are judging the success of CAFE solely on fuel efficiency it appears to have achieved great gains but has stagnated from the mid 80s onward. Can a new CAFE standard force the industry to achieve more gains in fuel efficiency?

But why does the industry have to wait? Perhaps the new higher gas prices might be the impetus needed to invest in the R&D that can achieve gains in fuel efficiency or has auto technology reached a plateau as far as this is concerned? Can fuel efficiency be achieved only by making vehicles lighter? Is there little advance left to be made in combustion engine technology?