Of course, such statistics can be interpreted any way you like. And those Chaos reports tend to take a lot of liberties in interpreting the data, to make their point.
And of course, they completely ignore the up front design costs when they state that civil engineering projects are usually built on time these days. If the project sees new light, and the Heathrow 3rd runway gets built, I'm sure that the company which lays the concrete will claim that they did it on time and to budget. But I'm pretty sure that won't be the same time, budget or design, that was conceived back in 2006. And the budget won't include all the lawyers and lobbyists who have spent years fighting over it. Given that a rather large proportion of money made by civil engineering companies, comes from calling everything a 'change request' I'm not so sure that software development is as bad in comparison to traditional engineering as many proponents of improved methodologies would like us to think.
The results of those reports seem to compare sizes of company rather than sizes of project. Larger companies are more likely to run multi-year large projects, whereas smaller companies just can't afford to even attempt that. It's fairly obvious to me that a 3 month (or shorter) project is likely to have much better defined goals and expectations than a multi-year project. It's also fairly obvious to me that business needs are unlikely to change during a 3 month project, whereas anything that takes a year or more, is likely to be impacted by changing business requirements, regulations, politics, new management and many other changes.
Which just tells me that methodologies that encourage shorter iterations (which actually go live!), are probably the most efficient.
Based on my experience of small and large companies, large companies tend to have original requirements driven by the egos of the senior management, rather than their actual necessity in the project. This means that a large proportion of the original requirements are not truly necessary in the final product. Small companies have less cash to spare, so they don't try to build monuments to their own egos.
As such, I would expect large companies to drop more of their originally required features than smaller companies. And that statistic would bear no relation to which software methodology they were using. In fact, dropping more requirements could be a sign that the software development methodology is working!
Finally, returning to the discussion of Agile, it's probably worth emphasising that although a method or technique may claim to be Agile, Agile is not a method or a particular technique.
… in fact, the first statement of the manifesto tells us to care for "individuals and interactions over processes and tools". Unfortunately, that doesn't seem to be implemented by many proponents of so-called Agile methods.
Personally, I lean towards the Lean camp, and theory of constraints. Reduce work in process, bring quality checks forward and always look to improve. If drawing a diagram enables me to express my thoughts about a design effectively so that the design can be reviewed and I can get feedback early, then I have avoided wasting time building the wrong thing. However, if I take this to the extreme applied in many 'waterfall' implementations, I could spend months writing documents and drawing pictures before any developer is permitted to start building anything. I therefore lose out on the feedback of the built product, which is telling me that although the design seemed okay at the start, it's not working in practise.
To me it seems that most 'Agile' methods and techniques have evolved from Lean in that they were originally intended to help achieve goals such as early feedback on quality, and reduced WIP. Unfortunately many are now applied without that context. For instance, companies/teams often have a target N% unit test coverage for all code. By removing the context of *why* a particular line of code needs a unit test, we are no longer encouraging developers to consider how it might be tested better, or whether some other piece of code should be tested more. In fact, these techniques without context are reminiscent of Ben's coconut headphones.