Initial research paper on using Aspect Oriented Programming (AOP) techniques to analyze the effectiveness of commercial web application security scanners:
4-23-2010
As a postmortem to my recent study, I have been conversing with some of the vendors who had issues with my results and wanted to spend some time to do their own verification of the results. Cenzic took the time to verify each reported issue and present me with their version of the results and also tried to clarify some of the reasons why they believed they received a lower score than they should have. As part of this post I will include an updated spreadsheet that represents the new information arrived at by their tests and also my attempt at verifying the new results. I give the readers the task to make up their own minds, but I admit Hailstorm can do better if a bit of a knowledgeable training effort is made.
One point of contention was the Point and Shoot category in which they did not fare so well. They explained to me that I used the Best Practices category for configuring the Point and Shoot scan - which was not optimal. To be fair in my initial discussions with them I did ask which category would be good for a fairly complete scan and someone did indicate Best Practices would be the way to go. The reality was that they did not have a preconfigured scan that included comprehensively all of their vulnerability categories, thus resulting in a lower score. After they provided me the results with a custom scan configuration, their Point and Shoot improved significantly.
My conclusion here is that Cenzic Hailstorm is a top-notch tool. I do maintain, however, that Hailstorm takes some getting used to. Its architecture is quite a bit different from the other scanners and requires a bit of understanding on how vulnerability categories are configured in order to get as comprehensive a scan as possible. After reviewing Cenzic's updated results, I acknowledge that they made a significant improvement. Also I believe there was a little disconnect between myself and the support staff when it came to configuring Point and Shoot. Since Cenzic's deployment model is not generally geared towards the Point and Shoot usage pattern many of the default scans are focused towards a compliance perspective which then can be tuned based on a particular customer need. I think they now see that I was looking for more of a comprehensive scan configuration that pretty much handles as many vulnerability categories as possible with as little configuration as necessary. As you can see from the spreadsheet we had some contention on whether SQL errors and things of that nature counted as vulnerabilities. I think currently I acknowledge that SQL errors while not false positives also should not be counted as vulnerabilities.
Cenzic also asked me to publish these points on their behalf. Since they were willing to participate in the study by their choice, I think the readers should be aware of these statements:
Cenzic Hailstorm has 107 categories of attacks more that most scanners - including many attacks like session hijacking, session id randomness, privilege escalation etc. -they use stateful assessment (i.e browser is embedded in the scanning engine) technology
With Enterprise capabilities like dashboard with trends, application discovery, LDAP integration, role based deployment are very important for customers - Cenzic has desirable enterprise features.
The 6.0 results we sent to you were using the exact same version that you used.
Our 6.5 has significant improvement in results. We released the new version two weeks ago. This has about a year's worth of improvements in spidering and attacks. We used 6.5 internally against the same targets using your guidelines and found significantly higher number of vulnerabilities found with very few false positives.