========================================= File 7EVAL-WNT.TXT Evaluation of results for File, Macro and Script Virus/Malware detection under Windows-NT in VTC Test "2001-04": ========================================= Formatted with non-proportional font (Courier) Content of this file: ********************************************************************** Eval WNT: Development of detection rates under Windows-NT: ********************************************************************** Eval WNT.01: Development of W-NT Scanner Detection Rates Table WNT-A: File/Macro/Script Virus Detection Rate in last 8 VTC tests Eval WNT.02: Evaluation of overall W-NT AV detection rates (zoo,ITW) Eval WNT.03: In-The-Wild (File,Macro,Script) Detection under W-NT Eval WNT.04: Evaluation of detection by virus classes under W-NT WNT.04.1 Grading the Detection of file viruses WNT.04.2 Grading the Detection of macro viruses WNT.04.3 Grading the Detection of script viruses WNT.04.4 Grading of Poly-virus detection WNT.04.5 Grading of VKit virus detection Eval WNT.05: Detection of Packed File and Macro Viruses under W-NT Eval WNT.06: Avoidance of False Alarms (File, Macro) under W-NT Eval WNT.07: Detection of File and Macro Malware under W-NT Eval WNT.08: Detection of "Exotic" malware Eval WNT.SUM Grading of WNT products ********************************************************************** This part of VTC "2001-04" test report evaluates the detailed results as given in section (file): 6GWNT.TXT File/Macro Viruses/Malware results (W-NT) The following (18) products participated in this scanner test for WNT products: -------------------------------------------------------- Products submitted for aVTC test under Windows-NT: -------------------------------------------------------- AV3 v: 3.0.304.0 sig: Dec.04,2000 AVG 6 v: 6.220 sig: Dec.11,2000 AVK 10 v: 10,0,0,0 sig: Dec.07,2000 AVP Platinum v: 3.5.311.0 sig: Dec.07,2000 CMD v: 4.60 sig: Dec.11,2000 DRW v: 4.21 (see problems list) FPR v: 3.08b sig: Dec.11,2000 FPW v: 3.08b sig: Dec.11,2000 FSE v: 5.21 sig: Dec.01,2000 INO v: 4.53 Enterprise Ed. sig: Dec.11,2000 NAV v: 5.01.01 sig: Dec.07,2000 NVC v: 4.86 sig: Dec.01,2000 PAV v: 3.0.132.4 sig: Dec.07,2000 PER v: 6.60 sig: Nov.30,2000 PRO v: 7.0.A11 sig: Dec.09,2000 RAV v: 8.1.001 sig: Dec.11,2000 SCN v: 4.12.0 sig: Dec.06,2000 VSP v: 12.02.2 sig: Dec.11,2000 -------------------------------------------------------- Eval WNT.01: Development of Scanner Detection Rates under Windows-NT: ==================================================================== The number of scanners running under Windows NT is growing. For the first time in VTC tests, more scanners were available for Windows-NT than for DOS. Evidently, AV producers invest now more work into the development of the W32-related platforms, and here into the detection of macro (and script) viruses whereas detection fo file viruses degrades. The following table summarizes results of file and macro virus detection under Windows-NT in last 8 VTC tests: Table WNT-A: File/Macro/Script Virus Detection Rate in last 8 VTC tests under W-NT: =================================================================================== Scan ======== File Virus Detection ========== + ========= Macro Virus Detection ============= + =ScriptVirusDet= ner 9707 9802 9810 9903 9909 0004 0104 Delta I 9707 9802 9810 9903 9909 0004 0008 0104 Delta I 0008 0104 Delta -----------------------------------------------+-----------------------------------------------+----------------- ANT 88.9 69.2 91.3 - 87.2 92.8 - - I 92.2 - 85.7 - 89.3 90.2 96.4 - - I 55.2 - - ANY - - 69.7 - - - - - I - - 70.5 - - - - - - I - - - ATD - - - - - 100% - - I - - - - - 99.9 - - - I - - - AVA - 97.4 96.6 97.1 97.4 97.2 95.2 -2.0 I - 91.9 97.2 95.2 93.3 94.3 94.1 95.7 +1.6 I 15.0 29.1 +14.1 AVG - - - 87.3 87.0 85.4 81.9 -3.5 I - - - 82.5 96.6 97.5 97.9 98.3 +0.4 I 45.8 57.9 +12.1 AVK - - 99.6 90.2 99.8 99.7 99.8 +0.1 I - - 99.6 99.6 100% 99.9 100~ 100~ 0.0 I 91.8 99.8 +7.0 AVP - - 83.7 99.9 99.8 99.9 99.9 0.0 I - - 100% 99.2 100% 99.9 100~ 100~ 0.0 I 88.2 99.8 +11.6 AVX - - - - - - - - I - - - - - - 99.0 - - I 61.4 - - AW - 56.4 - - - - - - I - 61.0 - - - - - - - I - - - CLE - - - - - - - - I - - - - - - - - - I 4.2 - - CMD - - - - - 99.6 97.8 -1.8 I - - - - - 100% 100% 100% 0.0 I 93.5 96.9 +3.4 DRW/DWW - - 93.3 98.3 98.3 - - I - - - 98.3 98.8 98.4 97.5 - - I 59.8 - - DSS/E 99.6 99.7 99.9 99.3 - - - - I 99.0 100% 100% 100% - - - - - I - - - ESA - - - - - 58.0 - - I - - - - - 88.9 - - - I - - - FPR/FMA - 96.1 - 98.7 99.4 - 97.8 - I - 99.9 99.8 99.8 99.7 - - 100% - I - 97.1 - FPW - - - - - 99.6 97.8 -1.8 I - - - - 99.7 100% 100% 100% 0.0 I 90.8 96.9 +6.1 FSE - 85.3 99.8 100% 99.9 100% 51.1 -48.9#I - - 99.9 100% 100% 100% 100% 100% 0.0 I 96.7 100% +3.3 FWN - - - - - - - - I - - 99.6 99.7 - 99.9 - - - I - - - HMV - - - - - - - - I - - 99.0 99.5 - - - - - I - - - IBM 95.2 95.2 77.2 * * * * * I 92.9 92.6 98.6 * * * * * * I * * * INO - 92.8 - 98.1 98.0 98.7 97.8 -0.9 I - 89.7 - 99.8 99.7 99.7 99.8 99.7 +0.1 I 78.1 92.7 +14.6 IRS - 96.3 - 97.6 - - - - I - 99.1 - 99.5 - - - - - I - - - IVB - - - - - - - - I - - 92.8 95.0 - - - - - I - - - MKS - - - - - 78.0 - - I - - - - - 97.1 - - - I - - - MR2 - - - - 61.9 - - - I - - - - 69.6 - - - - I - - - NAV 86.5 97.1 - 98.0 97.6 96.8 93.9 -2.9 I 95.6 98.7 99.9 99.7 98.7 98.0 97.7 97.0 -0.7 I 36.6 54.5 +17.9 NOD - - - 97.6 98.2 98.3 - - I - - - 99.8 100% 99.4 - - - I - - - NVC 89.6 93.8 93.6 96.4 - 99.1 98.1 -1.0 I 96.6 99.2 - 98.9 98.9 99.9 99.9 99.8 -0.1 I 83.7 88.5 +4.8 NVN - - - - 99.0 - - - I - - - - 99.5 - - - - I - - - PAV 97.7 98.7 98.4 97.2 99.6 100% 99.9 -0.1 I 93.5 98.8 99.5 99.4 99.7 99.9 100~ 100~ 0.0 I 90.2 99.8 +9.6 PCC 63.1 - - - - - - - I - 94.8 - - - - - - - I - - - PER - - - - - - - - I - 91.0 - - - - 85.0 68.2 -16.8 I 0.0 22.0 +22.0 PRO - - - 37.3 42.4 45.6 70.5 +14.9 I - - - 58.0 61.9 67.4 69.1 67.1 -2.0 I 13.1 35.8 +22.7 QHL - - - - - - - - I - - - - - 0.0 - - I 6.9 - - RAV - 81.6 84.9 85.5 - 88.0 93.5 +5.5 I - 98.9 99.5 99.2 - 97.9 96.9 99.6 +2.7 I 47.1 84.9 - RA7 - - - 89.3 - - - - I - - - 99.2 - - - - - I - - - SCN 94.2 91.6 71.4 99.1 99.8 99.8 99.8 0.0 I 97.6 99.1 97.7 100% 100% 100% 100% 100% 0.0 I 95.8 100% +4.2 SWP 94.5 96.8 98.4 - 99.0 99.6 - - I 89.1 98.4 97.5 - 98.4 98.6 - - - I - - - TBA - 93.8 92.6 * * * - - I 96.1 - 98.7 * * * * * * I * * * TNT - - - * * * - - I - - 44.4 * * * * * * I * * * VET 64.9 - - 65.4 * * - - I - 94.0 - 94.9 * * * * * I * * * VSA - 56.7 - - - - - - I - 84.4 - - - - - - - I - - - VSP - - - 87.0 69.8 78.1 74.5 -3.6 I - - - 86.7 0.3 0.0 - 0.0 0.0 I - 85.3 - -----------------------------------------------+-----------------------------------------------+----------------- Mean: 87.4 88.1 89.0 89.2 90,0 91.0 90.6% -2.8%I 94.7 95.9 91.6 95.3 95,1 96.5 96.3 95.3 -0.9%I 57.7 78.9 +11.0% -----------------------------------------------+-----------------------------------------------+----------------- Remark: for abbreviations of products (code names), see appendix A5CodNam.txt. Remark #: this result is "influenced" by the fact that the scanner didnot access even within 2 post-scans several (many) viral directories. The same may apply, though to a much lesser degree, to other products. This may be a side-effect of some software fault in the operating system when transversing the directory tree (FindFirst/FindNext routines). Generally, the ability of W-NT scanners to detect file zoo viruses "in the mean" is stable with a tendency to decrease. The detection level is still insufficient (<91%). Moreover, those scanners present in last test reduced detection rates by 2.8% (mean). Concerning macro viruses, "mean" detection rate is slightly reduced on a still acceptable level (95.3%); here, products having participated in last VTC tests succeed in following the growth by keeping detection rates (+0.1%) on high levels, where spectacular improvements are difficult. Concerning script viruses which is presently the fastest growing sector, the detection rate is very low (78.9% mean). But those (14) scanners which participated in last test succeeded to improve their detection rate by impressing 11% "in the mean". ************************************************************** Findings WNT.1: For W-NT, file and macro zoo virus detection rates in the mean are decreasing. 4 (of 16) scanners detects almost all zoo file viruses (>99.8%), whereas 5 (out of 17) scanners detect ALL macro zoo viruses (and 3 more almost all). Detection rates for script viruses are, in the mean, still inacceptably low (78.9%) but the mean detection rate of those scanners which also participated in last VTC test is significantly improving, and 2 (of 17) scanners detect ALL zoo script viruses. ******************************************************************** Eval WNT.02: In-The-Wild (File,Macro,Script) Detection under W-NT ================================================================= Concerning "In-The-Wild" viruses, the following grid is applied: - detection rate is 100% : scanner is "perfect" - detection rate is >99% : scanner is "excellent" - detection rate is >95% : scanner is "very good" - detection rate is >90% : scanner is "good" - detection rate is <90% : scanner is "risky" 100% detection of In-the-Wild viruses also esp. detecting ALL instantiations of those viruses is now ABSOLUTE REQUIREMENT, for file, macro and script viruses. The following 11 WNT products (of 17) reach 100% for ITW file, macro and script virus detection and are rated "perfect" in this category (alphabetically ordered): ITW Viruses&Files ( File Macro Script) ------------------------- "Perfect" WNT ITW scanners: AVK (100.0% 100.0% 100.0%) AVP (100.0% 100.0% 100.0%) SCN (100.0% 100.0% 100.0%) ------------------------ The following scanners reached the grade "perfect" for file and macro ITW virus detection and did also detect all ITW script viruses but not in all instantiations (files): ITW Viruses&Files ( File Macro ) --------------------- "Perfect" file/macro only ITW scanners: AVG (100.0% 100.0%) CMD (100.0% 100.0%) FPR (100.0% 100.0%) FPW (100.0% 100.0%) FSE (100.0% 100.0%) INO (100.0% 100.0%) NAV (100.0% 100.0%) NVC (100.0% 100.0%) --------------------- ************************************************************ Findings WNT.2: 3 AV products (out of 17) detect ALL In-The-Wild file, macro and script viruses in ALL instantiations (files): AVK, AVP, SCN 7 more products can be rated "perfect" con- cerning detection of file and macro viruses but they still fail to detect all script viral files (objects): AVG, CMD, FPR, FPW, FSE, INO, NAV, NVC ************************************************************ Eval WNT.03: Evaluation of overall W-NT AV detection rates (zoo,ITW) ==================================================================== The following grid is applied to classify scanners: - detection rate =100% : scanner is graded "perfect" - detection rate above 99% : scanner is graded "excellent" - detection rate above 95% : scanner is graded "very good" - detection rate above 90% : scanner is graded "good" - detection rate of 80-90% : scanner is graded "good enough" - detection rate of 70-80% : scanner is graded "not good enough" - detection rate of 60-70% : scanner is graded "rather bad" - detection rate of 50-60% : scanner is graded "very bad" - detection rate below 50% : scanner is graded "useless" To assess an "overall AV grade" (including file, macro and script virus virus detection, for unpacked objects), the lowest of the related results is used to classify each scanner. Only scanners where all tests were completed are considered. (For problems in test: see 8problms.txt). Besides grading products in related categories according to their performance, it is interesting to compare how products developed. In comparison with previous results (VTC test "2000-04") and with respect to file, macro and script viruses, it is notified whether some product remained in the same category (=), improved into a higher category (+) or lost some grade (-). The following list indicates those scanners graded into one of the upper three categories, with file and macro virus detection rates in unpacked samples, and with perfect ITW virus detection (rate=100%). Under W-NT, NO product reached 100% detection rate for file, macro and script viruses, both zoo and In-The-Wild, and could be rated "perfect" (last time, 1 product was rated "perfect"). But 3 scanners are graded "Excellent" (>99%), and 3 more scanners are rated "very good" (>95%): (zoo:file/macro/script;file/macro/script:ITW) ---------------------------------------------- "Perfect" W-NT scanners: NONE ---------------------------------------------- "Excellent" W-NT scanners: SCN ( 99.8 100% 100% ; 100% 100% 100% ) (=) AVP ( 99.9 100~ 99.8 ; 100% 100% 100% ) (=) AVK ( 99.8 100~ 99.8 ; 100% 100% 100% ) (=) ---------------------------------------------- "Very Good" W-NT scanners: FPR ( 97.8 100% 97.1 ; 100% 100% 100% ) (+) CMD ( 97.8 100% 96.9 ; 100% 100% 100% ) (=) FPW ( 97.8 100% 96.9 ; 100% 100% 100% ) (-) ---------------------------------------------- Moreover, 3 products would be rated "Excellent" for file and macro virus detection but with lesser grades for script virus detection: INO ( 97.8 99.7 92.7 ; 100% 100% 100% ) NVC ( 98.1 99.8 88.5 ; 100% 100% 100% ) ************************************************************** Findings WNT.3: No WNT product is overall rated "perfect". 3 "excellent" overall scanners: SCN, AVP, AVK 3 "very good" overall scanners: FPR, CMD, FPW ************************************************************** Eval WNT.04: Evaluation of detection by virus classes under W-NT: ================================================================== Some scanners are specialised on detecting some class of viruses (either in deliberately limiting themselves to one class, esp. macro viruses, or in detecting one class significantly better than others). It is therefore worth notifying which scanners perform best in detecting file, macro and script viruses. Two special tests of file viruses were also performed to determine the quality of AV product maintenance. One test was concerned with almost 11,000 viruses generated from the VKIT virus generator. Some AV products count each of the potential 14,000 viruses as new variant while others count all VKIT viruses just as ONE virus. Fortunately, a high proportion of tested products detects these viruses (see 4.5), although reliability of detection is significantly less than normally (see 6BDOSFIL.TXT). Another special test was devoted to the detection of 10,000 polymorphic generations each of the following 6 polymorphic viruses: Maltese.Amoeba, MTE.Encroacher.B, NATAS, TREMOR, One-Half and Tequila. Detection rates were "almost perfect". Products rated "perfect" (=100%), "excellent" (>99%) and "very good" (>95%) are listed. WNT.04.1 Grading the Detection of file viruses under WNT: --------------------------------------------------------- "Perfect" WNT scanner: === NONE === "Excellent" WNT scanners: AVP ( 99.9%) PAV ( 99.9%) AVK ( 99.8%) SCN ( 99.8%) "Very Good" WNT file scanners: NVC ( 98.1%) CMD ( 97.8%) FPW ( 97.8%) INO ( 97.8%) AV3 ( 95.2%) WNT.04.2 Grading the Detection of macro viruses under WNT ---------------------------------------------------------- "Perfect" WNT macro scanners: CMD (100.0%) FPR (100.0%) FPW (100.0%) FSE (100.0%) PAV (100.0%) SCN (100.0%) "Excellent" WNT macro scanners: AVP ( 100~ ) AVK ( 100~ ) NVC ( 99.8%) INO ( 99.7%) RAV ( 99.6%) "Very Good" WNT macro scanners: AVG ( 98.3%) NAV ( 97.0%) WNT.04.3 Grading the Detection of Script viruses under WNT: ------------------------------------------------------------ "Perfect" WNT script scanners: FSE (100.0%) SCN (100.0%) "Excellent" WNT script scanners: AVK ( 99.8%) AVP ( 99.8%) PAV ( 99.8%) "Very Good" WNT script scanners: FPR ( 97.1%) CMD ( 96.9%) FPW ( 96.9%) WNT.04.4 Grading of Poly-virus detection under WNT -------------------------------------------------- "Perfect" Poly-detectors which detect all instantiations of all (6) polymorphic file viruses always reliable: AVG, AVP, FSE, RAV "Excellent" Poly-detectors which detect all instantiations of all (6) polymorphic file viruses but not always reliably: CMD, FPR, INO, NAV, NVC and SCN. WNT.04.5 Grading of VKit virus detection: ------------------------------------ "Perfect" VKit detectors which detect all generations of VKit viruses always reliably: FSE, SCN "Excellent" Kit detectors which detect all generations of VKit viruses viruses but not always reliably: AVK, AVP, PAV ***************************************************************** Finding WNT.4: Performance of WNT scanners by virus classes: --------------------------------------------- Perfect scanners for file zoo: =NONE= Excellent scanners for file zoo: AVP,PAV,AVK,SCN Perfect scanners for macro zoo: CMD, FPR, FPW, FSE, PAV, SCN Perfect scanners for script zoo: FSE, SCN Perfect scanners for polymorphic set: AVG,AVP,FSE,RAV Perfect scanners for VKit set: FSE, SCN ***************************************************************** Eval WNT.05: Detection of Packed File and Macro Viruses under W-NT ================================================================== Detection of file and macro viruses within packed objects becomes essential for on-access scanning, esp. for incoming email possibly loaded with malicious objects. It seems therefore reasonable to test whether at least ITW viral objects compressed with given popular methods are also detected. In addition to those 4 packers used in previous tests (PKZIP, ARJ, LHA, RAR), the following packers were added to detection tests: WinRAR and CAB. Tests are performed only on In-The-Wild viruses packed once (no recursive packing). As last test showed that AV products are rather far from perfect detection of packed viruses, testbed has essentially be unchanged to ease comparison and improvement. A "perfect" product would detect ALL packed viral samples (100%) file AND macro for all (6) packers: --------------------------------------------- "Perfect" packed virus detectors: AVK and SCN --------------------------------------------- An "excellent" product would reach 100% detection of packed viral samples (file¯o) for at least 5 packers: -------------------------------------------------------- "Excellent" packed macro virus detector: =NONE= ----------------------------------------------- A "very good" product would detect viral samples (ITW file¯o) for at least 4 packers: ------------------------------------------------------ "Very Good" packed macro virus detector: FPR, FPW, NAV ------------------------------------------------------ Concerning only detection of packed file virus samples, the following products can be rated "perfect" as they detect ALL samples: -------------------------------------------------------------- "Perfect" packed file virus detectors: AVK, AVP, FSE, PAV, SCN -------------------------------------------------------------- Concerning only detection of packed macro virus samples, the only products rated "perfect" are those (2) which detect ALL (file AND macro) viral samples: ------------------------------------------------ "Perfect" packed macro virus detectors: AVK, SCN ------------------------------------------------ Remark: Much more data were collected on precision and reliability of virus detection in packed objects. But in the present state, it seems NOT justified to add differentiation to results discussed here. ************************************************************************* Findings WNT.5: Detection of packed viral objects needs improvement Perfect packed file/macro virus WNT detector: AVK, SCN Perfect packed file macro detector: AVK, AVP, FSE, PAV, SCN Perfect packed macro virus detector: AVK, SCN ************************************************************************* Eval WNT.06: Avoidance of False Alarms (File, Macro) under W-NT: ================================================================ First introduced in VTC test "1998-10", a set of clean (and non-malicious) objects has been added to the file and macro virus testbeds to determine the ability of scanners to avoid False-Positive (FP) alarms. This ability is essential for "excellent" and "very good" scanners as there is no automatic aid to customers to handle such cases (besides the psychological impact on customerīs work). Therefore, the grid used for grading AV products must be significantly more rigid than that one used for detection. The following grid is applied to classify scanners: - False Positive rate = 0.0%: scanner is graded "perfect" - False Positive rate < 0.5%: scanner is graded "excellent" - False Positive rate < 2.5%: scanner is graded "very good" - False Positive rate < 5.0%: scanner is graded "good enough" - False Positive rate <10.0%: scanner is graded "rather bad" - False Positive rate <20.0%: scanner is graded "very bad" - False Positive rate >20.0%: scanner is graded "useless" Regarding the ability of scanners to avoid FP alarms, 7 (out of 17) products in test reported NO SINGLE False Positive alarm in file and macro zoo testbeds and are therefore rated "perfect": ---------------------------------------------------------------- "Perfect" FP avoiding WNT scanners: AV3, AVG, AVK, INO, PRO, SCN ---------------------------------------------------------------- (VSP also avoids FPs, but at low level of virus detection) Several more (totally 16) WNT scanners gave NO FP alarm EITHER on clean files: ------------------------------------------------------------------ "Perfect" file FP avoiding WNT scanners: AV3, AVG, AVK, AVP, CMD, FPR, FPW, FSE, INO, NAV, NVC, PAV, PRO, RAV, SCN ---------------------------------------------------------------- (VSP also avoids FPs, but at low level of virus detection) **************************************************************** Findings WNT.6: Avoidance of False-Positive Alarms is improving though still regarded insufficient. FP-avoiding perfect W-NT scanners: AV3, AVG, AVK, INO, PRO, SCN **************************************************************** Eval WNT.07: Detection of File and Macro Malware under W-NT =========================================================== Since test "1997-07", VTC tests also the ability of AV products to detect non-viral malware. An essential argument for this category is that customers are interested to be also warned about and protected from non-viral and non-wormy malicious objects such as trojans etc, the payload of which may be disastrous to their work (e.g. stealing passwords). Since VTC test "1999-03", malware detection is a mandatory part of VTC tests, both for submitted products and for those downloaded as free evaluation copies. A growing number of scanners is indeed able to detect non-viral malware. The following grid (admittedly with reduced granularity) is applied to classify detection of file and macro malware: - detection rate =100% : scanner is "perfect" - detection rate > 90% : scanner is "excellent" - detection rate of 80-90% : scanner is "very good" - detection rate of 60-80% : scanner is "good enough" - detection rate of < 60% : scanner is "not good enough" Presently, there is NO product which "perfectly" detects all (non-viral) file and macro malware in VTC testbed but a the number of "excellent" products (>90%) grows, and 2 products detect ALL macro malware samples: Concerning file AND Macro malware detection: -------------------------------------------------------------- "Perfect" file/macro malware detectors under WNT: === NONE === -------------------------------------------------------------- --------------------------------------------------- "Excellent" file/macro malware detectors under WNT: File / Macro FSE (98.8% 100.0%) SCN (90.4% 100.0%) AVP (96.4% 99.3%) PAV (96.3% 99.3%) AVK (96.1% 99.3%) FPR (94.0% 99.8%) FPW (94.0% 99.8%) CMD (93.5% 99.8%) --------------------------------------------------- Concerning only, macro malware detection 2 products are rated "perfect", and 9 more reach grade "excellent": --------------------------------------------------- "Perfect" macro malware detectors under WNT: FSE (100.0%) SCN (100.0%) --------------------------------------------------- "Excellent" macro malware detectors under WNT: CMD ( 99.8%) FPR ( 99.8%) FPW ( 99.8%) AVK ( 99.3%) AVP ( 99.3%) PAV ( 99.3%) NVC ( 99.0%) RAV ( 97.0%) INO ( 93.8%) --------------------------------------------------- ******************************************************************* Findings WNT.7: File/Macro Malware detection under WNT is slowly improving. NO product is "perfect" but 8 products are rated "excellent" (>90% detection rate): FSE, SCN, AVP, PAV, AVK, FPR, FPW, CMD *************************************************** Concerning only macro malware detection, 2 products are rated "perfect": FSE, SCN And concerning macro malware detection only, 9 more products are rated "excellent": CMD, FPR, FPW, AVK, AVP, PAV, NVC, RAV, INO ****************************************************************** Eval WNT.08: Detection of "Exotic" malware under W-NT ===================================================== With growing exchange of objects which may be activated under other platforms (esp. including Java, Linux etc), scanners must also detect related malware to warn customers before activating such malcode. For the first time in VTC tests, a selected (small) testbed of viruses active on other platforms - we presently call these "exotic" viruses" - has been used to determine the detection quality of contemporary on-demand scanners (engines, signatures). Even with "reduced" quality requirements, the following products detect exotic malware on an initially low level of at least 70% (viruses, files): virus/file FSE (92.2% 92.7%) PAV (92.2% 92.7%) AVK (90.4% 92.0%) AVP (89.6% 86.9%) SCN (70.4% 80.3%) ******************************************************************* Findings WNT.8: Exotic viruses are detected by scanners under WNT only to a lesser degree. The following products detect at least 70%: FSE, PAV, AVK, AVP, SCN ******************************************************************* Eval WNT.SUM: Grading of WNT products: ====================================== Under the scope of VTCs grading system, a "Perfect WNT AV/AM product" would have the following characteristics: Definition (1): A "Perfect AntiVirus (AV) product" -------------------------------------------------- 1) Will detect ALL viral samples "In-The-Wild" AND in at least 99% of zoo samples, in ALL categories (file, boot and script-based viruses), with always same high precision of identification and in every infected sample, 2) Will detect ALL ITW viral samples in compressed objects for all (6) popular packers, and 3) Will NEVER issue a False Positive alarm on any sample which is not viral. Remark: detection of "exotic viruses" is presently NOT rated. Definition (2): A "Perfect AntiMalware (AM) product" ---------------------------------------------------- 1) Will be a "Perfect AntiVirus product", That is: 100% ITW detection AND >99% zoo detection AND high precision of identification AND high precision of detection AND 100% detection of ITW viruses in compressed objects, AND 0% False-Positive rate, 2) AND it will also detect essential forms of malicious software, at least in unpacked forms, reliably at high rates (>90%). Remark: detection of "exotic malware" is presently NOT rated. ***************************************************************** In VTC test "2001-04", we found *** NO perfect WNT AV product *** and we found *** No perfect WNT AM product *** ***************************************************************** But several products seem to approach our definition on a rather high level (taking into account the highest value of "perfect" defined on 100% level and "Excellent" defined by 99% for virus detection, and 90% for malware detection): Test category: "Perfect" "Excellent" --------------------------------------------------------------- WNT zoo file test: --- AVP,PAV,AVK,SCN WNT zoo macro test: CMD,FPR,FPW,FSE,PAV,SCN --- WNT zoo script test: FSE, SCN --- WNT zoo Poly test: AVG,AVP,FSE,RAV --- WNT zoo VKit test: FSE, SCN --- WNT ITW tests: AVK,AVP,SCN AVG,CMD,FPR,FPW,FSE, INO,NAV,NVC WNT pack-tests: AVK,SCN --- WNT FP avoidance: AV3,AVG,AVK,INO,PRO,SCN --- WNT Malware Test: --- FSE,SCN,AVP,PAV,AVK, FPR,FPW,CMD ---------------------------------------------------------------- In order to support the race for more customer protection, we evaluate the order of performance in this WNT test with a simple algorithm, by counting the majority of places (weighing "perfect" twice and "excellent" once), for the first places: ************************************************************ "Perfect" WNT AntiVirus product: =NONE= "Excellent" WNT AV products: 1st place: SCN (13 points) 2nd place: FSE ( 9 points) 3rd place: AVK ( 7 points) 4th place: AVG, AVP ( 5 points) 6th place: CMD,FPR,FPW,INO,PAV ( 3 points) 11th place: AV3,PRO,RAV ( 2 points) ************************************************************ "Perfect" WNT AntiMalware product: =NONE= "Excellent" WNT AntiMalware product: 1st place: SCN (14 points) 2nd place: FSE (10 points) 3rd place: AVK ( 8 points) 4th place: AVP ( 6 points) 5th place: CMD,FPR,FPW,PAV ( 4 points) ************************************************************