========================= File 5PROTOCO.TXT AV Product Test Protocol: ========================= Formatted with non-proportional font (Courier) This document specifies the test procedures applied to test the precision of detection as well as the reliability of detection of PC-based boot, file and macro viruses. Moreover, test procedures for determining detection of packed viral objects and non-viral malware are also described. Where relevant, details concerning differences against previous VTC tests (esp.1999-09) are given. 1) Hardware and System Software used: ------------------------------------- Test "2000-04" installation differs from last test (1999-09) essentially in updated testbeds (which were frozen on October 31, 1999). Concerning the ability to detect *many* generations of polymorphic file viruses, a special testbed with 10,000 generations each of 6 polymorphic viruses was included. Again, a special test was devoted to the detection of about 11,000 viruses generated with the VKit generator. Again, the detection of viral code in packed (file and macro) objects was tested for the set of In-the-Wild viruses, including 4 popular packers (ZIP, LHA, ARJ and RAR). Moreover, a set of non-malicious objects was used to determine the ability to avoid false-positive warnings, and a special (file/macro) malware database was included to determine the degree to which trojan horses are detected. As in test "1999-09", 3 platforms (DOS, W-98 and W-NT) were used, esp. as W-95 results did not differ too much from the W-98 results in previous test. The virus databases of BOOT, FILE/VKit/Poly and MACRO viruses was held on a Win NT 4.0 SP5 server: Win-NT Server (1) has the following hardware: Pentium 200 MHz, 64 MB RAM, 2 GB hard disk (boot) 2*4,3 GB data/reports, 2*9,1 GB virus database (mirror) 3 network cards: 2*100 MBit/sec, 1*10 MBit/sec Protected against electrical faults (USV: APC 420 VA) Operating system: Windows NT Server 4.0 SP 6 Network: 1* 10 MBit/sec BNC for 4 DOS clients 1*100 MBit/sec via 2 cascaded switches for all other clients with 10 MBit/sec cards 1*100 MBit/sec via 100 MBit/sec hub for all other clients Additionally, 29 clients (20 MS-DOS, 6 Win-98 and 3 Win-NT) were used for the test. DOS-Clients are essentially used to test AV-products with boot viruses. DOS-Clients work on MS-DOS 6.22. Hard disks are only used for the boot process. Win-98 client works under English version. Win-NT clients work under Windows NT 4.0 Workstation with SP 5, English version. All clients are connected to the server using Microsoft NetBEUI. Generally, clients were flexibly allocated to optimize scanning processes. DOS Clients (20) have the following hardware: 20* Intel 80486 50 MHz, 8 MB RAM, 270 MB hard disk, 10 MBit/sec switched to 5 monitors over switchboard Win-98 Clients (6) have the following hardware: 2*Pentium 133 MHz, 64 MB RAM, 2 GB hard disk, 10 MBit/sec Pentium 133 MHz, 64 MB RAM, 2 GB hard disk, 100 MBit/sec Pentium 90 MHz, 32 MB RAM, 1 GB hard disk, 100 MBit/sec Pentium-II 350 MHz, 64 MB RAM, 2 GB hard disk, 100 MBit/sec Win-NT Clients (3) have the following hardware: Pentium 233 MMX MHz, 64 MB RAM, 2 GB hard disk, 100 MBit/sec Pentium-II 233 MHz, 64 MB RAM, 4 GB hard disk, 100 MBit/sec Pentium-II 350 MHz, 64 MB RAM, 4 GB hard disk, 100 MBit/sec Specially developed software supporting semi-automatic execution of test scans and evaluation of protocols consist of batch programs and scripts (PERL and AWK). Some UNIX programs like AWK, GAWK, JOIN etc have also been applied. 2) The Databases of File/Boot/Macro viruses: -------------------------------------------- An overview of entries in the VTC virus databases (status: March 31, 1999) is given in Appendix 3: "A3TSTBED.zip". TESTBED.VTC contains the following entries (in ZIPped form): ALLBOOT.VTC index of VTC boot virus database (complete) ALLFILE.VTC index of VTC file virus database (complete) ALLMACR.VTC index of VTC macro virus database (complete) ITWBOOT.VTC index of VTC boot virus database (ITW) ITWFILE.VTC index of VTC file virus database (ITW) ITWMACR.VTC index of VTC macro virus database (ITW) MALFILE.VTC index of VTC file virus database (Malware) MALMACR.VTC index of VTC macro virus database (Malware) PACKFIL.VTC index of VTC packed file virus database PACKMAC.VTC index of VTC packed macro virus database POLY.VTC index of VTC database of 4 polymorphic viruses VKIT.VTC index of VTC database of file viruses generated with VKit These entries (which also indicate the multiplicity of infected objects in the resp. directory) also conform with related entries in scanner evaluation protocols. All file and boot viruses are sorted into their resp. database according to diagnostic messages of three "standard" scanners (AVP, DSAV, F-Prot). The database of the file viruses consists of four parts. If the three scanners identify a virus with the same name, it is stored in the first part of the resp. database, named "NORM", as those standard scanners reflect an agreed CARO name. All file viruses for which no such agreement on their name is obvious, are stored in a second directory "NONORM" (formerly Not-Yet-CARO). If a virus is operating-system specific, it is stored in resp. directories named Win95 or OS2. The following file extensions are present in the file viruses database: EXE, COM, SYS, BAT and CMD. Contents of the file virus database: ------------------------------------ 18,357 different file viruses (excluding VKit viruses) 135,907 files infected each with exactly ONE file virus 39 different file viruses reported "In-The-Wild" (ITW) 1,047 files infected with exactly ONE ITW-virus 60,000 instantiations of 6 polymorphic viruses, each with 10,000 different generations (size: 1,8 GByte) 10,706 file viruses generated with VKit virus generator 104,640 different objects each infected with one VKit virus (size: 452 MByte) 39 ITW file viruses packed packed with 4 packers (ZIP, LHA, ARJ, RAR) 1,042 different objects infected with one ITW file virus and packed with one of 4 packers (ZIP, LHA, ARJ, RAR) 1,851 different entries with non-malicious/non-viral objects used for false-positive (fp) test 6,639 non-malicious/non-viral objects for malware test (size: 136 MByte) Similar to the file viruses database, all boot viruses are sorted into a special directory structure. The boot viruses are not divided into different categories. Boot viruses are stored as images of boot sectors and processed with SIMBOOT (see 5PROTOCO.TXT). Contents of the boot virus database: ------------------------------------ 1,237 different boot viruses 5,379 images representing ONE boot virus each, 33 different boot viruses found ITW 423 images representing ONE boot virus, found "In-The-Wild" The macro virus database is organised according to the CARO macro naming convention. Related testbeds contain macro viruses known at end-April 1998 (see VTCs List of Known Macro Viruses). For each macro virus, different goat documents were stored to test consistent identification and reliable detection. Contents of the macro virus database: ------------------------------------- 4,525 different macro viruses 6,639 files infected each with exactly ONE macro virus 80 different macro viruses reported "In-The-Wild" 672 files infected with exactly ONE ITW-virus 80 ITW macro viruses in 672 infected objects, packed with one of 4 packers (ZIP,LHA,ARJ,RAR) 329 totally non-malicious/non-viral objects in 26 different directories for fp-test 2A) Additional File Malware Databases: -------------------------------------- Concerning non-viral malware, VTC maintains a large collection of trojans, virus generators, droppers, worms such as MIRC, as well as intended and first generation viruses, etc. A subset of these non- viral file malware was tested; from the huge database of potential file malware, samples were selected to determine the ability of AV products to protect customers from these threats. This testbed included: 6,639 different specimen of file malware in 4,282 different directories. 2B) Additional Macro Malware Database: -------------------------------------- Concerning non-viral malware, the subset of non-viral macro malware tested is well documented (see VTCs "List of Known Macro Malware" which summarizes both viral and non-viral macro malware). This testbed included: 394 specimen of macro malware in 260 different directories. 2C) Additional test for False Positive Detection: ------------------------------------------------- In order to test the ability of scanners to avoid "false positive" alarms on non-malicious non-viral objects (files and macros), 2 sets of "clean" objects were mixed into the resp. viral databases. Clean files collected from several CD-ROMs were used for tests: 1,851 non-malicious non-viral objects (*.exe, *.com etc) were stored in 38 different directories. The list of CD-ROMs used for false positive testing is listed in appendix 3 (A3TSTBED.ZIP). Concerning testing for false positive alarms on macro viruses, a set of 329 non-malicious non-viral objects (*.doc, *.dot, *.xls) were stored in 26 different directories. Remark: concerning copyright of related CD-ROMS, we use selected active content to help protecting the copyright holder for wrong allegations concerning false alarms. We never use the code actively but only for assurance that scanners dont falsely alarm on these samples. 3) Testing scanners on standard database of file infecting viruses: ------------------------------------------------------------------- (Text essentially same as in previous tests: cf. 1999.09). The viruses are stored in a huge subdirectory tree, the hierarchical structure of which reflects the CARO virus naming scheme, with the samples of each virus stored in the leaf directories of the tree. A virus can be (and usually is) represented by more than one replicant, although different viruses are not represented by one and the same number of replicants. All replicants that contain one and the same virus, are stored in one and the same directory. If two files are in two different directories, this means that they contain two different viruses. Each sample in the CARO subset was at least reported by three scanners. All efforts have been made to ensure that the samples used during the test are natural replicants of working viruses: no Germs, Corrupted files, or Intended viruses. Nevertheless, it is possible that we have made some mistakes in this aspect. If somebody notices any mistakes of this kind, we shall appreciate being told about them. If we received arguments from some AV producer that some sample may be non-viral, we have removed such a sample from the test if we could not immediately prove its virality Each scanner is run on this directory tree, and the resulting report file is preprocessed. The preprocessing is done with a set of batch files, some Unix utilities ported to DOS (sort, join, cut, paste, awk), and a set of awk scripts. The preprocessed report contains four columns. The first column con- tains the directory containing viruses. The second contains the number of scanned files in the directory. The third contains the number of detected files. The fourth contains the information whether all files are reliable detected (with the same name). For each scanner, the report and the preprocessed data are stored in special directory. Not the whole output of the scanner is contained in the third column, because this output often tends to be too verbose. We have put there only the distilled information that we have judged important for that particular scanner. If we have missed some important information, we shall appreciate being told about it. Additional remark for Test 1999-03: with linear but fast growth of virus numbers, naming became less organised. When this test was prepared, less than 25% virus names could be regarded as "CARO agreed", esp. as members of the CARO naming committee were overloaded in their daily fight against new viruses and in helping victims of viral events. While VTC testers hope that the chaotic situation of virus naming may improve, we have left the second column out of this report. Concerning macro viruses, VTC uses itīs "List of Known Macro Viruses and Malware" (which is maintained upon consent of experts working together in CAROs "VMacro list") as naming standards. 4) Testing scanners on database of boot sector infecting viruses: ----------------------------------------------------------------- (Text essentially same as in Vesselin Bontchev's test 1994-07) The boot sector viruses are kept in a similar subdirectory tree, as files, containing the images of the infected boot sectors. For the purposes of the test, we used a program, called SimBoot, developed by Dmitry Gryaznov. This program is still under development and is not available to the general public, but we will make it available to those producers of the scanners, who have reasons to suspect that the program has unfairly interferred with their product and has not allowed it to be tested properly. The program takes a file, of which the first 512 bytes are supposed to contain the first sector of a boot sector virus. It then emulates a blank formatted floppy disk in drive A:, boot sector of which is replaced by the image in the file. If the file is smaller than 512 bytes, it is padded with zeroes. If the image contains a valid diskette BPB which indicates a particular diskette size, a diskette with that particular size is emulated. If a valid BPB is not found,a 360 Kb diskette is emulated. Currently only the first sector of the boot sector virus is put on the emulated diskette. Program SimBoot is able to handle complete viruses, consisting of several sectors, but this requires that the file image of the virus conforms to a particular format. We did not have the time to prepare all our boot sector viruses in this way, although we are considering to do this in the future. One major flaw of this approach is that hard disk, and respectively MBRs are not emulated. Testing of a virus which infects only MBRs (e.g., Tequila) but not boot sectors of floppy disks, is still done by putting an image of the infected MBR on the boot sector of the simulated diskette. We understand that this is not very correct - a scanner may refuse to look for a particular virus on a diskette boot sector, if it knows that this particular virus just cannot be there. The author of SimBoot is considering to improve it in the future, in order to make it able to simulate hard disks too. Once SimBoot creates the simulated infected diskette, it runs the scanner to be tested, as specified in the configuration file for this scanner. (The configuration files are available in the archive SCRIPTS.ZIP.) The scanner is supposed to scan the diskette (SimBoot intercepts all INT 13h requests to drive A: and redirects them to access the simulated diskette), reports its status in the report file, and prompt the user to insert the next diskette to be scanned. SimBoot intercepts the prompt and simulates user input from the keyboard. Both the prompt and the required user input are specified in the configuration file for each scanner. SimBoot is able to handle scanners that write their prompts directly to the video RAM. It is also able to handle scanners that poll directly the keyboard when waiting for user input instead of using the BIOS. SimBoot is even able to simulate changing the status of the floppy drive from Closed to Open and then again to Closed, in order to handle those scanners which poll the DiskChanged line and in order to figure out when the user has put a new diskette. Methodological remark: SIMBOOT is selected as more "realistic" test methods would be difficult to practice (e.g. tesitng viruses on diskettes requires either a permanent formatting/infection/testing or a sequential test of many diskettes). But as any simulated method (even if as well done as SIMBOOT), this method may be unfair to scanners which scan for real floppy characteristics. We have been informed that McAfee's Scan works in such a way; in this case, the real detection rate of such a product can only be assessed using some different test method. The resulting report of each scanner is further preprocessed with a similar set of batch files, awk and PERL scripts as the report of the file virus scanning. 5) Testing scanners on ITW databases of boot/file infecting viruses: -------------------------------------------------------------------- Since VTC test "1999-03", VTC tests of ITW virus detection are based on the set of samples prepared from "Wildlist Organisation" for itīs monthly publication of "The Wildlist"; we wish to thank Wildlist.Org for their support on this issue. When freezing VTC file, boot and macro virus testbeds (for this test: November 30, 1998), samples received from Wildlist.org are compared with VTC samples and properly sorted into related testbed entries. Tests run over the whole "zoo" database, while ITW reports are extracted from the zoo reports of each scanner. 6.) Testing scanners on standard database of Macro Viruses: ----------------------------------------------------------- All AV scanners are tested against two large macro-related database. The main datadabse contains all "zoo" and ITW macro viruses, both in uncompressed and compressed forms; mixed into this dfatabase, there are also specific directories contaiining non-viral macro objects for false-positive detection. The second (smaller) database contains all non-viral maco malware (trojans, droppers, intendeds etc). All malware included in those databases matches the contents of the VTC Macro Virus List, which is published regularly (previously: monthly, now at the end of each quarter) For details, see http://agn-www.informatik.uni-hamburg.de/vtc. The malware database contains also some file viruses which are being created ("dropped") by macro viruses. We decided to test them in the context of the macro malware test because they only appear in the context of macro malware. The directory structure of the virus database reflects the CARO naming scheme for macro viruses with all samples of one variant stored in one subdirectory. Starting from the root directory of the database, the first level contains directories describing the host software (Word, Word97, Excel, Excel97, Lotus123, AmiPro). The second level contains subdirectories with the names of the families of the viruses and the next level hosts subdirectories of all variants of that family, in which the viruses can be found. Optionally (only in malware database), we have another subdirectory called "FILE" which contains the file viruses mentioned above. The number of samples for each virus varies between one and 78 samples (for Concept.A), although the average is 2-3 infected objects each. Our results are split into two sections: "detection of viruses" and "detection of files", where "detection of viruses" has two sub- sections: "unreliable detection" and "unreliable identification". (An index of the malware databases is available in a3tstbed.zip) After each scanner is run, all report files are preprocessed by those AWK scripts already mentioned in the desciption of file virus test. 8) Creating the final summary of the results: --------------------------------------------- (Text essentially same as in previous test: 1998-10). The final evaluations for all tests are similar. Only one report of file and macro viruses tests is used to get the total number of files in the directory. As for boot viruses, the configuration file from Simboot is used (if there was no specific need for manual operation). Three new files result from these processes. New files contain the directory name and the total number of files in this directory. Each preprocessed report is joined with the new file. One AWK-scripts evaluates the result of the joining. The results are listed as follows: - The number of viruses (+malware) detected: it is not necessary that all examples of the virus are detected. - The number of viruses with unreliable (=inconsistent) identification: all variants of a viruses are detected but at least one sample is identified with a different name. - The number of viruses with unreliable detection: here, not all samples of a virus are detected but at least one. The files containing the preprocessed information mentioned above are huge, although they are reduced to contain essentially the virus names. For all tested scanners (latest version), they are included in a separate archive (Scan-Res) for anonymous ftp.