========================================= File 8PROBLMS.TXT List of problems experienced during test: ========================================= Formatted with non-proportional font (Courier) Content of this file: --------------------- 1. Introduction: General Problems 1.1 Problems likely related to FindFirst/FindNext anomaly 2. List of benevolently behaving AV products 3. Problems of AV products observed during tests 1. Introduction: General Problems: ---------------------------------- For automatic tests on large viral databases, and for automatic processing of large scanner log files, a set of test conditions is prerequisite for scanners to participate in a VTC test (see: 4TESTCON.TXT). In many cases, serious problems were observed during some tests. DOS scanners were either not suitably running under SIMBOOT and crashed, or problems appeared with the (rather large) file virus database. In some cases, scanners crashed upon detecting some specific virus; in few cases, "manual" operation instead of automatic (batch) operation helped solving some of these problems. Such curative action was also applied when possible in cases where log files were inadequate (e.g.needing manual operation for export). With growing velocity of processors, DOS scanners (running without any problem on INTEL 386 and 486) growingly crash on Pentium II/III systems faster than 250 MHz. Another general problem with DOS scanners is related to counters for files and viruses which often seem to be designed as integers, so they start after 65,536 with 0. During preparation and test, we again experienced a serious problem reported in VTC Test "2001-04", according to which management of large sets of directories in FAT and NTFS may not reliably work. Both when attempting to move large parts of our file virus database, as when some scanner proceeded scanning subsequent viral directories, we found that several directories were not moved or touched. This effect seems to happen stochastically, such that subsequent attempts gave different results. Concerning omitted (=unscanned) directories, we overcame this "dysfunctional" behaviour of FAT and NTFS by repeat- ing scanning so long until the number of scanned files agreed with the (known) number of directories in testbeds. Overcoming this problem was extremely time-consuming, and this was a reason for delaying publication of results. In cases where scanners crashed during detection test upon the rather large file virus database, tests were performed in several runs on partitions (essentially on directories with same first letters of names). In most cases (apart those reported below), these tests were completed, and resulting files were joined and evaluated. Finally, with growing testbeds, test protocols produced by scanners grow equally. When processing such protocols, we need meanwhile up to 6 GByte of disk space, and our evaluation scripts (in AWK) become more complex. Under these conditions, we also suffered from an evident bug in the AWK processor which inhibited proper evaluation and required additional quality assurance (including time and efforts). 1.1 Problems very likely related to FindFirst/FindNext anomaly: --------------------------------------------------------------- In several cases, scanners finished a first scan although they had not touched all directories with infected objects. In such a case, a postscan was started adressing only those untouched objects; a second postscan was started when again objects were observed untouched, but after the 2nd postscan, no more scan was started. This behaviour may originate from a reported anomaly in the behaviour of FindFirst/FindNext (those routines are used to handle objects in directory trees) which has not been cured so far by Microsoft. In the "problems list", these postscans are ked as "minor problems": different from crashes, postscans "only" required time for running the tests and evaluating test protocols (which significantly delayed results). 2. List of benevolently behaving AV products: --------------------------------------------- In general, few scanners could be tested without any problem. When considering the large number of postscans, few products had only minor problems and are regarded as "relatively benevolent". Such "benevolent behaviour" can be reported only for a minority of DOS scanners: ========================================== Out of 12 DOS scanners, 2 had NO problems: ------------------------------------------ AVG and FPR ========================================== In comparison, W-98 scanners were significantly less stable: ============================================ Out of 23 W-98 scanners, 2 had only no or minor problems: --------------------------------------------------------- - executed with no problem: (1) AT5 - executed with minor problems: (1) PAV ============================================ Concerning W-NT related scanners: ================================================== Out of 18 W-NT scanners, the following 6 products -------------------------------------------------- - executed with no problem: (1) AVG - executed with minor problems: (5) AVP, CMD, FPR, FPW and PAV ================================================== Concerning Windows-2000 based scanners: ============================================== Out of 17 W-2000 scanners, the following 13 products had NO problems: ---------------------------------------------- - executed with no problem: (4) AVG, CMD, FPR, PAV - executed with minor problems: (5) AV3, AVK, AVP, FPW, NAV ============================================== Finally, concerning Linux-based scanners (tested for the first time): ------------------------------------------- Out of 5 Linux scanners, 3 had no problems: CMD, RAV and SCN ------------------------------------------- Concerning overall stability: *************************************************************** the following 3 products behaved with least problems (no crash, "only postscans"): CMD, FPW, RAV *************************************************************** 3. Problems of AV products observed during tests: ------------------------------------------------- Concerning postscans (either due to untouched objects - see FF/FN anomaly - or due to crashes of a product), the following list summarizes those products where at least 1 postscan was initialised: W-NT: EXOT AVP, PRO MACRO SCN POLY SCN(2x), VSP(2x), FPW VKIT VSP, FPR(2x) FILE CMD, FPR, FPW, FSE(2x), NVC, SCN, VSP(2x) FILEMAL AVK(2x), NVC(2x), VSP(2x) FILEPACK AVP MACPACK AVP, FSE, PRO, SCN W-98: FILE ADO, AV3, AVG(2x), CLE, CMD, DRW, DSE, FPR, FPW, INO(2), MR2(2x), NAV, NVC(2x), PRO, SCN(2x), VSP FILEMAL ADO, ATR, AV3(2x), AVK(2x), AVP, DSE(2x), FPW(2x), MR2(2x), NAV(2x), NVC, PAV, PER, SCN(2x) FILEPACK ADO, FPR, FSE, PER, QHL, RAR MACRO ADO, AV3, AVK, AVP, CLE, DSE, FPW, INO, MR2(2x), NAV, NVC(2x), PER, PRO, QHL, SCN, VSP MACROITW AV3, FSE MACROMAL ADO MACPACK ADO(2x), AVP, FSE, INO, NAV, PAV(2x), QHL POLY DRW, MR2, SCN, VSP SCRIPT ADO, AV3, FSE SCRIPTITW ADO VKIT FSE W-2k: FILE AV3, CMD, FPR, FSE, INO, NAV, NVC(2x), PAV, PRO(2x), SCN(2x), VSP FILEMAL AVK, AVP, FPW, FSE, INO, NVC(2x), PER, PRO, RAV(2x), VSP MACRO AV3, AVK, INO, NAV, NVC, PER, PRO, SCN(2x), VSP POLY SCN, VSP VKIT SCN The following list reports specific problems observed for products as indicated ("spoon-feeding" means that scanner was restarted on each subsequent directory when a crash was experienced): ANT Product couldnot be installed and tested, as CD was physically broken (CD arrived significantly after deadline) ATD: W-98: Date reset (Jan.1, 2001) Saving logs requires clicking with right mouse button on "infected list" ATR: W-98: no problems AVA DOS: Crashed three times on BOOT-ZOO; excluded then from this part of the test (no results) Crashed three times on BOOT-ITW; excluded then from this part of the test (no results) AV3 W-98: minor problems (postscans) W-NT: no problems W-2k: minor problems (postscans) AVG DOS: no problems W-98: Crashed at u:\23w\mronon\c\otpyrc\lld\dlb_000_.dll, Diagnosis: the instruction at 0x00440115 referenced memory at 0x00851000. the memory could not be read. W-NT: no problems W-2k: no problems AVK DOS: AVK has a maximum size for its report files of 512 Kb. Several reports were uncomplete because of this limitation. This forced several postscans. Was unable to scan several files; for this product, these files were nonexistent. W-98: Hangs after long scans (poly, Vkit, file viruses) W-NT: minor problems (postscans) W-2k: Crashed once when scanning File viruses AVP DOS: Crashed once during Boot-tests Problems when scanning CAB-Packed files: sometimes crashed with exception #0e, error code 0004 Was unable to scan several files; for this product, these files were nonexistent. W-98: minor problems (postscans) W-NT: minor problems (postscans) W-2k: minor problems (postscans) Linux: Product crashed when processing several CABbed viral objects such as: /k:packmac/W97M/MARKER/X/CAB.CAB /k:packmac/W97M/MELISSA/A@MM/CAB.CAB /k:packmac/W97M/MELISSA/AL@MM/CAB.CAB /k:packmac/W97M/MELISSA/I@MM/CAB.CAB CLE: W-98: Product always hangs when scanning File-Malware at T:\NAJORT\MRONON\S\NEVESBUS\ROODKCAB\312v\EXA_002_.EXE CMD: W-98: no problems W-NT: minor problems (postscans) W-2k: no problems Linux: no problems DSE: W-98: minor problem (postscans) DRW: DOS: Crashed three times on BOOT-ZOO; excluded from this part of the test (no results) Crashed twice on u:\sod\mronon\a\aesa\4443\exb_000_.exe Crashed on u:\sod\mronon\w\59w\ortsib\neg\exb_000_.exe Crashed on u:\sod\mronon\h\cllh\0967\exa_000_.exe; excluded from this part of the test (no results) Crashed on t:\ksir_ces\mronon\b\roodkcab\enilovla.v\bab_001_.bat Crashed three times during Malware-test; excluded from this part of the test (no results) W-98: Crashed more than 3 times at T:\NAJORT\MRONON\B\G-ROODC.AB\RUS\EXB_001_.EXE W-NT: Scanner attempted to clean any virus; this failed as all objects were right-protected. FPR: DOS: no problems W-98: reports were truncated for larger testbeds on R: (Macrovir), T: (Filemal), V: (Poly), W: (vkit) and U: (Filevir); problems were solved in several postscans. W-NT: minor problems (postscans) W-2k: minor problems (postscans) FPW W-98: minor problems (postscans) W-NT: minor problems (postscans) W-2k: minor problems (postscans) FSE W-98: serious problems/several postscans W-NT: Crashed with blue screen when scanning U:\ at file 117; when scan was repeated with memory dump enabled, no crash observed W-2k: Crashed 3 times when scanning File viruses; deinstallation impossible Linux: Problems with installation script (no input control) caused erroneous diagnoses. Otherwise: no problems. INO DOS: Crashed on t:\najort\mronon\b\db-roodk.cab\rvs\exb_001_.exe Crashed during Malware-Tests Displayed "ueberlauf bei Division" =overflow at division after scanning W-98: minor problems (postscans) W-NT: minor problem (postscans) W-2k: minor problems (postscans) MR2: DOS: Crashed two times on v:\oneh3544\onea0000.e\7\one9799.exe Crashed while scanning folder v:\oneh3544\onea0000.e\8\ Crashed three times on BOOT-ZOO; excluded then from this part of the test (no results) Crashed three times on BOOT-ITW; excluded then from this part of the test (no results) Crashed three times on u:\sod\mron\s\kwauqs\258\exa_003_.exe Crashed three times scanning file-zoo during after-test; excluded from this part of the test (no results) W-98: Incomplete logs when scanning R,T,V,W,U; problems were solved in several postscans. NAV DOS: Crashed once during BOOT-test Didnot report all files checked (no OK-Message). Several post-tests were made, with the last one starting the scanner dirctly on every folder. Even after this (rather exhaustive and time consuming) test, it was impossible to determine whether missed files where not identified or completely missed. W-98: Crashed once when scanning U: (filevir), but 2nd run was successfull W-NT: minor problems (postscans) W-2k: minor problems (postscans) NVC: DOS: Crashed once during FILE-tests Crashed while unpacking u:\sod\mronon\z\reppiz\9772\a\exa_003_.exe Crashed while scanning FILE-ZOO Crashed while scanning u:\59W; excluded from this part of the test (no results) Crashed three times with total system reboot on file-malware; excluded from this part of the test (no results) Crashed with memory crash Crashed twice after scanning k:\o97\halfcros\a_doc_\arj.arj with message: "Unpacking Zip.Zip" Crashed after scanning k:\wm\cap\a\arj.arj with message: "Unpacking Zip.Zip"; excluded from this part of the test (no results) Crashed 3 times after scanning L:\file\23w\mronon\f\evolnuf\9904\arj.arj with message: "Unpacking Zip.Zip"; excluded from this part of the test (no results) W-98: seriuous problems (postscans) W-NT: no problems W-2k: Crashed once when scanning File viruses PAV: DOS: PAV crashed twice while loading antiviral databases while scanning Boot; once, it created a 1,5 GB report file W-98: minor problems (postscans) W-NT: minor problems (postscans) W-2k: minor problems (postscans) PER: W-98: Crashed when scanning U:\sod\mronon\t\laivirt\121\b\c0a_000_.com W-NT: Crashed when scanning U:\ at File 128027 W-2k: Crashed 3times when scanning File viruses (after 128,027 files) PRO W-98: Hangs upon scanning W: (vkit) and U: (Filevir) W-NT: minor problems (postscans) W-2k: minor problems (postscans) QHL: W-98: Many crashes for several testbeds observed: FileVir, FileMal, ScriptVir, ScriptVir-ITW. No results where more than 3 crashes. Remark: product detected macro viruses only when packed. RAV W-98: minor problems (postscans) W-NT: no problems W-2k: minor problems (postscans) Linux: no problems SCN DOS: Crashed once (during postscan) on u:\sod\mronon\s\f_gnimae.rcs\729\coa_002_.com W-98: Incomplete logs when scanning R,T,V,W,U; problems were solved in several postscans. W-NT: minor problems (postscans) W-2k: minor problems (postscans) Linux: no problems VSP DOS: Crashed on v:\maltese\ma1000.exe\0\7\ma0762.exe Crashed on v:\oneh3544\one2500.e\8\one5892.exe Crashed on v:\oneh3544\onea0000.e\8\one9854.exe; excluded from poly-test after triple crash (no results) Crashed three times on BOOT-ZOO; excluded then from this part of the test (no results) Crashed three times on BOOT-ITW; excluded then from this part of the test (no results) Crashed on u:\sod\mronon\t\laivirt\821\a\coa_000_.com Crashed during FILE-ZOO test Crashed on u:\sod\mronon\t\laivirt\82\_nwonknu\coa_000_.com; excluded from this part of the test (no results) W-98: Incomplete logs when scanning R,T,V,W,U; problems were solved in several postscans. Comment: when scanned from a batch file, every batch had to started with CRLF; with VTC testbed complexity, this required about 65 explicit CRLFs on U:\file :-) W-NT: minor problems (postscans) W-2k: minor problems (postscans)