Giuseppini / Burnett Microsoft Log Parser Toolkit

A Complete Toolkit for Microsoft's Undocumented Log Analysis Tool
1. Auflage 2005
ISBN: 978-0-08-048939-1
Verlag: Elsevier Science & Techn.
Format: EPUB
Kopierschutz: 6 - ePub Watermark

E-Book, Englisch, 350 Seiten

HIGHLIGHT
Written by Microsoft's Log Parser developer, this is the first book available on Microsoft's popular yet undocumented log parser tool. The book and accompanying Web site contain hundreds of customized, working scripts and templates that system administrators will find invaluable for analyzing the log files from Windows Server, Snort IDS, ISA Server, IIS Server, Exchange Server, and other products.

System administrators running Windows, Unix, and Linux networks manage anywhere from 1 to thousands of operating systems (Windows, Unix, etc.), Applications (Exchange, Snort, IIS, etc.), and hardware devices (firewalls, routers, etc.) that generate incredibly long and detailed log files of all activity on the particular application or device. This book will teach administrators how to use Microsoft's Log Parser to data mine all of the information available within these countless logs. The book teaches readers how all queries within Log Parser work (for example: a Log Parser query to an Exchange log may provide information on the origin of spam, viruses, etc.). Also, Log Parser is completely scriptable and customizable so the book and accompanying Web site will provide the reader with hundreds of original, working scripts that will automate these tasks and provide formatted charts and reports detailing the results of the queries.

* Written by Microsoft's sole developer of Log Parser, this is the first book available on the powerful yet completely undocumented product that ships with Microsoft's IIS, Windows Advanced Server 2003, and is available as a free download from the
Microsoft Web site.

* The book and accompanying Web site contain dozens of original, working Log Parser scripts and templates for Windows Server, ISA Server, Snort IDS, Exchange Server, IIS, and more!

* This book and accompanying scripts will save system administrators countless hours by scripting and automating the most common to the most complex log analysis tasks.

Autoren/Hrsg.

Giuseppini, Gabriele

Burnett, Mark

Inhaltsverzeichnis

1;Cover;1
2;Contents;14
3;Foreword;26
4;Chapter 1 Introducing Log Parser;28
5;Chapter 2 Monitoring IIS;70
6;Chapter 3 Exploring the Windows Event Log;116
7;Chapter 4 Examining Network Traffic and Performance Logs with Log Parser;152
8;Chapter 5 Managing Snort Alerts;174
9;Chapter 6 Managing Log Files;192
10;Chapter 7 Investigating Intrusions;218
11;Chapter 8 Security Auditing;236
12;Chapter 9 Enhancing Log Parser;270
13;Chapter 10 Formatting, Reporting, and Charting;326
14;Chapter 11 Handling Complex Data;360
15;Appendix A SQL Grammar Reference;386
16;Appendix B Function Reference;404
17;Appendix C Input Format Reference;414
18;Output Format Reference;442
19;Index;456
20;Related Titles;465

Leseproben

Chapter 2 Monitoring IIS Scripts and Samples in this Chapter Analyzing Request Details Analyzing Error Requests Analyzing Illegal Requests In This Toolbox IIS 6.0 is the latest version of the Internet Information Services (IIS) offered by Microsoft. It includes a variety of well-known services including Hyper Text Transfer Protocol (HTTP), File Transfer Protocol (FTP), Simple Mail Transfer Protocol (SMTP), and Network News Transfer Protocol (NNTP). Although each of these services handles its own site access activity logging, there are no built-in tools that are able to parse from these plain text log files in order to extract useful information. With additional log sources such as the HTTP error log (HTTPERR) and URL filtering log (URLSCAN, if installed), it is a tedious task to analyze these different log sources and try to understand the activities taking place in the IIS server. This chapter will showcase the magic of Microsoft’s Log parser tool in trying to make any system administrator’s life easier with many creative parsing queries. Monitoring Performance and Usage One of the major advantages of monitoring the site access activity logging is that the log file helps you to keep track of all the details when a particular request is sent to your IIS server. It provides you with who, when, where, and how contents are being accessed. Site access details are vital to understanding the usage and health performance of your IIS server. Information that is logged includes a visitor’s IP address, user account accessing the contents, timestamp of when requests were made, server status reply about the request, the requested resource location, the amount of bytes used in the request, and more. Table 2.1 shows the types of IIS services and supported log formats. Table 2.1 IIS Services and Logging Formats NOTE It is recommended that you configure logs using the World Wide Web Consortium (W3C) extended format. This is the most comprehensive log format in IIS and it allows you to customize different logging property fields. The queries shown in this chapter are based on W3C extended format. Analyzing Request Details Let’s start with some basic information about the IIS site logging feature. By default, the World Wide Web service (w3svc) and Microsoft FTP service (msftpsvc) are configured with W3C extended format. However, not all fields are enabled. SMTP service (smtpsvc) uses the same W3C extended format, but is not enabled by default. For the NNTP service (nntpsvc), logging is not enabled with Microsoft IIS log format as the default. It is recommended that you use the W3C extended format and enable all extended log fields for the maximum amount of access details. With such details, you are able to analyze the requests pattern in a more precise manner. Table 2.2 exhibits the available log fields supported in W3C extended format. Table 2.2 W3C Extended Log Fields NOTE Even though W3C extended log format provides many extended log fields, some fields do not provide useful meaning to certain IIS services. For example, cs-host, cs(User-Agent) cs(Referer) cs(Cookie) will be a NULL value and show as ‘-’ in the Microsoft FTP service log file, as those fields are not related to the service. The default log path for IIS is located at %Windir%/system32/Logfiles/. Each service has its own logging directory using the service name (w3svc, msftpsvc) followed by site ID X. The following lists the default log folder names of IIS services: FTP MSFTPSVCX W3C W3SVCX SMTP SMTPSVCX NNTP NNTPSVCX The X represents the service site ID. For example, a default website site ID is 1, and the w3svcl will be the default log path. This site ID is the identification number generated by IIS when you create a new service site. In previous versions of IIS, the identification numbers are incremental, but with IIS 6.0, the site ID is randomly created by IIS based on the website’s name. It is recommended that you relocate this default log path to a dedicated disk volume and secure it with proper NTFS permissions listed here: Administrators Full Control System Full Control Backup Operator Read If you need to grant access to a user or user’s application to access the log file, you should only grant read permission, as the log file should not be modified at all. When relocating the default log path, you can either place it on a dedicated disk partition or you can configure remote logging; this is another way to help you secure the log file from being modified by attackers. NOTE It is important to note that time logged in W3C extended format uses Greenwich Mean Time (GMT) per W3C specification. For more information on how to enable W3C extended log fields, please refer to www.microsoft.com/resources/documentation/WindowsServ/2003/?standard/proddocs/en-us/log_customw3c.asp. In this section, we will focus on the following log fields used to diagnose performance information for different IIS requests: Bytes Sent and Received (sc-bytes, cs-bytes) The number of bytes IIS uses to accept and reply to a request. This can give you bandwidth usage information about IIS server, allowing you to plan for future network bandwidth upgrades. It can also tell you when something is wrong with your server. For example, if there is a sudden increase in bytes sent or received by the FTP server, you might want to check if there are users uploading or downloading a huge file that could be compromising disk and bandwidth resources. Status Code (sc-status) IIS reply status code tells you whether the request was successfully fulfilled by IIS or why the request failed. Again, this not only helps you in troubleshooting IIS server, but it also give you clues as to whether someone is trying to gain unauthorized access to your IIS server. Time Taken (time-taken) The amount of time IIS took to fulfill the request. This is helpful in determining how long a request was served. For example, if an active server page (ASP) query took more than 2 minutes to complete, you might want to review the coding to determine if there is a problem with the logic flow. Request Content (cs-uri-stem) The requested resource filename, particularly useful in locating most and least popular content in your IIS server. Coupled with other fields, you will be able to identify what content page take a long time to process and the bandwidth occupied by the content. Obtaining Long Running Web Requests The Ch02ToplOWebRequests.sql query returns the top 10 long-running web requests from a particular web log source. It includes details about the requested filename, the number of times it was called, the maximum time spent for the request, and the average bytes sent. The output result is grouped by the requested filename in order of the maximum time taken followed by the number of hits. This query is particularly useful in identifying problematic web requests. For example, if there is an ASP query listed in the result, you need to check the code and figure out why it took such a long time to process. A long running request might affect the server’s entire performance, as the application is taking CPU processing cycles from other requests. The %source% at FROM clause is the log source parameter. It could be ‘ex*.log’, meaning the query will traverse down from the current query directory looking for W3C extended log files, You can also specify the website ID as 3svc/l> to parse the log files located in w3svcl log directory. The %destination% at TO clause lets you specify the output filename for storing the result. In order to get the correct output you need to specify -o:(output) format when running the query. If omitted, NATIVE mode or —o:NAT will be the default output format. To run the query, access a command prompt, navigate to the directory where you installed the query samples, and enter the following: TIP Since IIS 6.0 no longer applies incremental site ID as it did in previous versions, to quickly identify which site ID associated with different websites, you can use the built-in Active Directory Services Interfaces (ADSI) query, iisweb.vbs /query, at the command prompt. This will list all websites registered locally with site name, site id, status, IP, port, and host header name. The query instructs Log parser to read Ch02ToplOWebRequests.sql with the log source (local machine website ID 1) and generate output in NATIVE format to theToplOWebRequests.txt text file. If you do not want to see the process statistics, you can specify -stats:OFF at the end of the command syntax. The following shows the sample output ofToplOWebRequests.txt: The output shows that the potential problematic query includes reg.asp, which took a...