Sileika Pro Python System Administration

1. ed
ISBN: 978-1-4302-2606-2
Verlag: Apress
Format: PDF
Kopierschutz: 1 - PDF Watermark

E-Book, Englisch, 416 Seiten

As time goes on, system administrators are presented with increasingly complicated challenges. In the early days, a team of engineers might have had to look after one or two systems. These days, one engineer can administer hundreds or thousands of systems. System administrators are gradually replacing their tools with more advanced and flexible ones. One of the choices is Python. Structurally, Python is a modern, high-level language with a very clean syntax. Python comes with many built-in libraries that can make automation tasks easier. It also has extensive set of third-party libraries and a very active development community. This flexibility makes Python a good choice for a wide variety of tasks, from prototyping science applications to system maintenance and administration jobs. This book explains and shows how to apply Python scripting in practice. Unlike the majority of the Python books, it will show you how to approach and resolve real-world issues that most system administrators will come across in their careers. In this book, you will find several projects in the categories of network administration, web server administration, and monitoring and database management. In each project, we will define the problem, design the solution, and go through the more interesting implementation steps. Each project is accompanied with the source code of a fully working prototype, which you'll be able to use immediately or adapt to your requirements and environment.

Rytis Sileika has over ten years experience in system administration and system design. Previously, he worked as a system integration engineer and platform designer for enterprises, designing and building telecom platforms. Currently, Rytis administers systems in one of the largest online betting exchanges.

Autoren/Hrsg.

Sileika, Rytis

Inhaltsverzeichnis

1;Title Page;1
2;Copyright Page;2
3;Contents at a Glance;4
4;Table of Contents;5
5;About the Author;14
6;About the Technical Reviewer;15
7;Acknowledgments;16
8;Introduction;17
8.1;Prerequisites for This Book;17
8.2;Structure of This Book;18
8.3;The Example Source Code;20
9;CHAPTER 1 Reading and Collecting Performance Data Using SNMP;21
9.1;Application Requirements and Design;21
9.1.1;Specifying the Requirements;21
9.1.2;High-Level Design Specification;22
9.2;Introduction to SNMP;22
9.2.1;The System SNMP Variables Node;24
9.2.2;The Interfaces SNMP Variables Node;25
9.2.3;Authentication in SNMP;27
9.2.4;Querying SNMP from the Command Line;27
9.3;Querying SNMP Devices from Python;31
9.3.1;Configuring the Application;31
9.3.2;Using the PySNMP Library;33
9.3.2.1;The SNMP GET Command;34
9.3.2.2;The SNMP SET Command;35
9.3.2.3;The SNMP GETNEXT Command;36
9.3.3;Implementing the SNMP Read Functionality;38
9.4;Storing Data with RRDTool;38
9.4.1;Introduction to RRDTool;38
9.4.2;Using RRDTool from a Python Program;40
9.4.3;Creating a RoundRobin Database;40
9.4.4;Writing and Reading Data from the RoundRobin Database;42
9.4.5;Plotting Graphs with RRDTool;45
9.4.6;Integrating RRDTool with the Monitoring Solution;48
9.5;Creating Web Pages with the Jinja2 Templating System;51
9.5.1;Loading Template Files with Jinja2;52
9.5.2;The Jinja2 Template Language;53
9.5.2.1;Accessing Variables;53
9.5.2.2;Flow Control Statements;54
9.5.3;Generating Web Site Pages;55
9.6;Summary;59
10;CHAPTER 2 Managing Devices Using the SOAP API;60
10.1;What Is the SOAP API?;60
10.1.1;The Structure of a SOAP Message;61
10.1.2;Requesting Services with SOAP;61
10.1.3;Finding Information about Available Services with WSDL;63
10.2;SOAP Support in Python;64
10.3;Converting WSDL Schema to Python Helper Module;65
10.4;Defining Requirements for Our Load Balancer Tool;66
10.4.1;Basic Requirements;67
10.4.2;Code Structure;67
10.4.3;Configuration;68
10.5;Accessing Citrix Netscaler Load Balancer with the SOAP API;69
10.5.1;Fixing Issues with Citrix Netscaler WSDL;69
10.5.2;Creating a Connection Object;70
10.5.3;Logging In: Our First SOAP Call;73
10.5.3.1;Finding What Is Being Returned in the Response from a Web Service;75
10.5.3.2;How Is the Session Maintained After We Have Logged In?;76
10.6;Gathering Performance Statistics Data;78
10.6.1;SOAP Methods for Reading Statistical Data and Their Return Values;78
10.6.2;Reading System Health Data;79
10.6.3;Reading Service Status Data;81
10.7;Automating Administration Tasks;84
10.7.1;Device Configuration SOAP Methods;85
10.7.2;Setting a Service State;85
10.8;A Word About Logging and Error Handling;87
10.8.1;Using the Python logging Module;87
10.8.1.1;Logging Levels and Scope;87
10.8.1.2;Configuring and Using the Logger;89
10.8.2;Handling Exceptions;91
10.9;Summary;92
11;CHAPTER 3 Creating a Web Application for IP Address Accountancy;93
11.1;Designing the Application;93
11.1.1;Setting Out the Requirements;93
11.1.2;Making Design Decisions;94
11.1.3;Defining the Database Schema;95
11.1.4;Creating the Application Workflow;97
11.1.4.1;The Search and Display Functions;97
11.1.4.2;The Add Function;98
11.1.4.3;The Delete Function;98
11.1.4.4;The Modify Function;98
11.1.4.5;The System Health Check Function;98
11.1.4.6;The Name Resolution Function;98
11.2;The Basic Concepts of the Django Framework;98
11.2.1;What Is Django?;99
11.2.1.1;The Object-to-Relation Database Mapper;99
11.2.1.2;The Administration Interface;99
11.2.1.3;A Flexible Template System;99
11.2.1.4;Open Source Community Support;99
11.2.2;The Model/View/Controller Pattern;99
11.2.2.1;The Data Model Component;100
11.2.2.2;The View Component;100
11.2.2.3;The Controller Component;100
11.2.3;Installing the Django Framework;100
11.2.4;The Structure of a Django Application;101
11.2.5;Using Django with Apache Web Server;103
11.3;Implementing Basic Functionality;106
11.3.1;Defining the Database Model;106
11.3.2;URL Configuration;109
11.3.3;Using the Management Interface;111
11.3.3.1;Enabling the Management Interface;111
11.3.3.2;Allowing the Administration Plug-in to Manage New Models;113
11.3.4;Viewing Records;114
11.3.5;Using Templates;117
11.3.6;Deleting Records;120
11.3.7;Adding New Records;121
11.3.8;Modifying Existing Records;123
11.4;Summary;124
12;CHAPTER 4 Integrating the IP Address Application with DHCP;125
12.1;Extending the Design and Requirements;125
12.1.1;Extending the Database Schema;127
12.1.2;Additions to the Workflow;128
12.2;Adding DHCP Network Data;128
12.2.1;Defining Data Models;128
12.2.2;Additional Workflows;129
12.2.3;The Add Function;129
12.2.4;The Modify Function;131
12.2.5;The Delete Function;131
12.3;Extending DHCP Configuration with Address Pools;131
12.3.1;The Address Pool Data Model;132
12.3.2;Displaying DHCP Network Details;132
12.3.3;The Add and Delete Functions;134
12.4;Reworking the URL Structure;134
12.4.1;Generating URLs in the Model Class;135
12.4.2;Reverse Resolution of URLs;135
12.4.3;Assigning Names to URL Patterns;136
12.4.4;Using URL References in the Templates;136
12.5;Adding Client Classification;138
12.5.1;Additions to the Data Model;138
12.5.2;Using Template Inheritance;139
12.5.3;Class Rules Management;140
12.5.3.1;Generic Views;140
12.5.3.2;Displaying a List of Objects;140
12.5.3.3;A Detailed View of the Object;141
12.5.3.4;Adding and Modifying New Objects;143
12.5.3.5;Deleting Objects;144
12.6;Generating the DHCP Configuration File;145
12.7;Other Modifications;149
12.7.1;Resolving IPs to Hostnames;149
12.7.2;Checking Whether the Address Is In Use;150
12.8;Summary;153
13;CHAPTER 5 Maintaining a List of Virtual Hosts in an Apache Configuration File;154
13.1;Specifying the Design and Requirements for the Application;154
13.1.1;Functional Requirements;155
13.1.2;High-Level Design;155
13.2;Setting Up the Environment;155
13.2.1;Apache Configuration;156
13.2.2;Creating a Django Project and Application;156
13.2.3;Configuring the Application;157
13.2.4;Defining the URL Structure;158
13.3;The Data Model;159
13.3.1;The Basic Model Structure;160
13.4;Modifying the Administration Interface;164
13.4.1;Improving the Class and Object Lists;164
13.4.1.1;Customizing the Class Names;164
13.4.1.2;Adding New Fields to the Object List;165
13.4.1.3;Reorganizing the Form Fields;167
13.4.2;Adding Custom Object Actions;169
13.5;Generating the Configuration File;171
13.6;Summary;174
14;CHAPTER 6 Gathering and Presenting Statistical Data from Apache Log Files;175
14.1;Application Structure and Functionality;175
14.1.1;Application Requirements;176
14.1.2;Application Design;176
14.2;Plug-in Framework Implementation in Python;176
14.2.1;The Mechanics of a Plug-in Framework;177
14.2.1.1;Interface Model;177
14.2.1.2;Plug-in Registration and Discovery;178
14.2.2;Creating the Plug-in Framework;179
14.2.2.1;Discovery and Registration;179
14.2.2.2;Defining the Plug-in Modules;182
14.3;Log-Parsing Application;183
14.3.1;Format of Apache Log Files;183
14.3.2;Log File Reader;185
14.3.3;Calling the Plug-in Methods;189
14.3.3.1;Tagging the Plug-in Classes;190
14.3.3.2;Plug-in Methods and the Call Mechanism;190
14.4;Plug-in Modules;193
14.4.1;Installing the Required Libraries;193
14.4.1.1;Using the GeoIP Python Bindings;194
14.4.2;Writing the Plug-in Code;195
14.5;Summary;196
15;CHAPTER 7 Performing Complex Searches and Reporting on Application Log Files;197
15.1;Defining the Problem;197
15.1.1;Why We Use Exceptions;200
15.1.2;Are Exceptions Always a Bad Sign?;200
15.1.3;Why We Should Analyze Exceptions;200
15.2;Parsing Complex Log Files;202
15.2.1;What Can We Find in a Typical Log File?;202
15.2.2;The Structure of an Exception Stack Trace Log;203
15.3;Handling Multiple Files;205
15.3.1;Handling Multiple Files;205
15.3.2;Using the Built In Bzip2 Library;209
15.3.3;Traversing Through Large Data Files;209
15.3.4;What Are Generators, And How Do We Use Them?;209
15.4;Detecting Exceptions;211
15.4.1;Detecting Potential Candidates;211
15.4.2;Filtering Legitimate Exception Traces;212
15.5;Storing Data in Data Structures;213
15.5.1;The Structure of Exception Stack Trace Data;214
15.5.2;Generating an Exception Fingerprint for Unknown Exceptions;214
15.5.3;Detecting Known Exceptions;217
15.5.3.1;The Configuration File;218
15.5.3.2;Parsing XML Files with Python;219
15.5.3.3;Storing and Applying Filters;220
15.5.3.4;The Benefits of a Precompiled Search Over a Plain-Text Search;222
15.6;Producing Reports;222
15.6.1;Grouping Exceptions;223
15.6.2;Producing Differently Formatted Output for the Same Dataset;224
15.6.3;Calculating Group Statistics;224
15.7;Summary;226
16;CHAPTER 8 A Web Site Availability Check Script for Nagios;227
16.1;Requirements for the Check System;227
16.2;The Nagios Monitoring System;228
16.2.1;Nagios Plug-In Architecture;228
16.3;The Site Navigation Check;229
16.3.1;Installing the Beautiful Soup HTML Parsing Library;230
16.3.2;Retrieving a Web Page;230
16.3.3;Parsing the HTML Pages with Beautiful Soup;232
16.3.4;Adding the New Check to the Nagios System;238
16.3.5;Emulating the User Login Process;240
16.4;Summary;245
17;CHAPTER 9 Management and Monitoring Subsystem;246
17.1;Design;246
17.1.1;The Components;246
17.1.1.1;The Monitoring Server;247
17.1.1.2;The Monitoring Agent;248
17.1.1.3;The Sensors;248
17.1.2;The Data Objects;248
17.1.2.1;Configuration;249
17.1.2.2;Performance Readings;249
17.1.2.3;Site Configuration;249
17.1.2.4;Scheduling;249
17.2;The Data Structures;249
17.2.1;Introduction to Data Normalization;250
17.2.2;Configuration Data;252
17.2.3;Performance Data;255
17.2.4;Scheduling;256
17.2.5;Site Configuration;257
17.2.5.1;Representing the Information in an ER Diagram;257
17.3;Communication Flows;258
17.3.1;XML-RPC for Information Exchange;259
17.3.1.1;Structure;259
17.3.1.2;Python Support;260
17.3.2;CherryPy;261
17.4;The Server Process;262
17.4.1;Storing Data in a SQLite3 Database;262
17.4.1.1;Initializing the Database File;262
17.4.2;Actions;267
17.4.2.1;Accepting Sensor Readings;267
17.4.2.2;Supplying a New Configuration;268
17.4.2.3;Providing New Sensor Code;269
17.4.2.4;The Server Health Check;270
17.5;The Scheduler;270
17.5.1;Actions;270
17.5.2;Running Multiple Processes;270
17.5.3;Multithreading, Multiprocessing, and GIL;271
17.5.4;Basic Usage Patterns and Examples;272
17.5.5;Running Methods at Equal Intervals;275
17.5.5.1;A Simple Clock Implementation;275
17.5.6;A Cron-Like Scheduler;278
17.5.7;Ticket Dispatcher;279
17.6;Summary;282
18;CHAPTER 10 Remote Monitoring Agents;283
18.1;Design;283
18.1.1;The Passive Component;283
18.1.2;Architecture;283
18.1.3;Actions;283
18.1.3.1;Accepting a New Configuration;284
18.1.3.2;Upgrading the Sensors;284
18.1.3.3;Submitting Sensor Readings;284
18.2;The Security Model;285
18.3;Configuration;285
18.3.1;The ConfigParser Library;285
18.3.1.1;The File Format;286
18.3.1.2;Using the ConfigParser Class Methods;286
18.3.1.3;The Configuration Class Wrapper;291
18.3.2;The Configuration Class Wrapper;291
18.4;The Sensor Design;295
18.5;Running External Processes;296
18.5.1;Using the subprocess Library;296
18.5.2;Controlling the Running Processes;300
18.5.3;Communicating with External Processes;303
18.5.3.1;Using File Descriptors;303
18.5.3.2;Using File Objects;303
18.5.3.3;Using the Pipe Objects;304
18.5.3.4;Redirecting Standard Error;305
18.6;Automatically Updating Sensor Code;306
18.6.1;Sending and Receiving Binary Data with XML-RPC;306
18.6.2;Working with Files and Archives (TAR and BZip2);307
18.7;Summary;310
19;CHAPTER 11 Statistics Gathering and Reporting;311
19.1;Application Requirements and Design;311
19.2;Using the NumPy Library;311
19.2.1;Installing NumPy;312
19.2.2;NumPy Examples;312
19.2.2.1;Working with Arrays;312
19.2.2.2;Basic Mathematical and Statistical Operations;316
19.2.2.3;Calculating the Mean and Standard Deviation;317
19.2.2.4;Finding the Trend Line of a Dataset;321
19.2.2.5;Reading and Writing Data to Files;323
19.3;Representing Data with matplotlib;323
19.3.1;Installing matplotlib;323
19.3.2;Understanding the Library Structure;324
19.3.3;Plotting Graphs;325
19.3.3.1;Changing the Appearance of the Plot Primitives;326
19.3.3.2;Drawing Bars and Using Multiple Axes;327
19.3.3.3;Working with Text Strings;329
19.3.4;Saving Plots to a File;330
19.4;Graphing Statistical Data;331
19.4.1;Collating Data from the Database;331
19.4.1.1;Displaying Available Hosts;331
19.4.2;Drawing Timescale Graphs;332
19.4.2.1;The Index Page;333
19.4.2.2;Host Details Page;334
19.4.2.3;Graph Collection Pages;337
19.4.2.4;Plotting Performance Graphs;339
19.5;Summary;342
20;CHAPTER 12 Automatic MySQL Database Performance Tuning;343
20.1;Requirements Specification and Design;343
20.1.1;Basic Application Requirements;344
20.1.2;System Design;344
20.2;Modifying the Plug-in Framework;346
20.2.1;Changes to the Host Application;346
20.2.2;Modifying the Plug-in Manager;347
20.3;Writing the Producer Plug-ins;348
20.3.1;Accessing the MySQL Database from Python Applications;348
20.3.2;Querying the Configuration Variables;353
20.3.3;Querying the Server Status Variables;355
20.3.4;Collecting the Host Configuration Data;356
20.4;Writing the Consumer Plug-ins;357
20.4.1;Checking the MySQL Version;358
20.4.2;Checking the Key Buffer Size Setting;360
20.4.3;Checking the Slow Queries Counter;361
20.5;Summary;362
21;CHAPTER 13 Using Amazon EC2/S3 as a Data Warehouse Solution;363
21.1;Specifying the Problem and the Solution;363
21.1.1;The Problem;363
21.1.2;Our Solution;364
21.1.3;Design Specifications;364
21.2;The Amazon EC2 and S3 Crash Course;365
21.2.1;Authentication and Security;365
21.2.1.1;Account Identifier;365
21.2.1.2;Access Credentials;366
21.2.1.3;X.509 Certificates;366
21.2.1.4;EC2 Key Pair;366
21.2.2;The Simple Storage System Concepts;366
21.2.3;The Elastic Computing Cloud Concepts;368
21.2.3.1;Amazon Machine Images and Instances;368
21.2.3.2;Elastic Block Store;371
21.2.3.3;Security Groups;372
21.2.3.4;Elastic IPs and Load Balancers;372
21.2.4;User Interfaces;372
21.3;Creating a Custom EC2 Image;373
21.3.1;Reusing Existing Images;373
21.3.2;Making Modifications;374
21.3.2.1;Install the Additional Packages;374
21.3.2.2;Create and Setup an Elastic Block Store Volume;374
21.3.2.3;Configure the MySQL Instance;375
21.3.3;Bundling the New AMI;376
21.4;Controlling the EC2 Using the Boto Python Module;378
21.4.1;Setting Up the Configuration Variables;378
21.4.2;Initializing the EC2 Instance Programmatically;379
21.4.2.1;Launching the EC2 Instance;380
21.4.2.2;Attaching the EBS Volume;381
21.4.2.3;Mounting the EBS Device;381
21.4.2.4;Starting the MySQL Instance;382
21.4.3;Transferring the Data;383
21.4.4;Destroying the EC2 Instance Programmatically;383
21.4.4.1;Shutting Down the MySQL Instance;383
21.4.4.2;Unmounting the File System;383
21.4.4.3;Detaching the EBS Volume;384
21.4.4.4;Taking a Snapshot of the Volume;384
21.4.4.5;Shutting Down the Instance;384
21.4.4.6;The Control Sequence;385
21.5;Summary;386
22;Index;387