Michael Medin_Advanced Windows monitoring - netways

Going beyond the basics

These slides represent the work and opinions
of the author and do not constitute official
positions of any organization sponsoring the
author’s work
This material has not been peer reviewed and
is presented here as-is with the permission of
the author.
The author assumes no liability for any
content or opinion expressed in this
presentation and or use of content herein.

Developer (not system manager)
◦ Quite a big difference
◦ Not working with Nagios
Accidentally ended up in our NOC
◦ Hated BB
The birth of NSClient++
◦ 2003:ish
◦ NSClient sucked (Broke Exchange)
◦ NRPE_NT was to hard to use
The open source of NSClient++
◦ 2004:ish
◦ “just for fun”
The rebirth of NSClient++
◦ 2007
◦ A lot of users emailed me
◦ Got a lot of hits on the webpage
◦ Intense development lead to 0.3.0!

Agents
◦ An overview of the agents
◦ An overview of the protocols
About NSClient++
◦ Quick Introduction
Using NSClient++
◦ Eventlog Checking
◦ WMI (Windows Management Instrumentation)
◦ Scripts
Q/A

An overview of the agents

Agent Age Protocol Licence
SNMP 1990-2008 SNMP Proprietary
NSClient 200x NSClient GPL
NRPE_NT 200x-2006 NRPE GPL
NSClient++ 2004-2008 NRPE,NSClient,NSCA GPL
NC_NET 2004-2008 NSClient,NSCA GPL
“Agentless” WMI N/A
OpMonAgent 2008 NSClient,NRPE GPL

The good:
◦ Standard solution
◦ Hardware extensions
• HP, IBM, DELL, etc...
The bad:
◦ Complex to use
◦ No encryption
◦ Not extensible
◦ Not “popular” on
windows
◦ Needs a lot of
extensions to be
useful

The good:
◦ Very stable
• works “if it works”
◦ Built in checks
• easy to use
The bad:
◦ Requires client install
◦ Outdated
• no longer maintained
◦ Very few checks
• only basic checks
◦ Not extensible

The good:
◦ Extensible
◦ Standard protocol
• same as on *nix
The bad:
◦ Requires client install
◦ No built-in checks
• hard for simple checks
◦ Old

The good:
◦ Lots of features
◦ Built in checks
• easy to use
◦ Can check:
• WMI, EventLog, Scripts
◦ Supports
• NSCA, NSClient
• (no encryption)
The bad:
◦ Requires client install
◦ Written in .net
◦ Not (that) extensible
◦ Requires custom
plug-in (Nagios side)
◦ No Encryption
◦ No NRPE Support

The good:
◦ Built in checks
• easy to use
◦ Can check:
• WMI, EventLog, Scripts
◦ Supports:
• NSCA, NRPE, NSClient
• encryption (NRPE/NSCA)
◦ Very extensible
• Scripts, Lua, modules, etc
The bad:
◦ Requires client install
◦ Hard to use at times
◦ Has had a few bugs
over time

The good:
◦ No client-side install
• (Usually requires a
“proxy”)
The bad:
◦ Proprietary
◦ Not extensible
◦ Limited functionality

A new client (haven't looked into it much)
Seems to be a new version of NSClient (still
written in Delphi)
Script and NRPE support
Not that much new features

I would use either:
◦ NSClient++
◦ NC_NET
I would not use (unless I have a specific reason):
◦ SNMP
• Complex to use
◦ NSClient
• Old and outdated
◦ NRPE_NT
• Hard for some (simple) checks
◦ OpMonAgent
• I don’t see the benefit
◦ “Agentless” WMI
• Limited functionality

An overview of the protocols

Protocol Method Encryption Auth Payload Args
.
NSClient Active No Yes Unlimited 1 Yes 1 Yes 1
NRPE Active Yes No 1024 2 Yes No
NSCA Passive Yes Yes Unlimited Yes Yes
Future 3 Active Yes Yes Unlimited Yes Yes
Multi Commands
1) Protocol supports it but not check_nt
2) NRPE Payload can be extended with recompile of check_nrpe and configured in NSClient++
3) A future protocol I am thinking of adding to NSClient++ (NRPE 3.0)

I would use:
◦ NRPE
• For Active checks
◦ NSCA
• For passive checks
I would not use:
◦ NSClient
• Two words: No encryption!
• If you use it, NEVER use a “secret” password.

Quick Introduction

Internals:
◦ C++ using W32 API
◦ Around 20.000 lines of code (30.000 with comments)
◦ Actively developed (unfortunately only by me)
◦ Modularized design (low on resources)
Runs on:
◦ NT4, w2k, XP, w2k3, Vista, w2k8 ...
◦ X86, x64, IA64 (I lack a compiler for that platform, but it works)
Current Version:
◦ 0.3.4 (out this weekend, maybe not… *grumle*)
◦ Don’t use 0.2.7!
Most features require NRPE
◦ (or custom “NSClient”-client)
Documentation online (WIKI)
◦ http://nsclient.org

Not supported by a commercial entity
◦ Donations welcome
◦ Sponsoring available (contact me for details)
Used by a lot of people (I think)
◦ Impossible to estimate any figures
Website has:
◦ Around 8.000 unique visitors per month
◦ Around 10.000 downloads per month

Starting/Stopping:
◦ nsclient++ /start (net start nsclientpp)
◦ nsclient++ /stop (net stop nsclientpp)
◦ nsclient++ /test
Configuration:
◦ notepad nsc.ini
nsclient++ /test
Is your friend!
Testing:
1.Local (nsclient++ /test)
2.From CLI (check_nrpe ...)
3.From Nagios (add command)
Enabling debug log (always on with /test):
◦ [log]
◦ debug=1
Log File:
◦ nsclient.log (nsc.log)

Eventlog checking

The good:
◦ Powerfull interface
The bad:
◦ Hard to use!
◦ Requires configuration
◦ no out-of-the-box solution!
• (might come in next version)
A lot of theory!
◦ (please dont despair)

Two different filtering strategies
◦ Exclusive filtering (-filter=out)
• If you want all errors (except…)
◦ Inclusive filtering (-filter=in)
• If you only want specific errors
◦ Remember (-filter=new)
• Dont forget this!
• There is an “old” outdated syntax as well

Simplest to start with
By default:
◦ Everything is an error
Produces a lot of noise
◦ False positives
Good if you just want to be warned
Sample (all entries for last 2 days):
◦ CheckEventLog file=application filter=new filter=out
MaxWarn=1 MaxCrit=1 filter-generated=>2d

For advanced use
By default:
◦ Nothing is an error
Easy to make mistakes (and miss errors)
Good if you are only looking for specifics
◦ Raid controllers, active directory, etc...
Sample (all entries for last 2 days):
◦ CheckEventLog file=application filter=new filter=in
MaxWarn=1 MaxCrit=1 filter+generated=

Filter rule
◦ A rule to match against every single line in the
eventlog
Chain
◦ A set of filter rules used when finding errors
◦ Linear (when a rule matches chain is terminated)

Order is important
Start with the rule which will discard the most
items.
◦ filter-generated=>2d

Mode
◦ If the filter is additive, subtractive (or “maybe”)
Type (keyword)
◦ What to match
• Message
• Event category
• Event date
• Etc...
Equal Sign
Operator
◦ =, !=, > < etc...
Value
◦ The value to match

filter+ generated =< 2h

Consider The following rules:
◦ filter-generated=2d
• WRONG! (No equal sign)
◦ filter-generated==2d
• Correct!
equal sign
operator
Always remember the “extra” equal sign!

Type Description
eventType Type of error. (Microsoft says this is severity)
error, warning, info, auditSuccess or auditFailure
eventSource The name of the source of the event.
The program who logged the message
generated Time ago the message was generated.
When it happened
written Time ago the message was written to the log (don’t use)
message
eventID
severity
Filter strings in the message
NOT the entire message!
Filter based on the event id of the log message
error code
Filter based on event severity (I think this is severity)
success, informational, warning or error

Option
file
filter
MaxWarn
MaxCrit
Description
The “eventlog file” to open.
Use multiple file-options to check multiple files.
Set filter mode (out, in, old, new)
Maximum hits before a warning state is issued.
Maximum hits before a critical state is issued.
A list of filter rules to be matched (in order)

Option
truncate
syntax
unique
descriptions
Description
Length of returned data.
Since NRPE (and NSClient++) has a limited capacity this is
important. Usually 1023 is a good value.
How to format the return data
Only “one of each” record will be returned.
(“count” (MaxWarn/MaxCrit) is not affected)
If you plan on using the %message% syntax option.
(Will impact performance “severely”)

CheckEventLog
◦ file=application
◦ file=system
◦ filter=new
◦ filter=out
◦ MaxWarn=1
◦ MaxCrit=1
◦ filter-generated=>2d
◦ filter-severity==success
◦ filter-severity==informational
◦ truncate=1023
◦ unique
◦ descriptions
◦ "syntax=%severity%: %source%: %message% (%count%)“

DEMO

Don’t be discouraged by your first attempt
Remember filter=new (on older versions)
Use the truncate option (1023 is reasonable)
Start small filter your way “up” (whilst testing)
Not so hard once you get down to it.

Start with “everything” and work your way down.
Both System and Application logfile
Reasonable start filter:
◦ filter-generated=>2d
◦ filter-severity==success
◦ filter-severity==informational
Need to customize it for your environment.
A good idea is to use more then one check
1.Check “all errors” (Exclusive)
2.Check “my service” (Inclusive)
Don’t overdo it (eventlog checking is slow)

Would it make sense to check ”new entries”
◦ Just check entries added since last check
◦ Would be faster
◦ Would be “better”
◦ But would you use it

WMI - Windows Management
Instrumentation

The purpose of WMI is to define a non-proprietary set of
environment-independent specifications which allow
management information to be shared between management
applications.
WMI prescribes enterprise management standards and related
technologies that work with existing management standards,
such as Desktop Management Interface (DMI) and SNMP.
WMI complements these other standards by providing a
uniform model. This model represents the managed
environment through which management data from any
source can be accessed in a common way.
…yada yada yada…
In short: Like SNMP but “modern” ☺

Everything
◦ Almost...
There is a lot of objects (tables)
◦ win32 has 450 objects
◦ Various services will add more (AD, SQL Server, ...)
You can:
◦ Read, write and work with “objects”.
◦ (only read via NSClient++)
But you cant:
◦ Check your application

Dangerous!
◦ No security, allows access to a lot of things.
Fairly “unexplored” in NSClient++
Two commands:
◦ CheckWMI
• Check a result set
• NSClient++ does filtering
• Good for check if “more (or less) then n items...”
◦ CheckWMIValue
• Check a specific value
• WMI Does filtering

WQL - WMI Query Language
◦ Based upon SQL
◦ Only select features (no update/insert/delete)
“Tables” are called objects in WMI
◦ An object usually correspond to a logical “types”.
Example:
◦ select * from win32_Processor
• Retrieves everything from the win32_Processor ”object”.

Object
Win32_Fan
Win32_TemperatureProbe
Win32_DiskDrive
Win32_PhysicalMedia
Win32_TapeDrive
Win32_BaseBoard
Win32_BIOS
Win32_IDEController
Win32_MemoryArray
Win32_OnBoardDevice
Win32_Processor
Win32_SCSIController
Win32_USBControllerDevic
e
Win32_NetworkAdapter
Win32_Battery
Win32_PortableBattery
Win32_PowerManagementEve
nt Win32_UninterruptiblePow
erSupply
Win32_Printer
Win32_PrintJob
Description
Represents the properties of a fan temperature device in sensor the computer (electronic system.
thermometer).
Represents a physical disk drive as seen by a computer running the
Windows operating system.
Represents any type of documentation or storage medium.
Represents a tape drive on a computer system running Windows.
Represents a the baseboard attributes (also of known the computer as a motherboard system's basic or system input board). or output
services Represents (BIOS). the capabilities of an Integrated Drive Electronics (IDE)
controller Represents device. the properties of the computer system memory array and mapped
addresses.
Represents common adapter devices built into the motherboard (system
board). Represents a device capable of interpreting a sequence of machine
instructions Represents a on small the computer. system interface (SCSI) controller on a
computer Relates a system USB controller running Windows. and the CIM_LogicalDevice instances connected
to it.
Represents a network adapter on a computer system running Windows.
Represents a the battery properties connected of a to portable the computer battery, system. such as one used for a
notebook computer.
Represents the power capabilities management and events management resulting capacity from power of an state changes.
uninterruptible power supply (UPS).
Represents a device connected to a computer system running Windows that
is capable of reproducing a visual image on a medium.
Represents a print job generated by a Windows-based application.

Object
Description
Win32_SystemDriver Represents the system driver for a base service.
Win32_Directory
Represents a directory entry on a computer system running Windows.
Win32_DiskQuota
Tracks disk space usage for NTFS file system volumes.
Win32_LogicalDisk Represents a data source that resolves to an actual local storage device.
Win32_Volume
Represents an the area file of used storage for handling on a hard virtual disk. memory file swapping on a
Win32_PageFileUsage computer system running Windows.
Win32_NetworkConnection Represents an active network connection in a Windows environment.
Win32_NTDomain
Represents a Windows NT domain.
Win32_PingStatus
Represents the values returned by the standard ping command.
Win32_ComputerSystem
Represents a an computer operating system operating installed in on a Windows a computer environment.
system running
Win32_OperatingSystem Windows.
Win32_Process
Represents a the sequence startup of configuration events a computer of a computer system system running running Windows.
Win32_ProcessStartup Windows.
Win32_ScheduledJob
Represents a executable job scheduled objects using that the are Windows installed NT schedule in a registry service. database
Win32_BaseService maintained by the SCM.
Win32_Service
Represents Describes the a service logon session a computer or sessions system associated running Windows. with a user logged on
Win32_LogonSession to Represents Windows 2000 information Windows about NT. a user account on a computer system running
Win32_UserAccount Windows.
Win32_UserInDomain
Win32_WindowsProductActi Association class
vation
Contains properties and methods related to WPA.
Win32_NTEvent...
Yes you can even check the eventlog!

NSClient++ has support for executing WQL
queries ”as is” and get the result.
◦ nsclient++ -noboot CheckWMI
◦ This does not support namespaces (yet).
◦ (Namespaces are supported via check commands)
Sample use
◦ nsclient++ -noboot CheckWMI select * from win32_Processor

Best way to start
Simple to use...
◦ ...if you know your WMI
A sample query:
◦ CheckWMIValue
• "Query=Select * from win32_Processor“
• MaxWarn=80
• MaxCrit=90
• Check:CPU=LoadPercentage
• ShowAll=long
◦ (a bit like CheckCPU)

Option
MaxWarn
MaxCrit
MinWarn
MinCrit
ShowAll
Query
Check
truncate
AliasCol
Description
The maximum allowed value for the column(s).
The maximum allowed value for the column(s).
The minimum allowed value for the column(s).
The If present minimum will allowed display value information the column(s). even if an item is
not reporting a state.
If The set WMI to query long to will ask display (not stackable, more information. only one query at a
time) A column name to check (if * all columns will be checked)
(this is stackable, so you can compare any number of
columns)
The A column maximum to length be included of the (prefixed) query-result. in the result when
there are errors.

DEMO

Scripts

External Scripts
◦ VB, Perl, Python, ...
◦ .exe files
◦ .net
◦ ...
Lua
◦ Lua is a simple programming language
◦ Used INSIDE NSClient++
◦ Very powerful, and simple
◦ A fairly new feature so feel free to suggest things
Modules
◦ Written in C++, Vb, .net, ...
◦ Very powerful, but “hard”

Configuration:
◦ [modules]
◦ CheckExternalScripts.dll
◦ ...
◦ [External Scripts]
◦ =
• is the command from nrpe
• is the command to execute
• check_es_ok=scripts\ok.bat

Sample Code:
◦ @echo CRITICAL: Everything is not going to be ok!
◦ @exit 2
Exit statuses:
◦ 0OK
◦ 1 Warning
◦ 2 Critical
◦ 3 Unknown

Sample Code:
◦ Wscript.StdOut.WriteLine “Everything might not be ok"
◦ Wscript.Quit(1)
Exit statuses:
◦ 0OK
◦ 1 Warning
◦ 2 Critical
◦ 3 Unknown
NSC.ini syntax:
◦ [External Scripts]
◦ check_vbs=cscript.exe /T:30 /NoLogo scripts\check_vb.vbs

This is exactly as writing ”regular” Nagios
scripts.

Configuration:
◦ [modules]
◦ LUAScript.dll
◦ ...
◦ [LUA Scripts]
◦
• scripts\test.lua
What, no alias
◦ Not needed…

nscp.print('Loading test script...')
nscp.register('check_foo', ‘foo')
function foo (command)
◦ nscp.print(command)
◦ code, msg, perf = nscp.execute('CheckCPU','time=5','MaxCrit=5')
◦ return code, 'hello from LUA: ' .. msg, perf
end

The power of Lua scripts comes from:
◦ The ability to run and modify the result of other
commands
◦ The ability to run ”inside” NSClient++
◦ The simplicity of the language

Questions