CIPUG Morgan King Virtualization

Morgan KingCISSP-ISSAP, CISACompliance Auditor, Cyber SecurityVirtualized Environments: WECC Audit ApproachFebruary 2, 2012Anaheim, CA

3Why is a Compliance Auditor telling me aboutvirtualization?

History• Virtualization technology was firstdeveloped during the 1960’s• Need for multiple users and applications tobe run on one physical machine at sametime• 1967 IBM S/360-676

7IBM System/360-67

What is Virtualization?• Abstraction of hardware toallow multiple “virtualmachines” to co-exist onsingle physical system• The Hypervisor manages VM& hardware interactiono Enforce isolationo Manage resource8

Terminology• Host - Physical server• Guest – Virtual Machine(VM)/Workloadso Hardware Independenceo Isolationo Encapsulation• Hypervisoro Type 2o Type 19

Hypervisor Type 2 - Hosted• Requires a Host OS, installs like anapplication• VMware – Workstation, Fusion• Oracle – VM VirtualBox10

11Hypervisor Type 1Bare Metal/Non-Hosted/Native/Full

Platform Players• VMware – vSphere/(ESXi)• Citrix - XenServer• Microsoft - Hyper-V• RedHat - KVM12

Virtualization in Real World• Test and development environments• Applications• Production and Mission Critical• 16% of workloads running in VMs in 2009o Expected to be ~50% by end of 201213

Benefits• Server consolidation/utilization• Reduce hardware costs• Lower power and cooling costs• Maintain legacy applications• Centralized administration14

Agility – Increased Functionality• Simple/Rapid server provisioning• Multiple OS on single server• Easily scaled up/down/in and out• Reduce disaster recovery and backup time• High Availability15

Complexity• Server Virtualization• Network Virtualization• Storage Virtualization• Data Virtualization• Application Virtualization16

CIP Standards and VirtualizationCan I be compliant with an out of the boxinstallation?17

CIP-002-3 R3Critical Cyber Asset Identification• “…develop a list of associated CriticalCyber Assets essential to the operation ofthe Critical Asset.”18

CIP-002-3Critical Cyber Asset List19

Is Hypervisor in-scope?• Any Hypervisor running a VM determined tobe a CCA brings the Host in as a CCA• In addition ALL VM Cyber Assets on theHost machine are in-scope of CIPStandards20

Mixed-Mode(in-scope)CIP ProtectedNot CIP Protected(out-of-scope)21

Mixed-Mode• Configuration where both inscopeand out-of-scopevirtual Cyber Assets arerunning on the samehypervisor or host• Mixing VMs of different trustlevels is not a recommendedconfigurationCIP ProtectedNot CIP Protected(in-scope)(out-of-scope)22

CIP-005-3Electronic Security Perimeter• Is an ESP identified?• Does every virtualized CCA reside within anESP?• Are any virtualized Cyber Assets used inAccess Control and/or Monitoring of ESP(R1.5)?• Do any Hosts and VMs connect to a non-ESP network?23

1 Hypervisor – 3 Physical NICs5 Virtual NICs3 Virtualswitches3 Physical NICsPhysicalNetworkESP24

25Virtualized Network Map

CIP-007-3 R2Ports and Services• “…Only ports for normal and emergencyoperations are enabled.”• Are ALL virtualized Cyber Assets includingthe Hypervisor included in an annual CyberVulnerability Assessment? (CIP-007-3 R8)26

27Hypervisor Ports and Services

28~# esxcli network ip connection list

VM Guest ports and servicesC:\VM-1\CIPUG>netstat -b -o -a -n > netstat_boan.txtActive ConnectionsProto Local Address Foreign Address State PIDTCP 0.0.0.0:135 0.0.0.0:0 LISTENING 952 C:\WINDOWS\system32\svchost.exeTCP 0.0.0.0:445 0.0.0.0:0 LISTENING 4 [System]TCP 0.0.0.0:6002 0.0.0.0:0 LISTENING 428 [spnsrvnt.exe]TCP 0.0.0.0:7001 0.0.0.0:0 LISTENING 248 [sntlkeyssrvr.exe]TCP 0.0.0.0:7002 0.0.0.0:0 LISTENING 248 [sntlkeyssrvr.exe]TCP 127.0.0.1:1025 0.0.0.0:0 LISTENING 1656 [dirmngr.exe]TCP 127.0.0.1:1029 0.0.0.0:0 LISTENING 2484 [alg.exe]TCP 127.0.0.1:5152 0.0.0.0:0 LISTENING 1764 [jqs.exe]TCP 127.0.0.1:33333 0.0.0.0:0 LISTENING 1856 [PGPtray.exe]TCP 172.16.105.220:139 0.0.0.0:0 LISTENING 4 [System]TCP 127.0.0.1:2111 127.0.0.1:33333 ESTABLISHED 1616UDP 0.0.0.0:7001 *:* 248 [sntlkeyssrvr.exe]UDP 0.0.0.0:500 *:* 700 [lsass.exe]UDP 0.0.0.0:4500 *:* 700 [lsass.exe]UDP 0.0.0.0:445 *:* 4 [System]UDP 127.0.0.1:123 *:* 1084 c:\windows\system32\WS2_32.dllUDP 172.16.105.220:6001 *:* 428 [spnsrvnt.exe]29

CIP-007-3 R3Security Patch Management• “…security patch management program for tracking,evaluating, testing, and installing applicable cybersecurity software patches for all Cyber Assets withinthe Electronic Security Perimeter(s).”• Are all security patches/upgrades for the Hypervisorand VMs assessed for applicability?• Are dormant VMs and templates addressed?• Is a process in place to prevent unpatched virtualizedCyber Assets from being replicated?30

31Update Manager

32Hypervisor patches

33Virtual Machine applicable security patches

34Virtual Machine installed security patches

CIP-007-3 R4Malicious Software Prevention• “The Responsible Entity shall use anti-virus softwareand other malicious software (“malware”) preventiontools, where technically feasible, to detect, prevent,deter, and mitigate the introduction, exposure, andpropagation of malware on all Cyber Assets within theElectronic Security Perimeter(s).”• If not technically feasible submit TFE on Hypervisor• Is a process in place to update Anti-Virus softwareand signatures of all powered-off VMs?35

36VMware vShield Endpoint

CIP-007-3 R5Account Management• “…acceptable use of administrator, shared, andother generic account privileges including factorydefault accounts.”• Are roles and responsibilities defined to enforcesegregation of duties?• Do a limited number of unique users have accessto the management interfaces?• Server, network, storage and security dutiescollapsed37

38Privileged Service Console Access

CIP-007-3 R5.2Acceptable use of root39

CIP-007-3 R5.2.2Shared accounts40

41Virtualization Management Interface

42Are specific roles are defined?

CIP-007-3 R5.1“Need to know”43

CIP-007-3 R5.3Password enforcement44

45Password enforcement

CIP-007-3 R6Security Status Monitoring• “…monitor system events that are related tocyber security.”• Are cyber security events being logged forVMs and Hypervisor?o /var/log/vmware/webAccesso /var/log/secureo /var/log/vmware/esxcfg-firewall.log46

Virtual Machines• Dynamic nature – “VM Sprawl”o How are unauthorized virtual cyber assets prevented?• Virtual Machines should be treated no differently thanphysical machines in terms of -o Segmentationo Physical Securityo Least Privilege Accesso Security Patchedo Subject to Change control/Configuration Managemento Proper Disposal/Redeployment47

48Virtual Machines and Templates

Risks for Virtualized Environments• Lack of visibility• Increased complexity of virtualized systemsand networks• Vulnerabilities in the physical environmentapply to virtual machines• Virtualization-aware malware• Hypervisor creates new attack surface49

Attack Vectors• VM Escape• Hyperjackingo Blue Pillo SubVirto Vitriol• VM Migration attacks50

51VM Migration

52http/https running as who?

53VASTO – Virtualization ASsessment TOolkit

Sum it up!• CIP Standards apply to a virtualized environment asthey would in a physical environment• No one-size-fits-all method or solution to configurevirtualized environments to meet CIP requirements• Virtualization technologies may introduce new risk• Mixed-Mode is not a recommended configuration forsecurity and compliance• Know everything that is going on in your virtualizedenvironment54

Understand the technology• The datacenter becomes much moredynamic and flexible• Adapt existing security processes• Adapt existing security solutions• Misconfiguration is #1 Risko Vendor Trainingo NIST Special Publication 800-125• Guide to Security for Full Virtualization Technologies55

References56• http://history.cs.ncl.ac.uk/anniversaries/40th/images/ibm360_672/slide24.html• http://www.gartner.com/it/page.jsp?id=1211813• http://www.vmware.com/files/pdf/partners/security/mcafee-key-security-ent-arch-wp.pdf• http://www.vmware.com/pdf/vsphere4/r40/vsp_40_intro_vs.pdf• http://www.vmware.com/products/vshield-endpoint/overview.html• http://www.oocities.org/surfboardart/Mainframe/microcode.htm• http://nvd.nist.gov/scap/docs/2008-conf-presentations/day2/VMware_Security_NIST.pdf• http://www.isaca.org/Knowledge-Center/Research/ResearchDeliverables/Pages/Virtualization-Benefits-and-Challenges.aspx• http://www.isaca.org/Knowledge-Center/Research/ResearchDeliverables/Pages/VMware-Server-Virtualization-Audit-Assurance-Program.aspx• http://www.sans.org/reading_room/analysts_program/VMware_ITAudit_Sep09.pdf• http://www.rationalsurvivability.com/presentations/FourHorsemen.pdf• http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1024500• http://www.virtuallyghetto.com/2010/07/esxi-41-major-security-issue.html• http://www.vmware.com/technical-resources/virtual-networking/networking-basics.html• http://www.fyrmassociates.com/pdfs/Stealing_Guests_The_VMware_Way-ShmooCon2010.pdf• https://www.pcisecuritystandards.org/documents/Virtualization_InfoSupp_v2.pdf• http://jreypo.wordpress.com/tag/vsphere-cli/• http://www.shmoocon.org/• http://csrc.nist.gov/publications/nistpubs/800-125/SP800-125-final.pdf

Questions?Morgan King CISSP-ISSAP, CISACompliance Auditor, Cyber Securitymking@wecc.biz