Logical Domains - UKUUG

Logical Domains
Liam Merwick
Sun Microsystems, Inc.

Contents
• Background
• Components
• Features
• Architecture
• Configuration Examples
• Q & A
Page: 2

Page: 3
Background

Consolidation
• Conventional Unix server utilisation is 7 to 15%
• Data center compression
> Goal: Increase utilisation of equipment
10% Utilisation
20%
Time
10%
Page: 4

Solutions from Sun
Hard Partitions Virtual Machines OS Virtualisation Resource Mgmt.
App
Server
Database
Identity
Server
File
Server
Web
Server
Mail
Server
Calendar
Server
Database
Web
Server
SunRay
Server
Database
App
Server
App
OS
Server
Multiple OSes Single OS
Trend to flexibility
Dynamic System Domains
SPARC Logical Domains
Solaris Xen
Solaris Containers
(Zones + SRM)
Trend to isolation
Solaris Resource Manager
(SRM)
VMware ESX
Microsoft Virtual Server
Solaris Containers
for Linux Applications
Solaris Trusted Extensions
Page: 5

Virtualisation for SPARC platforms
• New “sun4v” architecture
Operating
System
Solaris X
update
(genunix)
sun4u code
Solaris X
(genunix)
Solaris X (sun4v)
US-Z CPU
code
sun4v
interface
CPU “Z”
SPARC hypervisor
SPARC CPU
Page: 6
Platform

Virtual Machine for SPARC
• Thin software layer between OS and
platform hardware
• Hypervisor + sun4v interface
• Virtualises machine HW and isolates OS from
register-level
• Delivered with platform not OS
• Not itself an OS
sun4v virtual machine
User
App
OpenBoot
User
App
Solaris
Hypervisor
User
App
SPARC hardware
stable
interface
“sun4v”
Page: 7

Logical Domains
• Partitioning capability
> Create virtual
machines each with
sub-set of resources
> Protection & Isolation
using HW+firmware
combination
LDom 1
Solaris 10
App
App
LDom 2
Solaris 10
App
App
Zone
LDom 3
Solaris
Express
App
App
Zone 1
App
App
Zone 2
Hypervisor
Hardware
Shared CPU,
Memory, IO
CPU
Mem
CPU
Mem
CPU
Mem
CPU
I/O
Page: 8

Logical Domains Technology
• Virtualisation and partitioning of machine resources
> Each domain is a full virtual machine, with a dynamically reconfigurable subset
of machine resources, and its own independent OS
> Protection & isolation via SPARC hardware and LDoms Hypervisor firmware
LDom A
LDom B LDom C LDom D
OS
Environment
of choice
Linux
FreeBSD
LDoms
Hypervisor
Platform
Hardware
CPU CPU CPU CPU CPU CPU CPU CPU
Memory
Memory Memory Memory
I/O I/O
Page: 9

Coupled with Throughput Computing
• Industry trend to many cores & strands per chip
> Most applications don't scale well
> Even Solaris doesn't scale perfectly
• Ability to dynamically partition system matches
throughput computing model
> Can run multiple services and OS instances on same
box / chip
> More flexibly be able to utilise available compute power
Page: 10

Page: 11
Components

Page: 12
Components – Hardware & Software
• Hardware
> UltraSPARC-T1 (Niagara) family of CMT processors
> T1000 Netra CP3060
> T2000 Netra T2000
• Operating System
> Solaris 10 11/06 (U3) + patches
> OpenSolaris (build 57+ recommended)
• Firmware
> 6.4.0
• LDoms Manager Software
> 1.0 (Solaris package - available as Free download)

Page: 13
Features

Fundamentals
• Each virtual machine should appear as an entirely
independent machine
> own kernel, patches, tuning parameters
> own user accounts, administrators
> own disks
> own network interfaces, MAC & IP addresses
> Start, stop and reboot independently of each other
• Paravirtualised OS
Page: 14

Features
• Add and remove cpus while OS is running
• Protection and isolation via SPARC hardware and
LDoms Hypervisor firmware
• Service domains for I/O
> direct I/O access
Page: 15

Features – UltraSPARC-T1 specific
• Up to 32 LDoms per box
> CPU threads individually assignable to different LDoms
• Memory assignable to with an 8KB granularity
• Two PCI busses independently assignable to the
same or different LDoms.
• Guest Domains can be configured, started and
stopped independently
> Without requiring a power-cycle of machine
Page: 16

Page: 17
Architecture

Architecture
• Logical Domains overview
• Hypervisor
• LDoms Manager
• Virtualised I/O
• Dynamic Reconfiguration (DR)
Page: 18

Hypervisor Support
• Hypervisor software is responsible for maintaining
separation between domains
> Using extensions built into a sun4v CPU
• Also provides Logical Domain Channels (LDCs) so
that domains can communicate with each other
> Mechanism by which domains can be virtually networked
with each other, or provide services to each other
Page: 19

LDoms Manager
• One Manager per host HV
> Controls Hypervisor and all its LDoms
• Exposes control interface via CLI
• Maps Logical Domains to physical resources
> Constraint engine
> Heuristic binding of LDoms to resources
> Assists with performance optimisation
> Assists in event of failures / blacklisting
Page: 20

Logical Domains
File
Server
Web
Server
Mail
Server
Application
OS
Server
Page: 21

Direct I/O
Logical Domain
• Traditional model
> Existing drivers and devices
continue to work
App
App
App
Device Driver
/pci@B/qlc@6
Logical Domain
owns PCI root
and tree
Nexus Driver
Privileged
/pci@B
Hyper
Privileged
Hypervisor
Virtual Nexus I/F
Hardware
I/O
Bridge
I/O MMU
PCI
Root
PCI-Express
Page: 22

Virtualised I/O
Privileged
Logical
Domain A
App
App
App
App
Virtual Device
Driver
Service
Domain
Virtual Device
Service
Device Driver
/pci@B/qlc@6
Nexus Driver
/pci@B
Hyper
Privileged
Hypervisor
Domain Channel
Virtual Nexus I/F
Hardware
I/O MMU
PCI
Roo
t
I/O
Bridge
PCI B
Page: 23

Virtual (Block) Disk device
Logical Domain
1
App
App
App
App
Logical
Domain 2
App
App
App
App
Logical
Domain 3
App
App
App
App
Service
Domain
vDisk
Server
vDisk
Server
vDisk
client
Driver
vDisk
client
driver
vDisk
client
driver
Device-
Driver
Hypervisor
Virtual SAN 1
Virtual SAN 2
I/O Bridge
FC-AL
I/F
Page: 24

Virtual Ethernet device
Logical
Domain 1
App
App
App
App
Logical
Domain 2
App
App
App
App
Logical
Domain 3
App
App
App
App
Service
Domain
V-Ether
Switch
V-Ether
Switch
V-Ether
Driver
V-Ether
Driver
V-Ether
Driver
Device-
Driver
Hypervisor
Virtual LAN 1: 192.168.0/24
Virtual LAN 2: 63.24/16
I/O Bridge
Gb
Ether I/F
Page: 25

Dynamic Reconfiguration (DR)
• Ability to dynamically grow or shrink compute
capacity of an LDom on demand
• No need to re-boot Solaris
• Simply add / remove:
> CPUs
> Memory (future)
> I/O (future)
• Improve utilisation by balancing resources between
LDoms
Page: 26

LDoms Manager
Logical
Domain 1
App
App
App
App
Logical
Domain 2
App
App
App
Logical
Domain 3
App
App
App
Service
Domain
LDoms
Manager
OS DR
Stack
OS DR
Stack
OS DR
Stack
Hypervisor
HV DR
Stack
Page: 27

Example: vCPU reconfiguration
Logical
Domain 1
App App
Logical
Domain 1
App App
Logical
Domain 2
App App
V
V
V
V
V
P
P
Hypervisor
Hypervisor
P P
P
Example command line operations:
% ldm remove-vcpu 1 Logical_Domain_1
% ldm add-vcpu 1 Logical_Domain_2
Page: 28

Page: 29
Configuration
Examples

Configuring a machine to run LDoms
• Make resources available (remove from control domain)
primary#> ldm set-mau 2 primary
primary#> ldm set-vcpu 8 primary
primary#> ldm set-mem 8g primary
• Configure services
primary#> ldm add-vcc port-range=5000-5100 primary-vcc0 primary
primary#> ldm add-vds primary-vds0 primary
primary#> ldm add-vsw net-dev=e1000g0 primary-vsw0 primary
Page: 30

Adding another domain (I)
• Add disk device
primary#> ldm add-vdsdev /dev/dsk/c1t0d0s2 vol1@primary-vds0
• Adding resources
primary#> ldm create domain1
primary#> ldm add-mem 2g domain1
primary#> ldm add-vcpu 4 domain1
primary#> ldm add-vdisk vdisk1 vol1@primary-vds0 domain1
primary#> ldm add-vnet vnet1 primary-vsw0 domain1
• Show domain constraints
primary#> ldm list -l domain1
Page: 31

Adding another domain (II)
• Booting the newly created domain
primary#> ldm bind domain1
primary#> ldm start domain1
• Connect to the console
primary#> telnet localhost 5000
Page: 32

More information
• BigAdmin Portal
> http://www.sun.com/bigadmin/hubs/ldoms/
• Sun Blueprint
> Beginners Guide to LDoms: Understanding and Deploying
Logical Domains
> http://www.sun.com/blueprints/0207/820-0832.pdf
Page: 33

Logical Domains
Liam Merwick
Liam.Merwick@Sun.COM
LDoms Development
SPARC Platform Software
34