Performance Analysis and Tuning â Part 1 - Red Hat Summit
Performance Analysis and Tuning â Part 1 - Red Hat Summit Performance Analysis and Tuning â Part 1 - Red Hat Summit
Load Balancing●●●●●Scheduler tries to keep all CPUs busy by moving tasks formoverloaded CPUs to idle CPUsDetect using “perf stat”, look for excessive “migrations”/proc/sys/kernel/sched_migration_cost●●Amount of time after the last execution that a task is consideredto be “cache hot” in migration decisions. A “hot” task is less likelyto be migrated, so increasing this variable reduces taskmigrations. The default value is 500000 (ns).If the CPU idle time is higher than expected when there arerunnable processes, try reducing this value. If tasks bouncebetween CPUs or nodes too often, try increasing it.Rule of thumb – increase by 2-10x to reduce load balancingIncrease by 10x on large systems when many CGROUPs areactively used (ex: RHEV/ KVM/RHOS)
Sched_Migration CostRHEL6.3 Effect of sched_migration cost on fork/exitIntel Westmere EP 24cpu/12core, 24 GB mem250.00140.00%200.00120.00%100.00%usec/call150.00100.0080.00%60.00%Percentusec/call default 500ususec/call tuned 4mspercent improvement40.00%50.0020.00%0.00exit_10 exit_100 exit_1000 fork_10 fork_100 fork_10000.00%
- Page 2 and 3: Performance Analysis andTuning - Pa
- Page 4 and 5: Red Hat Enterprise Linux: Scale Up
- Page 6 and 7: Red Hat Enterprise Linux 6Benchmark
- Page 8 and 9: Red Hat Enterprise Linux 6.4 vs Win
- Page 10 and 11: Red Hat Enterprise Linux 6Scheduler
- Page 14 and 15: sched_child_runs_first●●●fork
- Page 16 and 17: 2MB standard Hugepages# echo 2000 >
- Page 18 and 19: Transparent Hugepagesecho never > /
- Page 20 and 21: 32-bitMemory Zones64-bitUp to 64 GB
- Page 22 and 23: Per Node/Zone split LRU Paging Dyna
- Page 24 and 25: Typical System Building BlockMemory
- Page 26 and 27: Four NUMA node system,fully-connect
- Page 28 and 29: Per NUMA-Node ResourcesMemory zones
- Page 30 and 31: zone_reclaim_mode●●●●Contro
- Page 32 and 33: Visualize CPUs via lstopo(from hwlo
- Page 34 and 35: Sample remote access latencies4 soc
- Page 36 and 37: So, what's the NUMA problem?●●
- Page 38 and 39: numastat: compatibility mode# numas
- Page 40 and 41: numastat: per-node meminfo# numasta
- Page 42 and 43: numastat shows aligned guests# numa
- Page 44 and 45: How to manage NUMA manually●●
- Page 46 and 47: numad can help improve NUMA perform
- Page 48 and 49: numad aligns process memory and CPU
- Page 50 and 51: numad usage●●●●numad is int
- Page 52 and 53: To change utilization target● -u
- Page 54 and 55: To get pre-placement advice● -w :
- Page 56 and 57: numad “-w” shell script(the imp
- Page 58 and 59: numad “-w” shell script (advise
- Page 60 and 61: Multiguest Oracle OLTP WorkloadOrac
Load Balancing●●●●●Scheduler tries to keep all CPUs busy by moving tasks formoverloaded CPUs to idle CPUsDetect using “perf stat”, look for excessive “migrations”/proc/sys/kernel/sched_migration_cost●●Amount of time after the last execution that a task is consideredto be “cache hot” in migration decisions. A “hot” task is less likelyto be migrated, so increasing this variable reduces taskmigrations. The default value is 500000 (ns).If the CPU idle time is higher than expected when there arerunnable processes, try reducing this value. If tasks bouncebetween CPUs or nodes too often, try increasing it.Rule of thumb – increase by 2-10x to reduce load balancingIncrease by 10x on large systems when many CGROUPs areactively used (ex: RHEV/ KVM/RHOS)
Change language