Detecting MySQL IO problems on Linux at different ... - Percona

<strong>Detecting</strong> <strong>MySQL</strong> <strong>IO</strong> <strong>problems</strong> on 

Linux at different abstraction 

layers 

Nickolay Ihalainen 

Percona Live London 2011

● Dataflow layers 

● OS tools 

Agenda 

● <strong>MySQL</strong> instrumentation 

● Inside InnoDB: story of one insert 

www.percona.com

● Hardware Level 

Layers 

● On disk queue NCQ, TCQ and cache 

● RAID controllers queue, caches 

● OS Level 

● BLK device level 

● <strong>IO</strong> scheduler 

● File system 

● Page cache 


● A<strong>IO</strong> 

Software Layers 

● Kernel A<strong>IO</strong> (for win and Linux with O_DIRECT) 

● Glibc A<strong>IO</strong> (not used) 

● Normal read/pread and write/pwrite 

● Directories operations 

● <strong>MySQL</strong> 

● Data, InnoDb simulated A<strong>IO</strong> 

● Dictionaries (table cache and dictionary cache) 


● vmstat 

High-level tools 

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- 

r b swpd free buff cache si so bi bo in cs us sy id wa 

1 0 0 1216856 134096 1882012 0 0 0 26640 714 1215 3 2 88 7 

0 0 0 1216820 134096 1882360 0 0 0 0 501 697 2 0 98 0 

● iostat 

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz 

await r_await w_await svctm %util 

sda 0.32 0.70 1.26 3.19 0.03 0.08 51.20 0.02 

4.22 1.77 5.19 0.42 0.19 

● pt-diskstats 


<strong>MySQL</strong> <strong>IO</strong> tracing features 

● Slow log, high verbosity with Percona Server 

● Innodb status, look for pending* 

● information_schema.innodb_io_pattern 

● pt-mext 


● PMP 

Debug <strong>IO</strong> 

● strace, syscalls per sec and req per sec 


Block level read load 

blktrace /dev/sdg -a issue -a complete -w 3600 -o - | blkiomon -I 10 -h - 

sizes read (bytes): num 3203, min 4096, max 262144, avg 20504.3 

d2c read (usec): num 3203, min 85, max 36013, avg 2982.8 

throughput read (bytes/msec): num 3203, min 147, max 265059, avg 71893.0 

sizes histogram (bytes): 

0: 0 1024: 0 2048: 0 4096: 472 

8192: 19 16384: 1719 32768: 929 65536: 30 

131072: 37 262144: 10 524288: 1 1048576: 0 

d2c histogram (usec): 

64: 0 128: 40 256: 1685 512: 322 

1024: 59 2048: 41 4096: 144 8192: 414 

16384: 459 32768: 52 65536: 1 131072: 0 


Block level write load 

blktrace /dev/sdg -a issue -a complete -w 3600 -o - | blkiomon -I 10 -h - 

sizes write (bytes): num 1141, min 4096, max 294912, avg 6748.9 

d2c write (usec): num 1141, min 192, max 54462, avg 1835.4 

throughput write (bytes/msec): num 1141, min 75, max 175021, avg 25573.3 

sizes histogram (bytes): 

0: 0 1024: 0 2048: 0 4096: 771 

8192: 210 16384: 228 32768: 15 65536: 7 

131072: 8 262144: 10 524288: 1 1048576: 0 

d2c histogram (usec): 

64: 0 128: 2 256: 989 512: 87 

1024: 10 2048: 8 4096: 1 8192: 62 

16384: 25 32768: 57 65536: 9 131072: 0 


File System level 

● perf stat -e 'ext4:*' -p `pgrep -x mysqld` 

1,303 ext4:ext4_mark_inode_dirty 

3,357 ext4:ext4_da_write_begin 

3,357 ext4:ext4_da_write_end 

634 ext4:ext4_da_writepages 

1,328 ext4:ext4_da_write_pages 

634 ext4:ext4_da_writepages_result 

634 ext4:ext4_sync_file_enter 

634 ext4:ext4_sync_file_exit 

10 ext4:ext4_da_reserve_space 

580 ext4:ext4_ext_map_blocks_enter 

580 ext4:ext4_ext_map_blocks_exit 

36.567472319 seconds time elapsed 


File System level cont. 

● systemtap, tools like disktop.stp, io_submit.stp, 

and more 


Understand how mysql & innodb 

works 

● Use non-stripped sources with -g enabled 

● GDB 

● Full strace -ff listing 

● systemtap with ubacktrace 


● More RAM 

● Avoid randomness 

Avoid high <strong>IO</strong> load 

● Random primary keys 

● Useless updates/inserts 

● Redundant indexes and fields 

● Decrease number of rows fetched by using 

correct indexes 

● More disks in raid and/or SSD disks 

● Always test RAID adapters 


Benchmarking 

● Bare-metal disk with dumb RAID 

● Raid adapter (stripe size and etc) 

● File system 

● Tools 

● dd, dbench 

● iozone 

● sysbench 

● Replaying general log/queries from tcpdump 

● Mysql replication slave 


● Preparation 

#!/usr/bin/env stap 

Story of one INSERT 

# use: innodbio.stp --vp 00001 -d /usr/sbin/mysqld --ldd -x `pgrep -x mysqld` | 

c++filt|tee -a innodbio.log 

probe process("mysqld").function("os_file*") { 

printf("%d %d %s\n",gettimeofday_s(),tid(), probefunc() ); 

print_ustack(ubacktrace()); printf("\n"); 

} 

probe process("mysqld").function("os_aio*") { 

printf("%d %d %s\n",gettimeofday_s(),tid(), probefunc() ); 

print_ustack(ubacktrace()); printf("\n"); 

} 



● A long way to open table 

1319306314 7273 os_aio 

0x7f8bd7113b60 : os_aio+0x0/0x550 [/usr/sbin/mysqld] 

0x7f8bd70d813d : fil_io+0x1dd/0x4b0 [/usr/sbin/mysqld] 

0x7f8bd70bc3f5 : buf_read_page_low+0xd5/0x210 [/usr/sbin/mysqld] 

0x7f8bd70bc755 : buf_read_page+0x225/0x3e0 [/usr/sbin/mysqld] 

0x7f8bd70b33d7 : buf_page_get_gen+0x147/0xa50 [/usr/sbin/mysqld] 

0x7f8bd70a0d69 : btr_cur_search_to_nth_level+0x399/0xf40 [/usr/sbin/mysqld] 

0x7f8bd7178464 : btr_pcur_open_on_user_rec+0x64/0x2e0 [/usr/sbin/mysqld] 

0x7f8bd70c77b1 : dict_load_table +0x1f1/0x2450 [/usr/sbin/mysqld] 

0x7f8bd70c65a0 : dict_table_get+0x160/0x190 [/usr/sbin/mysqld] 

0x7f8bd709cd0d : ha_innobase::open(char const*, int, unsigned int)+0x1cd/0x6a0 

[/usr/sbin/mysqld] 



● A long way to open table 

1319306314 7273 os_aio 



0x7f8bd70bc3f5 : buf_read_page_low+0xd5/0x210 [/usr/sbin/mysqld] 

0x7f8bd70bc755 : buf_read_page+0x225/0x3e0 [/usr/sbin/mysqld] 


0x7f8bd70a0d69 : btr_cur_search_to_nth_level+0x399/0xf40 [/usr/sbin/mysqld] 

0x7f8bd7178464 : btr_pcur_open_on_user_rec+0x64/0x2e0 [/usr/sbin/mysqld] 

0x7f8bd70ca5ba : dict_load_indexes+0x26a/0x13e0 [/usr/sbin/mysqld] 

0x7f8bd70c82d4 : dict_load_table+0xd14/0x2450 [/usr/sbin/mysqld] 

0x7f8bd70c65a0 : dict_table_get+0x160/0x190 [/usr/sbin/mysqld] 


0x7f8bd6fde56d : handler::ha_open(st_table*, char const*, int, int)+0x3d/0x180 



● A long way to open table: statistics 

1319306314 7273 os_file_create_simple_no_error_handling 

0x7f8bd7111650 : os_file_create_simple_no_error_handling+0x0/0x1a0 


0x7f8bd70cf8c0 : fil_node_open_file+0x170/0x440 [/usr/sbin/mysqld] 

0x7f8bd70cfbe9 : fil_node_prepare_for_io+0x59/0x190 [/usr/sbin/mysqld] 

0x7f8bd70d8986 : fil_space_get_size+0xe6/0x140 [/usr/sbin/mysqld] 

0x7f8bd70bc5ce : buf_read_page+0x9e/0x3e0 [/usr/sbin/mysqld] 


0x7f8bd7170c0f : btr_get_size+0x1af/0x330 [/usr/sbin/mysqld] 

0x7f8bd70c631b : dict_update_statistics_low +0x4b/0x170 


0x7f8bd70c6554 : dict_table_get+0x114/0x190 [/usr/sbin/mysqld] 





● Statistics is not finished: 

● os_file_pread 

● os_file_close 

● os_file_create 

● os_file_lock 

● os_aio reads 

● Insert buffer operations: ibuf_merge_or_delete_for_page 

● Reserved pages fseg_n_reserved_pages 


● Start transaction 

1319306314 7273 os_aio 



0x7f8bd715f34b : trx_sys_flush_max_trx_id+0x9b/0xe0 


0x7f8bd7161e05 : trx_start_low+0xe5/0x1c0 [/usr/sbin/mysqld] 

0x7f8bd7161f4f : trx_start+0x6f/0xe0 [/usr/sbin/mysqld] 

● Read required pages before transaction start 

● Save rowid in header file page 

dict_hdr_flush_row_id 


● Commit 


● XA two phase fixation 

1319306314 7273 os_aio 



0x7f8bd70ffd6b : log_group_write_buf+0x1ab/0x370 [/usr/sbin/mysqld] 

0x7f8bd710037e : log_write_up_to+0x44e/0x7d0 [/usr/sbin/mysqld] 

0x7f8bd71644c9 : trx_prepare_off_kernel+0x2d9/0x330 [/usr/sbin/mysqld] 

0x7f8bd7164589 : trx_prepare_for_mysql+0x69/0x180 [/usr/sbin/mysqld] 

0x7f8bd709830b : innobase_xa_prepare(handlerton*, THD*, bool)+0xeb/0x150 


● And the same for normal logs 



● Periodic operations: 

● Write dirty pages 

1319306314 7273 os_aio 

1319306324 7270 os_aio 



0x7f8bd70b69ba : buf_flush_buffered_writes+0x26a/0x620 


0x7f8bd70b75f7 : buf_flush_batch+0x157/0x11f0 [/usr/sbin/mysqld] 

0x7f8bd714b666 : srv_master_thread+0xbe6/0xc60 [/usr/sbin/mysqld] 

● And log_checkpoint is called from the same 

thread 


Thank You to Our Sponsors 

Platinum Sponsor 

Gold Sponsor 

Silver Sponsors 


Percona Live London Sponsors 

Exhibitor Sponsors 

Friends of Percona Sponsors 

Media Sponsors 


Annual <strong>MySQL</strong> Users Conference 

Presented by Percona Live 

The Hyatt Regency Hotel, Santa Clara, CA 

April 10th-12th, 2012 

Featured Speakers 

Mark Callaghan, Facebook 

Jeremy Zawodny, Craigslist 

Marten Mickos, Eucalyptus Systems 

Sarah Novotny, Blue Gecko 

Peter Zaitsev, Percona 

Baron Schwartz, Percona 

The Call for Papers is Now Open! 

Visit www.percona.com/live/mysql-conference-2012/ 


nickolay.ihalainen@percona.com 

We're Hiring! www.percona.com/about-us/careers/

www.percona.com/live

Detecting MySQL IO problems on Linux at different ... - Percona

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?