Domino Memory Management NSF Buffer Pool - Nash!

Domino Server Performance BestPractices - BP113Kim GreenePresidentKim Greene Consulting, Inc.Daniel NashedPresidentNashed!Com

A holistic approach to PerformanceManagementYou need to take care about all levelsHardwareDisk Drives, Memory, CPU, NetworkOperating SystemsParameters, Subsystems, ...Notes InfrastructureServer, Server settings, Transaction Log, ...Databases & Database SettingsApplicationsUse the right techniques designing & coding your applications

What Appears to Be the Problem?This is always Step #1Need clear understanding of performance problemAre all users affected?When is problem occurring?What applications are running on system?How long has performance problem been occurring?Is the problem perceived response time problems?Is the problem defined as high CPU utilization?

Do We Have a Healthy System?Step #2 involves determining if overall system is healthyA number of tools to use for this stepNeed to look at each of the following:CPUMemoryI/ONetworkSome tools are "free" and some are "fee-based"

Overview Domino, OS and HardwareScript,@Formulas,JavaNotes ClientServertasksNotes C-APINSFDbOpen, NIFFindByKey, ...Notes Core ServicesNSF, NIF, SEC, FT, ... @FunctionsOperating System and ServicesHardwarePlatform independent Notes C-APIcalls based on lower level corefunctions & routinesNotes Core services whichencapsulate Notes and OS-specificfunctionalityPlatform specific calls and serviceslike NLS, memory management,TCP/IP, file servicesMemory, Disk, CPU, other I/Odevices

"Best platform for Domino?" really means"Which platform fits my environment?"Domino scales to the limit of the underlying platformLinux is a good choice for small/medium workloadAIX, Solaris & iSeries are the right choice for larger systemsA lot customers use Linux in DMZ for security reasons(IPTables stateful packet filter in kernel, SSH, ...)Mixed environment with Linux for smaller servers andUnix for larger machinesWith current Kernel 2.4 Linux is an excellent alternate platformfor Windows based Intel machinesD7 with 2.6 kernel will really rock (IOCP support via sys-epoll)!AIX 5.x on Regatta Servers is great technology forconsolidated server infrastructures - we have alreadycustomers using it.

Get the most out of your hardwareSome resources within Domino are limitedAgent Manager, Indexer, Task Scheduler take a lot of time to gothru all databasesControl of concurrent access to databases needs to becoordinated (semaphores) also for backend access (NIF, ...)It makes sense to use partitioning to get most out ofyour hardwareIn D6 Partitioning is part of the normal licenseUse PercentSysAvailResources=nEven large boxes only scale if partitionedDifferent platforms support different memory sizesDon't use memory/buffer tuning if not really needed!

Memory (RAM)Memory is "relatively" cheapbut can enhance performance a lotDomino uses memory for internal caching/processing (e.g. NSF Buffers)On Unix check the security settings for resources limitationse.g. AIX: /etc/security/limitsBe aware of differences between shared memory (pools etc.)and local memory (per servertask)Windows can only use about 2,3 GB Memory for all partitions ona single boxLinux can use up to 4 GB per box on multiple partitionsAIX and Solaris 32Bit application running on 64Bit OSeach partition can benefit from an own 32Bit address space

NetworkingMake sure you have the right network infrastructureCheck bottlenecks between different parts of the networkUse switched networks and maybe a backbone for your serversDo not use automatic speed/duplex settings for clients andservers!!Make sure you use the right networking parameters for yourenvironment like MTU (Domino will use it see e.g. LKB #192867for Windows and search for iSeries and Unix)Remove all network services/protocols you do not usenetstat -an is your friend

Domino Performance "Features"Use transaction log and View Rebuilde.g. view_rebuild_directory=c:\view_rebuild_dir\Disable Unread Marks for system databases and otherlarge databasesthis can really speed up opening databasesEnable multiple indexersnotes.ini updaters=nDiscard View indexes if you don't use themDEFAULT_INDEX_LIFETIME_DAYS=days

Performance TipsLarge numbers of deletion stubs can make databases slowBe careful with FT index because it can block updatersfrom indexing other databasesDon't schedule agents too oftenDisable tasks you do not needsched task uses a lot of resourcesHave separate connection documents for mail andreplicationthis can help to finetune replicationalso replicate different priorities at different times

"Extremely inefficient" Search Operations"Warning: Agent is performing full text operations on database 'xyz'which is not full text indexed. This is extremely inefficient."(STR_AGENT_NOFULLTEXTINDEX)Agents with Search Queries always use FT indexIf databases have no FT index a temporary on-the-fly FT index is createsD6: FT_FLY_INDEX_OFF=1 (TN #1161983) replaces old R5 zOS platformoption: OS390_508_FTFLYMODE=1Causes Agents using search operations on not FT indexed databases tofailHappens only in agents, Notes Client UI (View Search) uses adifferent way to search in databases without FT (caution: returnsless results)Work-aroundsUse Mail-Rules (but that could cause other limitations)Add logic into Script Agents to filter documentsUse unprocessed documents option and also set not matching docs toprocessed

Keep Domino versions updatedLater versions have better performance (in most cases)e.g. Domino >= R5.0.10 has some new technology from D6Some customers reported big performance increase!Domino 6 really rocks!A lot of areas are more optimized for performancee.g. Network Compression, Streaming Replication, Rewritten@Formula Engine, Client Multitasking, Rewritten IMAP, full HTTP 1.1support, Directory Changes, SCT, Optimized Logging, EnhancedMemory Management, ...D7 will bring even better performance -specially on Linuxsee details in session ID116 Lotus Domino and Linux - WorkingTogether for a Strong Future (Kenneth Brunsen, James Powers)

The Human FactorCheck for private agents - Some users are reallycreativeControl size of email messagesThink if FT index on mailfiles is really neededMake sure people do shutdown their machines properlyhanging TCP/IP sessions can cause problems!Make sure power users do not have to much impact onserver

Tuning your Domino Installation1 ) Monitor your system over an extended period of time2 ) Exercise caution when doing changes3 ) Read and understand documentation4 ) Keep records of all changes at any time5 ) Only one change at a time6 ) Be aware of potential side effects7 ) Evaluate results over time8 ) Apply next setting

Domino Server StatisticsStatistics are collected automatically by different partsof the serverThere are different categories (Facilities) containing alarge number of statistic valuesExample:show stat mem.*Mem.Allocated = 68863464Mem.Allocated.Process = 8210820Mem.Allocated.Shared = 60652644Mem.Availability = PlentifulMem.Free = 870.449.152Mem.PhysicalRAM = 1604321286 statistics found

Working with Domino StatisticsStatistics can be collected in staprep.nsfOr displayed in the Admin Client(Real-time Statistic Charting)They are also data source for Historical Statistic ChartingEach Servertasks/subsystem is responsible formaintaining their statistic valuesTherefore right now there is no way to reset statisticsStatistics are resetted when sub-system is restarted (e.g. HTTP)

Platform StatisticsA lot of information about system and resourcesprovides comparable platform stats within DominostatisticsAlready in R5 for Win NT & Solarisavailable in Domino 6 on all platformsEnable via notes.iniPLATFORM_STATISTICS_ENABLED=1 (R5)Platform_Statistics_Disabled=1 (D6)show stat platform.*reset platform

Windows PerfmonPlatform specificperformance monitorIncluded in Windowshas access to NotesStatistics if properlyinstalled

Monitoring Performance on UNIXiostat -xiostatistics and activityvmstatvirtual memory activity and some CPU statisticsmpstatdetailed look on CPU statisticsnetstat -isummarizes network activity

UNIX - iostatiostat -x reports disk statisticsWatch the %b column, values > 20 % need to bechecked against high svt_t column.

UNIX - vmstatvmstat summarizes virtual memory activity and someCPU statistics.Monitor the sr column to keep track of scan rate.us, sy and id columns to check CPU usager column threads are contenting for CPU timeshould not be higher then about 4x the number of CPU'sWatch out for a high number of Context Switches (cs)

UNIX - mpstatmpstat 60 gives more detailed look on CPUwatch out for high numbers in usr, sys or wt

What is Happening on the DominoServer?Use Domino console commandsshow tasklists which tasks are active for the Domino servershow task debugundocumented commandprovides job and thread number for SERVER tasks on iSeriesshow statfull set of Domino statistics for servershow stat dominostats helpful when working with Domino Web server performanceproblemtell http show thread stateshows which HTTP threads are waiting for work and which threadsare currently servicing requests

What is Happening on the Domino Server?Use Domino console commandsshow stat agentagent stats such as:scheduled runs, triggered runs, used run time, ...tell amgr statusshows number of agents executing and agent concurrency settingshow stat databaseprovides key information on how Domino memory management isfunctioningshow database dir/dbname.nsf vprovides list of views and their sizes that are built over the database

Domino Memory ManagementHow Domino divides main storageDomino has it’s own memory pools/buffersMemory subpools are dynamically adjusted by DominoNSFBufferDBCacheAppsUsers

Domino Memory ManagementNSF Buffer Pool – what it doesNIFNSF Buffer PoolDisk

Domino Memory ManagementDomino database statisticsDatabase.Database.BufferPool.Maximum.MegabytesDatabase.Database.BufferPool.Peak.MegabytesDatabase.Database.BufferPool.PercentReadsInBufferWant >= 95%Database.DbCache.CurrentEntriesNumber of entries in the database cacheDatabase.DbCache.HighWaterMarkPeak number of entries in db cache since Domino server wasstartedDatabase.DbCache.MaxEntriesNumber of databases that can be cachedDatabase.DbCache.OvercrowdingRejections

Domino Memory ManagementCustomer example – looking goodDatabase.Database.BufferPool.Maximum.Megabytes = 710Database.Database.BufferPool.Peak.Megabytes = 708Database.Database.BufferPool.PercentReadsInBuffer = 98.12Database.DbCache.CurrentEntries = 71Database.DbCache.HighWaterMark = 256Database.DbCache.MaxEntries = 300Database.DbCache.OvercrowdingRejections = 0

Domino Memory ManagementCustomer example – looking very badDatabase.Database.BufferPool.Maximum.Megabytes = 170Database.Database.BufferPool.Peak.Megabytes = 168Database.Database.BufferPool.PercentReadsInBuffer = 95.44Database.DbCache.CurrentEntries = 510Database.DbCache.MaxEntries = 510Database.DbCache.OvercrowdingRejections = 8385Huge problem!Options:Increase NSF buffer pool sizeSet NSF_DbCache_MaxEntries

Domino Memory ManagementTwo key notes.ini parameters affect Domino memorymanagementNSF_Buffer_Pool_Size_MBPercentAvailSysResourcesOther Domino memory buffers are allocated based onthese parameters

Domino Memory ManagementPercentAvailSysResourcesIntroduced in R5.0.4Controls % of memory resources available to Domino serverValid values = 2% to 100%Memory available depends on where Domino server is executingAllows dynamic adjustment of buffer sizes

Domino Memory ManagementNSF_Buffer_Pool_Size_MBStatic setting of NSF buffer poolUse in combination with PercentAvailSysResourcesPercentAvailSysResources won’t overrideNSF_Buffer_Pool_Size_MB setting

Domino Tuning - Overall Server SettingsPercentAvailSysResources (notes.ini)Per partition memory size in % of total system memory.Example: PercentAvailSysResources=50NSF_BUFFER_POOL_SIZE(_MB)Buffer pool allocation per partition Domino server. Might berequired if high number of memory > 4 GB RAM or old Dominorelease < 5.07. Can be 2000 MB at most per partition.E.g. NSF_BUFFER_POOL_SIZE=2000Don't use Buffer Pool Size if not really neededUse PercentAvailSysResources ONLY!!!

Domino Tuning - Overall Server SettingsServer_Max_Concurrent_Trans (notes.ini)Server_Pool_Tasks (notes.ini)For high number of NRPC connections increase the number ofServer_Max_Concurrent_Trans to a high value (500-1000) andServer_Pool_Tasks parameter to 50 - 100 (number of server)threads.NSF_DbCache_MaxEntriesIn case of high statistical valueDbCache.OvercrowdingRejections set this to maximum numberof concurrent users or the maximum number of databases open(whatever is higher)e.g. NSF_DbCache_MaxEntries=10000

Mail PerformanceImpact of number of documentsNumber of documents has greater impact as file size grows161412108641,024 docs4,096 docs7,196 docs28,672 docs20100 MB 700 MB

Mail PerformanceFile those documents!Inbox is most expensive container in a mail fileUpdated each time user opens mailUpdated each time user clicks Refresh buttonReducing number of documents in Inbox reduces:CPU utilizationMain storage required to update the view

3025201510Mail PerformanceReducing number ofdocuments stored in InboxMail file open performanceimproved 50%Steady state performanceimproved 12%OpenSteady State5025% in Inbox 100% in Inbox

Mail File Performance Best PracticesFor optimal performanceReduce overall document count in the fileUp to 2X CPU reductionKeep number of documents stored in Inbox to a minimumFull text indexing and searchingFull text indexing is cheapCPU usage increase < 0.5%Searches are expensive20% extra CPUMoral of the storyCreate permanent indexes if using FTI function

Mail File PerformanceMail statistics to watchMail.WaitingIndicator of ROUTER performanceNumber of mail items waiting to be delivered in all MAIL.BOX filesMay indicate need additional filesMail.TotalPendingNumber of mail messages resident in all MAIL.BOX filesIncludes all dead, held, and pending mail

HTTP StastisticsSome Important statistic valuesHttp.Worker.Total.BytesRead = 27.366Http.Worker.Total.BytesWritten = 368.333Http.Worker.Total.File.Requests = 13Http.Worker.Total.File.RequestTime = 971Http.Worker.Total.Notes.Requests = 70Http.Worker.Total.Notes.RequestTime = 18.490Http.Worker.Total.TotalRequestTime = 19.46

D6 HTTP Cache SettingsGeneral Setting is still therebut movedCache settings are still onDomino Web Engine Tab

Domino Platform Tuning AIX - 1AIX Kernel SettingsMaxUProc - maximum number of processes that a single useraccount can run. set this to at least 128.check the current value with: lsattr -E1 sys0 -a maxuprocAIX Domino Server Account SettingsAIXTHREAD_SCOPE=S (default in Domino 6)

Domino Platform Tuning AIX - 2AIX Domino Server Account Settings/etc/security/limitsdata = (min 3 x default value)stack = (min 3x default value)rss = (min 3 x default value)nofiles = (min 3 x default value) - don't set it to unlimitedPage Space adjustments/etc/inittab/vmtune (best is to have no paging)Page space 2x the amount of RAMif server uses > 20% paging space (check via lsps -a) us/usr/samples/kernel/vmtune -P 50 (down to 20)

Domino Platform Tuning AIX - 3Network Tuningtune to ensure large amounts of data can be transferred andreceived. Minimum of tx_queues should be 512.To view current settings :lsdev -Cc adapter -s available (list adapters)lsattr -e1 ¦grep size (list queue sizes)Default Socket BuffersSet send/receive window size to 16384 Bytes.check via no -a command.tcp_sendspace=16384tcp_recvspace=16384sb_max=65536

Domino Platform Tuning Solaris - 1File descriptors limitsincrease number of file descriptorsin /etc/systemset rlim_fd_max=65536 (mandatory)Memory tuning improves system response for heavily usedfilesystems. Cachefree settings should be 1/16 (up to 1/6) ofphysical memory divided by 8192 to get 8 KB pages.set priority_paging=1 and cachefree=8192 (for a 1GB)(only for Solaris 6 and 7)High filesystem page (RunFaster=1)To avoid high file system page in rates adjusts % of memorykernel will map into address space for file system cache.set segmap_percent=20 (default 12 up to 60)

Domino Platform Tuning Solaris - 2Excessive filesystem flush daemon overheadOn very busy Domino servers with a lot of I/O limit the CPUcycles that the file system flush daemon can use.set autoup=600set tune_t_fsflushr=1this wakes up the fsflush daemon once a second and allowsolder pages to remain in memory before being flushes to diskThrotted page daemon I/OFor multiple swap disks or 10'000 RPS disk you should increasethe I/O rate of the page daemon.set maxpgio=16384

Domino Platform Tuning Linux - 1Kernel ParameterTwo ways to set kernel parametersExample SuSE: echo "8192" >/proc/sys/fs/file-maxExample RedHat: file /etc/sysctl.conf: fs.file-max=49152Number of open filesfs.file-max=49152fs.inode-max = 3*fs.file-maxcheck actual values: echo /proc/sys/fs/file-no; inode-nr &inode-state

Domino Platform Tuning Linux - 2Increase maximum number of shared memory segmentskernel.shmmni=8192Increase maximum number of system-wide semaphoreskernel.sem=250 18432 32 1024Increase maximum number of threads/proc/sys/kernel/threads-max

Domino Platform Tuning Linux - 3Limitsyou need the following line in all /etc/pam.d/* control files like/etc/pam.d/login,su,... to ensure limits can be setsession required /lib/security/pam_limits.soNumber of open files for notesuser/etc/security/limits.confnotes soft nofile 49152notes hard nofile 49152Number of processes/threads notesuser/etc/security/limits.confnotes soft nproc 12500notes hard nproc 12500

Domino Platform Tuning Linux - 4IPC/Shared Memory (Domino specific)export Notes_SHARED_DPOOLSIZE=20000000 to enalarge theshared memory segments allocatedSet the "noatime" attribute flag to disable the file accessmodified timestamp on all filesystems for Domino, /var, /tmp in/etc/fstabSome additional paramters to check/proc/sys/vm/buffermem/proc/sys/vm/bdflushThere is a lot stuff to tune on Linux

Domino for Linux Hard Disks TuningDefault settings for disks are very conservativeTune settings with hadparmhdparm -A1 -a8 -c3 -d1 -Xnn -W1 devicename(-A) sets drive read lookahead flag(-a) sets FS read ahead. 8 sectors (4KB) to 12 sectors (6KB)(-c) sets EIDE 32bit I/O support(-d) enables DMA(-X) sets the DMA mode (see man page for details)(-W) IDE write caching mode (possible data lost when crash!)Example: hdparm -A1 -a8 -c3 -d1 -Xnn -W1 devicenameTest Performance before and after with(-t) perform device read timings(-T) perform cache read timings

iSeries Memory PoolsiSeries Memory PooliSeriesSubsystemJobs- Amgr- HTTP- Server-Update

iSeries Memory PoolsMemory Pool 1Memory Pool 2MAIL01SubsystemCalconnRouterSMTPMAIL02SubsystemCalconnAmgrROUTERSMTPAppSrvrSubsystemAmgrHTTPServer. . . . . . . . .

WRKSYSSTS

iSeries Constantly Strives forBalanced PerformanceUnit TypeSize(M)%UsedI/OReqsReqSizeReadReqsWriteReqsRead(K)Write(K)%Busy1 6607 4194 71.7 3.7 5.5 3.3 .3 4.3 15.2 152 6607 4194 71.6 3.6 9.3 2.9 .6 7.5 17.6 173 6607 4194 72.8 3.5 8.3 2.7 .8 6.6 13.7 134 6607 4194 67.8 3.2 8.2 2.8 .4 8.0 9.3 125 6607 4194 71.4 3.1 8.3 2.4 .6 7.2 15.1 146 6607 4194 71.3 3.5 9.0 2.3 1.1 7.4 15.3 167 6607 4194 71.4 3.5 8.9 4.3 .8 6.9 21.2 178 6607 4194 71.6 3.8 11.0 1.9 .9 8.0 17.4 14The operating system does thisautomatically...

Collection Services

Domino Subsystem Tuning - iSeriesChange individual job priority permanentlyVery useful when specific task is consuming large amounts ofCPUHelpful when have Domino running on same system asinteractive applicationsSteps to follow:Create a class in library QUSRNOTESCRTCLS CLS(QUSRNOTES/MAILCLASS) RUNPTY(30)Tell Domino which task (or tasks) should use this classEDTF ('/qibm/userdata/lotus/notes/domino_classes')

Domin Subsystem Tuning - iSeriesContents of domino_classes file:SERVER=Mail01CLASS=mailclassTASKS=calconn,routerCLASS=mailclass2TASKS=schedSERVER=App01CLASS=appclassTASKS=Amgr

Domino Subsystem Tuning - iSeriesAll tasks can be changed except for:QNNINSTSSoftware requirementsDomino for iSeries R5 or D6OS/400 V4R3 or later

Domino Performance RedpapersDomino 6 for iSeries Best Practices Guide, SG24-6937www.redbooks.ibm.com/redpieces/pdfs/sg246937.pdfSizing Large-Scale Domino Workloads on iSeriespublib-b.boulder.ibm.com/Redbooks.nsf/RedpaperAbstracts/redp3802.html?OpenIBM Lotus Domino for iSeries Performance and Tuning, REDP3843www.redbooks.ibm.com/redpapers/pdfs/redp2843.pdf

Additional ResourcesNotes.Net Inside Notes DatabaseNotes.Net articlesDomino Performance ZoneLotus & Microsoft Knowledge BaseReadme of each QMU releasemailto: nsh@nashcom.de

Performance ReferencesiSeries Domino Performancehttp://www.ibm.com/servers/enable/site/domino/perform.htmlPerformance Zonewww.lotus.com/performanceLotus Developer Domainwww.lotus.com/lddAsk Professor INIwww.lotus.com/ldd/today.nsf/profini?OpenViewIBM Workload Estimatorwww-912.ibm.com/servlet/EstimatorServlet

Where To Go For More InformationDomino for iSeries Sizing and Performance Tuning on the IBMeServer iSeries Server, SG24-5162-01www.redbooks.ibm.comAuthors: Wilfried Blankertz, Christina Fasth, Kim Greene,Brandon Rau, Colin Stamp, Deb LandoniSeries Domino Performancewww.iseries.ibm.com/developer/domino/perform/index.htmlPerformance Zonewww.lotus.com/performanceLotus Developer Domainwww.lotus.com/lddAsk Professor INIwww.lotus.com/ldd/today.nsf/profini?OpenViewIBM Workload Estimatorwww-912.ibm.com/servlet/EstimatorServlet