Performance Tuning for Oracle WebCenter Content 11g - Fishbowl ...

Performance Tuning for Oracle WebCenter Content 11g:
Strategies & Tactics
Chris Rothwell, Fishbowl Solutions
Paul Heupel, Fishbowl Solutions
Introduction
Oracle WebCenter Content 10g functionality was effectively contained in one container - the Content Server. This
fact alone made it easy to deploy, administer, and customize. However, for all of these easy capabilities, the product
was somewhat lacking on the scalability and performance side. With Oracle WebCenter Content 11g, and the
inclusion of Oracle Weblogic Server, SOA and BPM, the product has expanded its functionality to achieve best-in-
class performance and scalability. The new tradeoff is that these additional infrastructure components have created
a layer of complexity that often leads to delayed deployments and non-optimized optimized systems. The good news is that
with the right tuning strategies and appropriate use of reverse proxies and load balancing you can truly optimize
WebCenter Content 11g and maximize your technology investment.
Tuning WebCenter Content
Oracle’s ECM solution has its roots in the Stellent Content Management offering. From Xpedio 4.5 to Oracle 10gR3
the content management system’s core was a Java Standard Edition based solution. WebCenter Content 11g is
deployed as a Fusion Middleware solution. Optimization techniques that held true from the Xpedio days to UCM
10gR3 may no longer apply to the latest incarnation of Fusion Middleware’s e’s Enterprise Content Management (ECM)
system.
Content Server architecture
Memory and Java Virtual Machine (JVM) Tuning
Memory is still one of the most significant performance tuning areas with Oracle WebCenter Content. As part of the
Fusion Middleware e stack WebCenter Content requires a Java Enterprise Edition container. Prior versions of the
Content Server ran as a Java Standard Edition application. Under UCM 10gR3 and earlier, you could specify JVM
tuning in the $UCM_HOME/bin/intradoc.cfg
configuration file. Tuning was somewhat limited since the
JAVA_OPTIONS would append custom parameters with computed values.
© 2012. Fishbowl Solutions, Inc.

In WebCenter Content 11g, the content management solution is deployed inside Weblogic Server (WLS), with any
JVM tuning performed on the application server. You have full control, either by modifying the managed server
using the administrative console or modifying the USER_MEM_ARGS environment variable startup scripts.
Oracle’s documentation suggests the following on Unix and Windows with the JRockit JVM:
-Xms256m -Xmx1024m –XnoOpt
The –Xmx flag specifies the maximum heap size with this example specifying 1GB of memory. Best practice is to
keep your JVM heap settings under 75-80% of the available physical RAM, within limits for machines with
excessive amounts of memory. As heap size is increased, CPU load will also increase for larger garbage collections.
Under 32-bit operating systems, 1.5GB is the practical maximum limit assuming other services are not consuming
resources.
The –Xms flag specifies the minimum heap size on initial startup. Increasing the heap takes considerable time, so it
is best to set the Xmx and Xno parameters to the same size. For example:
-Xms1024m -Xmx1024m -XnoOpt -XgcPrio:throughput
On x86 and x64 hardware, JRockit should be the preferred JVM. JRockit was a Java virtual machine optimized for
x86 hardware by Intel, purchased by BEA, and acquired by Oracle. The JRockit JVM performs significantly faster
on x86 or x64 Windows and Linux architectures than Sun’s architecturally neutral JVM implementation.
An example of JVM tuning, from another Oracle whitepaper, started with:
-Xms3g
-Xmx3g
-XX:PermSize=512m
-XX:MaxPermSize=512m
-XX:+UseParallelGC
-XX:ParallelGCThreads=8
-verbose:gc
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:NewRatio=3
-XX:+UseAdpativeSizePolicy
-XX:+AggressiveHeap
-XX:+DisableExplicitGC
-Xnoclassgc
-Xloggc:
and continued to tune WebCenter as:
-d64
-server
-Xms3g
-Xmx3g
-XX:PermSize=512m
-XX:MaxPermSize=1024m
-XX:+AggressiveOpts
-XX:+UseParallelGC
-XX:ParallelGCThreads=16
© 2012. Fishbowl Solutions, Inc.

-verbose:gc
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:NewRatio=4
-Xnoclassgc
-Xloggc:
-Dweblogic.threadpool.MinPoolSize=72
-Dweblogic.threadpool.MaxPoolSize=72
-Dweblogic.SocketReaders=12cketReaders=12
-Djps.auth.debug=false
Operating system architecture does not on its own provide enough information to properly
tune the Content Server.
As seen in the above example, repeated tuning and testing was required to find an optimum configuration. The
content repository has the additional complexity of requiring different performance configurations for contribution
and consumption environments. A heavy ingestion pattern will benefited from a -XgcPrio:throughput garbage
collection, while searching may benefited from other GC models.
Confirm your capitalization is correct. In many cases, command-line options are case sensitive unless explicitly
stated. A configuration flag improperly set may be ignored, or cause unintended consequences.
Disk
Usage
WebCenter Content, like the earlier versions of the content repository, has a variety of disk mounting options, with
implications for what type of storage may be appropriate for each area. Directories within the content repository
may have different service level agreements and performance requirements. Using a single storage system does
not produce optimal performance-cost cost optimization.
The latest incarnation of the Oracle Content Repository, a shared file system is still required for clustering. The
ECM services run as Java processes. Prior to 11g, these services took the strategy of keeping a memory cache,
writing to a shared file system or database, and having the other nodes update their local cache. All content
management services continue to be stateless and utilize the same concurrency mechanism even though they are
living in a Java Enterprise Edition world.
© 2012. Fishbowl Solutions, Inc.

Content Server in a clustered configuration
High performance low latency shared disk space is critical for performance in the shared directory.
When a file is ingested into the content repository, it is placed in the
/user_projects/domains//ucm/cs/vault/~temp
directory. From that directory, the file is copied to any refineries, copied for full-text indexing, any necessary
transformations created, and moved to the appropriate vault and weblayout locations. File IO is key to that ~temp
directory, with five or more read operations as part of a standard check in.
All other sub directories within the vault are the ‘native’ or original file checked into the repository. The vault
directory is a long term archive for the asset, and should be viewed from a disaster recovery perspective. A copy of
the file, or a version intended for heavy consumption,
is typically placed in the weblayout directory. Any file in the
weblayout directory could be recreated, so emphasis should be on performance rather than reliability.
In 10gR4 and below, Content IDs or dDocNames had required optimizations like the Fast Checkin component to get
around row locking on the counter tables under heavy ingestion. The 11g repository changed the way the identifiers
were generated, caching a block of content identifiers. There may be minor gaps in the sequence of content
identifiers, which can be ignored.
Prior to 11g, a typical installation would have data, search, shared, and weblayout directories that were typically
excluded from virus scanning. These directories still exist in 11g, but are now found in the domain directory rather
than the base UCM path. For example, in 10g:
/server/weblayout
became
/user_projects/domains//ucm/cs/weblayout
WebLogic logging directories should also avoid virus scanning in version 11g and later.
© 2012. Fishbowl Solutions, Inc.

Logging
11g uses the Weblogic logging. The granularity of information sent to the logging system goes from
TRACE, DEBUG, INFO, NOTICE, WARNING, ERROR, CRITICAL, ALERT, to EMERGENCY. In production
environment, change the logging level to ERROR. One could modify the
/user_projects/domains//config/servers/UCM_server1/logging.xm
l
or modify the logging levels using the Weblogic administrative console.
File Store Providers
Oracle’s ECM solution moved a File Store Provider to accommodate different usage patterns. The default file store
provider in 11g continues to use the vault/weblayout file structure.
Classically, the Oracle ECM solution would store relational data in a database and files in a file system. As the
number of managed assets increased, some scalability issues became apparent. Three metadata fields – dDocType,
dSecurityGroup, and dSecurityAccount – were used to spread the assets out to multiple directory structures. There
is a limit to how many files can go into a directory structure, and as the number of assets grew into the tens of
millions, hundreds of millions, and eventually billions inode issues and disk management became a bottleneck. UCM
updated the default file store provider to add additional dispersion directories to spread out the files.
A database file store provider was added where the assets are persisted in the database rather than a file system.
The Oracle 11gR2 Database SecureFiles API improved performance by over 40% compared to the 10g
implementation. Performance matches, and in some cases exceeds, major networked file systems. In addition to
the I/O gains, repositories that have Database Compression will automatically have de-duplication performed
against content stored the repository.
When content is uploaded to the repository, a temporary file is placed in the vault/~temp location with a cache
cleanup eventually clearing out that disk space. The current version allows that cache to be limited to one day, so
care must be taken when ingesting very large volumes of content. Content must also be indexed before that
temporary area becomes a candidate for cleanup.
Virtualization
Oracle differentiates between hard and soft portioning from a licensing perspective. With hard partitioning in use,
one only licenses the CPU used by the virtual machine. Soft partitioning requires licensing for all CPUs in the host
machine. Oracle VM can be configured to qualify as hard partitioning, but EMC VMWare is considered soft
partitioning. Hardware prices are trivial compared to software, so optimize the virtual hosts to consolidate licenses.
Typically, multiple smaller instances perform better than fewer larger instances. Attempt to optimize CPU
utilization, adding additional CPUs to the host servers as needed.
While CPU architecture, socket, and cores impact the licensing costs, memory does not. A physical CPU may be
shared among multiple virtual machines, but memory should not be a pooled resource.
Services and Components
WebCenter Content continues the service-based architecture introduced in earlier versions of the content
repository. Services that return search results, metadata, or actual assets can be extended or overridden.
GET_SEARCH_RESULTS, for example, can return a large amount of data if a repository has many custom metadata
fields. The content repository will cache the search results, but network traffic can be significantly reduced by
creating a template that returns only the fields and result sets needed.
© 2012. Fishbowl Solutions, Inc.

IDOC script includes can be cached, so the html will be dynamically rendered and then placed in session scope for a
specific user or application scope for all users. The cacheInclude method takes the include name, scope, and life
span as required parameters. For example, the std_page_begin include would be cached for ten minutes for each
user.
11g continues to lack a default success status code or message returned with all services. Content Services
typically indicate an error by setting StatusCode property to a non-zero number. CIS, RIDC, and several other
integration methods will potentially throw an exception when there are problems, but will absolutely throw an
exception if you look for the StatusCode property and it is missing. One can either trust the service will throw an
exception and assume it works, or modify the content server to set a default success status code.
Reverse Proxy
When architecting a website for high performance using Webcenter Content, a reverse proxy can be used to
improve both performance and security for your site. A reverse proxy functions as a gateway to your network and
adds an additional layer of caching for site visitors that will help to improve page response time, particularly under
heavy load.
Typically, a reverse proxy will reside in the DMZ of your network and will be the entry point for users accessing your
site. The standard process flow for a user accessing a site behind a reverse proxy is as follows:
1. A user enters http://www.mysite.com in a browser.
2. DNS directs the user to the reverse proxy server that is sitting in your DMZ.
3. The reverse proxy determines if the request is being made for static content or dynamic content.
4. If static content is being requested, the reverse proxy will check its cache and will return the cached page to
the user if the page is found in the cache.
5. If dynamic content is being requested or if the reverse proxy does not have the page in its cache, it will send
a request through the firewall to a web server inside your network to retrieve the requested page and will
return that page to the user.
The performance gains from caching at the reverse proxy level are obviously contingent on a number of factors
including the number of static resources and pages that users are requesting, the frequency at which those items
are accessed, and the hardware-network infrastructure that is being used. One popular reverse caching application,
Varnish Cache, claims to improve delivery by a factor of 300 – 1000x depending on architecture when serving a
page from cache (www.varnish-cache.org),.
Besides the caching advantage of this model, your site also gains an additional level of security by implementing a
reverse proxy. All requests that are made to your site are being filtered through the remote proxy server, which
limits an end-user from distinguishing server names or other network architecture information that could
potentially be used to compromise your systems. Additionally, there is only a single entry point through your
firewall, namely between your proxy server and your web server, so network administrators have considerably more
control over limiting the traffic that is allowed past the firewall.
© 2012. Fishbowl Solutions, Inc.

Under a contribution-consumption
consumption site architecture model, utilizing a reverse proxy allows your network
administrator to keep both the contribution and the consumption Content Server instances inside the firewall. The
network architecture diagram below demonstrates using multiple reverse proxies with a load balancer ancer for added
performance in a contribution-consumption consumption Site Studio web site model.
© 2012. Fishbowl Solutions, Inc.