Tuesday, December 24, 2013

Disable WCS SOLR search features

The steps may vary based on how far you are with your configuration, I wish there was a simpler global setting to enable / disable this feature.

Step 1: Disable "store-enhancements" Feature

Replace /apps/websphere/wcs70 with WCS home directory
Replace -DinstanceName with your WCS instance name
Replace -DdbUserPassword with db passoword used by WCS instance

Here is a sample command

/apps/websphere/wcs70/bin/config_ant.sh -buildfile /apps/websphere/wcs70/components/common/xml/disableFeature.xml -DinstanceName=guest -DfeatureName=store-enhancements -DdbUserPassword=wcs7lab01

When you run the script to disable SOLR search it won't rollback WAS level profile changes, essentially the script will only rollback WCS EAR file and database changes.

Step 2: Delete SOLR WAS Profile

To delete WAS profile we need to follow couple of manual steps, if we don't delete SOLR profile any future re-enablement of search feature will report "SOLR profile already exists" error.

-- Delete profile
/apps/websphere/ws70/bin/manageprofiles.sh -delete -profileName guest_solr

-- Update Registry
/apps/websphere/ws70/bin/manageprofiles.sh -validateAndUpdateRegistry

-- Manually delete the solr profiles folder, E.g.
rm -rf /apps/websphere/ws70/profiles/guest_solr


Step 3: Database Tweaks

Let us assume you don't want to re-run the disablement script, instead want to do few DB tweaks to toggle back and forth between SOLR and non - SOLR based features

select * from EMSPOT where name like '%search%'
and storeent_id in (select storeent_id from storeent where identifier='AuroraStorefrontAssetStore')

You should notice a record with usagetype as "STOREFEATURE" for your store id, if this is the case delete this record
NOTE: Usually this is enabled for the Store asset store

select * from SEOURLKEYWORD where storeent_id in (select storeent_id from storeent where identifier='AuroraStorefrontAssetStore')
Mark all records as inactive by setting status column value as 0

NOTE: Usually this is enabled for the Store asset store As of V7, FEP5 there is no way to enable / disable SOLR from wc-server.xml

Step 4: Re-enable SOLR

use these steps if you ever want to re-enable SOLR alone and assuming the store enhancements feature is already enabled in WCS EAR and WCS DB.

This command can be run both in WCS box or any remote box to setup SOLR WAS profile.
Here is a sample.

/apps/websphere/wcs70/bin/./config_ant.sh -debug -buildfile /apps/websphere/wcs70/components/foundation/subcomponents/search/deploy/deploySearch.xml -DinstanceName=guest -DdbUserPassword=wcs7lab01 -DsolrHome=/apps/websphere/wcs70/instances/guest/search/solr/home -DautoConfigSearchWebserver=true -DisShareWCWebserverProduct=true > /apps/websphere/wcs70/instances/guest/logs/only_search_enable.log

Further Reading

Monday, December 23, 2013

WCS Feature Pack 6 Upgrade

1. su - root
2. Upgrade WAS/IHS and Plugin fix pack level to
Refer "Recommended fixes and settings for WebSphere Commerce Version 7" for further details

Download WAS for your os version from here

upgrade IHS, Plugin and WAS using filent install files
<WAS_UPDATE_INSTALLER>/update.sh -options "/home/wasuser/git/wcs_scripts/install_files/ihs.update.response.txt" -silent
<WAS_UPDATE_INSTALLER>/update.sh -options "/home/wasuser/git/wcs_scripts/install_files/plg.update.response.txt" -silent
<WAS_UPDATE_INSTALLER>/update.sh -options "/home/wasuser/git/wcs_scripts/install_files/was.update.response.txt" -silent

The install files used of this upgrade can be cloned from my git project locate here

Check the output from <WC_HOME>/bin/versionInfo.sh to validate upgrade was successful

Name                     IBM WebSphere Application Server - ND
ID                       ND
Build Level              cf271250.01
Build Date               12/13/12
Architecture             AMD (64 bit)

2. For GUI install refer this section
Here are the steps to install on developer toolkit

3. Silent install can be used to upgrade server environment.

/WC_V7.0_-_FEP6_FOR_MP_ML/server/install -options /media/sf_linux_share/WC_V7.0_-_FEP6_FOR_MP_ML/server/responsefile.txt -silent -is:javaconsole

Check the output from <WC_HOME>/bin/versionInfo.sh to validate upgrade was successful

Installed Product
Name                     IBM WebSphere Commerce
ID                       wc.fep6
Build Level              130322dev
Build Date               22/03/13

3. Restart DMGR, node agents and WCS instance to validate your instance.

Friday, December 20, 2013

WCS Quick Performance Tuning Cheat sheet

Performance tuning of an application is best achieved and measured through an iterative approach, Although not a comprehensive list the performance gains from these steps should be worth it, there isn't any silver bullet here, every application is different and hence tuning settings will differ based on your application / infrastructure / architecture and performance workload profile.

1. Tune Application

Begin with the basics, ensure you do not have any long running queries, you can easily debug WCS SQL using the RAD SQL profiler, make use of Oracle AWR reports to refine SQL, index etc..
Validate logging from application makes use of Apache logging level appropriately, on production you don't want to log details unnecessarily and cause high I/O, this will have impact on CPU and application performance.
Refer my previous blog on SQL tuning to read more on this topic.

2. JDBC Thin vs. Thick Driver

While using the JDBC OCI driver with RAC, the database can be specified with an Oracle Net keyword-value pair. The Oracle Net keyword-value pair substitutes for the tnsnames entry.
E.g. jdbc:oracle:oci:@MyHostString
This configuration requires Oracle client installed on WCS JVM and a TNS entry for MyHostString
Thick drivers are written in native language C and JNI code, they can potentially reduce the amount of GC, but they add an overhead of JNI calls.
Recommendation is to go with thick driver for RAC setup.

3. JVM Heap Size

There is a general misconception that more Heap means applications will perform better, WRONG... as a best practice never cross max JVM heap size of 1.5 GB, more heap means lesser GC cycles, but you might spend more time in every GC cycle, every GC cycle will pause JVM for a while and hence it will queue up the requests until GC completes, a longer GC cycle under high load could actually overwhelm the JVM and be a perfect recipe for a JVM crash.
On a 64 bit machine you could potentially set heap size higher than 2 GB in case you are using JVM for dynacache.

4.Webcontainer thread.

Every thread means consumption of additional CPU and Heap, you need to size this based on your server configuration, thumb rule is to have 3x times of the cores available on the box, so on a 8 core box we could have close to 25 threads
It is always good practice to leave it at default of 25/25 on a 8 core box and then reduce it if it consumes more CPU.

5. Database Connection Pool size

This should normally be equivalent to Thread pool size, it is quite rare that you will find all threads occupied and busy concurrently, Try to keep this value slightly higher than that thread pool size.
DataSource Conn Pool Size = (# Of Webcontainer Threads)
+ (# Of WC Scheduler Threads)
+ 1 (for WC Key Manager)
+ 1 (for WC Auditing)
+ (# of Parallel and Serial MQ listener threads)
+ (# of Non-Webcontainer Threads used by Custom Code).

You can also explore the usage of WAS reserve pool configuration for large application deployments, this feature is available through a  feature pack, more information is available here.


6. Server ULIMIT

This is one of the parameters which you would most certainly tune in every production environment, by default these are set at 1024, you can safely set this to 65 K or unlimited for WCS installtion.

7. WebServer Load Balancer Configuration

IHS supports many different types of algorithm, Here is an Excellent tech note on how to setup plugin configuration, ensure you define "LoadBalanceWeight" as recommended by this technote.

8. Generic JVM arguments

To generate a system dump automatically on an OOM you will need to set the following  JVM generic argument:-Xdump:system:events=systhrow,filter=java/lang/OutOfMemoryError,request=nodumps+exclusive+prepwalk
In order to generate java core dump, system core dump, heap dump and a snap dump at user signal, the dump agents must be configured through JVM options as follows.-Xdump:java+heap+system+snap:events=user

Further Reading

WCS WAS Tuning

Wednesday, December 18, 2013

Websphere Commerce 64 bit Install


Very recently, WCS V7 FEP2 introduced support for 64 bit WAS, I decided to take a potshot at this on my Oracle Linux 6.3 VM, review the excellent article from IBM in "Further Reading" section if you want to weigh in the merits of 64 vs 32 bit WCS installation.

Pre-Install Task

Although we are working on a 64 bit install here, WAS and WCS installers still depend upon 32 bit OS libraries,  the installation will error out if you are missing 32 bit OS libraries.

By default "yum install" command will only install the 64 bit libraries if the host OS is 64 bit, to install a 32 bit version first execute the yum search to locate a library and then install it with .i686 extension. An example is shown here to install libstdc++ 32 bit version

yum search libstdc++

Loaded plugins: refresh-packagekit, security
============================ N/S Matched: libstdc++ ============================
compat-libstdc++-296.i686 : Compatibility 2.96-RH standard C++ libraries
compat-libstdc++-33.i686 : Compatibility standard C++ libraries
compat-libstdc++-33.x86_64 : Compatibility standard C++ libraries
libstdc++.i686 : GNU Standard C++ Library
libstdc++.x86_64 : GNU Standard C++ Library
libstdc++-devel.i686 : Header files and libraries for C++ development
libstdc++-devel.x86_64 : Header files and libraries for C++ development
libstdc++-docs.x86_64 : Documentation for the GNU standard C++ library

Now install 32 bit version of compat-libstdc++-33 as follows.

yum install compat-libstdc++-33.i686
yum install gtk2.i686

For record, you will notice following error if 32 bit version of this library is missing from the host OS.
Exception in thread "AWT-EventQueue-0" java.lang.UnsatisfiedLinkError: ic_jni (libstdc++.so.5: cannot open shared object file: No such file or directory)

Installation Task

Begin with a 64 OS platform supported by WCS, I have tried this on Oracle Linux 6.3 , so these steps should work flawlessly on centos and redhat.
All of the silent install files, scripts etc.. are available in https://github.com/hariinfo/wcs_scripts

Step 1: Install Oracle 64 bit client
Make a note of the Oracle home directory and setup this is .profile or .bash_profile of wasuser and oracle user profile.
I usually make use of Oracle's excellent pre-install packege before installing Oracle client on the host OS
This package does all the magic needed to bring your host OS up to required configuration level for Oracle install.

yum install oracle-rdbms-server-11gR2-preinstall

Step 2: Install  IHS and WAS
IHS version packaged in the 64 bit WAS ND software is 32 bit install in reality, only WAS and more importantly the JVM is 64 bit, so don't be surprised if you continue to see a 32 bit version of IHS after the installtion, begin installtion from the downloaded copy of WAS ND 64 bit.
Part number for IBM WebSphere Commerce V7.0 WebSphere Application Server Network Deployment V7.0 (64-bit) for Linux on AMD and Intel as of this writing is "CZFA3ML"

unzip the downloaded tar file and begin your silent install using following steps, or you can perform a GUI install as well.
CZFA3ML\WAS\disk1\IHS\install -options "/home/wasuser/ihs_install_response.txt" -silent
CZFA3ML\WAS\disk2\WAS\install -options "/home/wasuser/was_install_response.txt" -silent

Step 3:WAS upgrade
Now update WAS, Plugin and IHS to level.

Ensure that the 64 bit versions of both WAS and WCS installer is installed before the WAS upgrade.
Following were the latest version as of this writing
Install WAS update installer, download the 64 bit version

Install WCS update installer, download the 64 bit version

<WAS_UPDATE_INSTALLER>/update.sh -options "/home/wasuser/ihs.update.response.txt" -silent
<WAS_UPDATE_INSTALLER>/update.sh -options "/home/wasuser/plg.update.response.txt" -silent
<WAS_UPDATE_INSTALLER>/update.sh -options "/home/wasuser/was.update.response.txt" -silent

Step 4: WCS Install
Install WCS V7, do note that WCS does not come in 32 or 64 bit flavor, so the installer you used for 32 bit should just work fine for 64 bit, but the support for 64 bit WAS was introduced in fix pack level 2 of WCS V7.

Step 5: WCS Fixpack 7
Upgrade WCS to fix pack 7 (this is the latest as of this writing, you need to be at fix pack 1 for 64 bit support)

Step 6: WCS Instance Creation
Create WCS instance and configure with Oracle database

Step 7: Federate WCS instance with DMGR

 Refer My previous blog on WAS, WCS, IHS installation which includes detailed steps for Step 4 through 7

Post-Install Validation

Name                     IBM WebSphere Application Server - ND
ID                       ND
Build Level              cf251235.04
Build Date               8/30/12
Architecture             AMD (64 bit)

The version info output should display 64 bit as the WAS version.

Save and sysnchronize the DMGR, now stop and start the server, once the server is started grep the PID to validate the min and max heap size
ps -elf |grep server
you should notice following in the grep output, this indicates WAS was started with more than 2048 heap size, "-Xms1024m -Xmx3024m"

Further Reading


WCS install silent files and utility scripts