Tuesday, October 2, 2012

Dynamically enabling (or disabling) the JDBC driver level logging using debug version of Oracle JDBC thin driver (ie ojdbc6_g.jar) on app server

This post is about dynamically enabling (or disabling) the  JDBC driver level logging with debug version of Oracle JDBC thin driver "ojdbc6_g.jar" in place while app server is up and running.

Typically, when this jar is put in place in application server environment, we enable the logging using JAVA system property "-Doracle.jdbc.Trace=true". This approach is static in the sense that application server restart is needed each time we want to enable or disable the logging. This static approach is not at all feasible in a production system where we want to trace the jdbc calls at some particular instance of time.

A programmatic (dynamic) approach to enable/disable driver level logging is also discussed in doc "http://docs.oracle.com/cd/B28359_01/java.111/b31224/diagnose.htm". However this approach is suitable for enabling/disabling the logging from within the application. So this method would essentially only trace JDBC calls at places where application code interacts with DB which might not be sufficient to troubleshoot jdbc issue in application server environment where jdbc connections are actually maintained and managed by application server.

Extending this dynamic enabling/disabling logging idea to application server is a bit tricky. From application server perspective, we can toggle driver level logging at runtime using similar programmatic approach. This would affect the logging at global level (be it app server interactions with DB or any application that is deployed on the app server, all DB interactions would be logged). This can be accomplished on " sun.misc.Launcher$AppClassLoader" instance that lies at "com.oracle.jdbc"/"diagnosability" in MBean tree.

A) Grab the instance name of MBean "com.oracle.jdbc"/"diagnosability" using JConsole, or any other JMX browser utility.

1. Start JAVA_HOME/bin/jconsole or jvisualvm
2. Connect to app server process where logging needs to be enabled/disabled
3. Goto MBeans tab
4. Expand "com.oracle.jdbc"/"diagnosability"
5. Select the available MBean
6. Get the name portion from field "ObjectName" (this would be the loader string, as referenced in sample code here  .

For example, if we have:

Then loader would be "sun.misc.Launcher$AppClassLoader@601bb1"

B) Target the sample application "JDBCDriverLogger.war" (download from here ) to the application server instance on which you want to enable/disable the driver logging.

C) Create a logging configuration, say "myConfig.properties"

######snippet from myConfig.properties - starts##########
######snippet from myConfig.properties - ends##########

D) set the JAVA System property to point to logging configuration file:
         set JAVA_OPTIONS=-Djava.util.logging.config.file=myConfig.properties

E) Restart the application server if necessary. Note this restart might be needed to pickup  file "myConfig.properties" and is one time activity. If this file is already available to the application server, then no need to restart.

F) hit the application on application server instance:

The IP and Port in above URL will correspond to the app server instance where logging needs to be enabled/disabled, and where the applicaton "JDBCDriverLogger.war" was targetted previously

G) Enter loader name (obtained from step A6) in the text field next to "Enter ClassLoader Object instance name:" on the JSP page, and hit "EnableLogging"

This would enable driver level logging on app server instance

H) to disable the logging, enter the same loader name in the text field next to "Enter ClassLoader Object instance name:" on the JSP page, and hit "DisableLogging"


Tuesday, July 31, 2012

Configuring WebLogic GridLink DataSource with secure SCAN URL and secure ONS

Configuring SCAN and ONS on WLS GridLink datasource is already covered in Oracle whitepaper here

In this discussion today, I will list high level steps that are needed to configure secure SCAN and secure ONS on WLS GridLink Datasource instead of using their plain counterparts.

Here are the high level steps:

1) TCPS based listener configuration needs to be implemented on DB server side. Once it is done, we would get a secure SCAN URL. Based on this scan URL, we would get jdbc URL for use with our GridLink datasource, something like:


2) ONS daemons running on RAC nodes need to be secured as well with user certificate (using wallet). Note that it is essential to have ditto wallet configuration in following(or equivalent) files to secure ONS daemon successfully:



Here is sample output of ons.config and ons.config.node1 from node1

[grid@sup1 conf]$ cat ons.config
localport=6100 # line added by Agent
remoteport=6200 # line added by Agent
nodes=node1:6200,node2:6200 # line added by Agent

[grid@sup1 conf]$ cat ons.config.node1
localport=6100 # line added by Agent
remoteport=6200 # line added by Agent
nodes=node1:6200,node2:6200 # line added by Agent

3) With above things in place, just follow the steps in doc:

Now some security specific highlights are:
4) Following needs to added to DOMAIN_HOME/setDomainEnv.sh:

set WEBLOGIC_CLASSPATH=D:\oracle\wls1034\modules\com.oracle.osdt_cert_1.0.0.0.jar;D:\oracle\wls1034\modules\com.oracle.osdt_core_1.0.0.0.jar;D:\oracle\wls1034\modules\com.oracle.oraclepki_1.0.0.0.jar;%WEBLOGIC_CLASSPATH%

What this does is it pulls these jar files into classpath so that WebLogic can play with Oracle wallets.

5) GridLink datasource security related configuration is:

keystore Driver Properties:

a) "identity_against_db.jks" is nothing but Java Keystore that has user certificate (used as client identity)
b) "db_trust.jks" contains root certificate(s) of CA(s) who signed user certificate(s) used by DB process

6) ONS nodes in GridLink datasource used:

7) wallet in ONS configuration on GridLink datasource used:
ons-wallet-file --> D:\apps\rac_db_repro
ons-wallet-password --> welcome1

a) the wallet file is "ewallet.p12", and is located at "D:\apps\rac_db_repro"
b) "ewallet.p12" contains root certificate(s) of CA(s) who signed user certificate(s) used by ONS process

Wednesday, May 9, 2012


WebLogic provides many ways to automate lifecycle of cluster members to achieve server/service high availability without affecting service quality. For instance, we can configure whole server migration to make sure that in an event of physical  machine failure cluster members hosted on that machine are started on another machine.

There are alternative methods to manage server (or process migration, at OS level), or simply, to control the lifecycle activity of  a process. These alternative methods include inbuilt (or packaged) clustering software that comes with various flavors of Linux/Unix.

This post discusses the possible way to make such clustering software control WebLogic instances.

The central thing about this is the use of WebLogic Scripting Tool ( or WLST) script. This WLST script file would be invoked from service script (which is just a specific form of shell script used to execute lifecycle activities of a process) with appropriate argument, like start, stop, status. Script file has methods to interact with nodemnager to start, stop and check the server status. Also, the same WLST script contains logic for LSB compliance, and returns appropriate exit code to the calling service script. Calling service script has to return the same exit codes to the OS, and can also display further diagnostic messages if need be. This WLST script file would need environment variable to get hold of nodemnager and server details. Typically these environment variables would be set in service script, just before it invokes the WLST  script. These variables are discussed in this document towards the end.  If WebLogic domain is spanned over multiple physical servers, then we would have one instance of this WLST script file (in DOMAIN_HOME) per box. And of course, there has to be one instance of service script per WebLogic server instance on a physical box. (Note to wise: it is alright if you do not understand all this at this point, just follow the remaining  post in a hope to understand this).

Here is what needs to be done:
0) Associate all the WebLogic server instances (including Admin Server) with machines and then configure nodemanagers for these machines. Here are the background details on nodemanager:

1. Make sure that you have an entry for the concerned domain in file:

2. Generate "userConfigFile" and "userKeyFile" so that we do not need to use clear-text  username/password in the scripts responsible for starting/stopping/status-checking the WebLogic instances
Here are the steps:
a) First make sure that domain nodemanager credentials(username and password) for this domain are same as that of domain admin. This is so because if they are different then we would have to generate two config and two key files (one for connecting to Admin Server, and another for connecting to nodemanger – review the WLST script to see what I am talking about)
From, admin console, navigate to DOMAIN_NAME>>Security>>General>>Advanced.

Change values for "NodeManager Username:", "NodeManager Password:", and "Confirm NodeManager Password:". Set them to domain administrator credentials

Also, for each server consider setting:
- set Xms, Xmx, XX:MaxPermSize
This is just to make sure that we don’t use the default values (especially for permanent generation which is often less to effectively start the server in RUNNING mode).
b) start ssh terminal, connect to server machine and navigate to domain home. Once here, set the environment from “bin” directory:
/opt/Oracle/Middleware/user_projects/domains/TestDomain1/bin$ . ./setDomainEnv.sh

Now, start WLST using command:
java WebLogic.WLST
c) Once at WLST offline prompt, connect to the running AdminServer
d) Execute following command:

For details on command “storeUserConfig “, refer: http://download.oracle.com/docs/cd/E17904_01/web.1111/e13813/reference.htm#i1064674
e) Also execute following command for each WebLogic instance that exists on this physical machine:
f) execute following command to exit from WLST:
g) Move the two generated files (“root-WebLogicConfig.properties” and “root-WebLogicKey.properties”) into DOMAIN_HOME. Also, place the file “service_helper.py” into DOMAIN_HOME

3. Assuming we have one admin server (“AdminServer”)  and one managed server (“MS1”), duplicate the file “Server” as “AdminServer” and “MS1”  and edit both of them to set following properties:
export DOMAIN_HOME=/opt/Oracle/Middleware/user_projects/domains/TestDomain1
export DOMAIN_NAME=TestDomain1
export SERVER_NAME=AdminServer (or MS1)
export IS_ADMIN=TRUE (“FALSE” if SERVER_NAME is set to MS1)
export ADMIN_URL=t3://mac11.oracle.vm:7001
export NM_HOME=/opt/Oracle/Middleware/wlserver_10.3/common/nodemanager
export NM_HOST=mac11.oracle.vm
export NM_PORT=5556
export USER_CONFIG_FILE=$DOMAIN_HOME/root-WebLogicConfig.properties
export USER_KEY_FILE=$DOMAIN_HOME/root-WebLogicKey.properties

4. Now we simply install the files as service into “/etc/init.d”
5. use following commands to start the services initially:
service AdminSever start
service MS1 start
Note there is a dependence of MS1 on AdminServer. So start MS1 only when AdminServer starts up successfully
6. Repeat following steps for each physical machine where WebLogic domain is scattered:
2b-g, 3, 4, 5
7. Now configure the High Availability solution’s monitoring console (HA MC) to import these two services as LSB services. And from this point onwards, you should be good to control them using MC.

I tried this setup while installing WebLogic instances as service into /etc/init.d, and then working with these instances as LSB instances in DRBD Management Console Release: 0.9.9  on SLES 10, and things look good.
mac11:~ # uname -a
Linux mac11 #1 Fri Sep 4 01:28:03 UTC 2009 x86_64 x86_64 x86_64 GNU/Linux

Since this setup involves checking the status, starting/stopping the servers using WLST and nodemanager these operations do take some time. So one piece of advice goes here; start, stop, status and monitor interval timouts need to be tuned to make things work. Also, if you are thinking of starting/shutting down the servers outside the HA MC, then you still would have to do this using methods that involve the use of nodemanager, otherwise the tracking of WebLogic instances (using nodemanager) would be lost, and things would break up.

Hope this helps you guys to go HA out there!