[Troubleshooting] Oracle Apps R12.2 Online Patching ADOP : Prepare Phase Issue

TroubleshootingADOP (R12.2)
Oracle

Share Post Now :

HOW TO GET HIGH PAYING JOBS IN AWS CLOUD

Even as a beginner with NO Experience Coding Language

Explore Free course Now

Table of Contents

Loading

This post covers, Online Patching (ADOP) issue reported by one of the trainees in Oracle AppsDBA Training with the prepare phase.

Note: There are five phases in Online Patching (ADOP) in R12.2  PREPARE -> APPLY -> FINALIZE -> CUTOVER  -> CLEANUP .

  • To know more about Online Patching (ADOP) click here

Troubleshooting ADOP Prepare Phase Issue:

1. Run Prepare Phase:

Any Patching Cycle starts with Prepare Phase and to start it run below command

adop phase=prepare

Prepare phase is failing with error message

    [UNEXPECTED]fs_clone has failed.
        [UNEXPECTED]Error calling runPendingConfigClone subroutine.

2. Look at ADOP logs:

First thing in troubleshooting is to look at log file (in this case ADOP log file)

Note: Kindly refer this post to know more on log file location in R12.2 Oracle Apps R12.2 Log Files Location every Apps DBA must know

A) Error in adop logs: 

/u01/oracle/PRD122/fs_ne/EBSapps/log/adop/6/prepare_20171109_032332/PRD122_1705ecloud05/log/adop.log
Note: Here 
  • /u01/oracle/PROD is Oracle EBS install base directory
  • fs_ne Non-Editioned File System
  • 6 – adop session id
  • prepare_20171109_032332 – <phase>_<date>_<time>
  • PRD122_1705ecloud05 – CONTEXT NAME

Oracle Apps R12.2 has Dual File System (fs1 & fs2), which helps organizations to have less downtime during patching.  By introducing duplicate file system, Oracle has reduced patching downtime and improved availability of the system.

We can focus on three major parts of the E-Business Suite 12.2 file system:

  • File System 1 (fs1):  /u01/oracle/PROD/fs1
  • File System 2 (fs2):  /u01/oracle/PROD/fs2
  • Non-Editioned File System (fs_ne):  /u01/oracle/PROD/fs_ne

Thu Nov  9 03:34:15 2017

*******FATAL ERROR*******
PROGRAM : (/u01/oracle/PRD122/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Thu Nov  9 03:34:15 2017
FUNCTION: main::migrateCloneComponentApply [ Level 1 ] 
ERRORMSG: /u01/oracle/PRD122/fs1/EBSapps/comn/adopclone_1705ecloud05/bin/adclone.pl did not go through successfully.
[UNEXPECTED]Error occurred running “perl /u01/oracle/PRD122/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/oracle/PRD122/fs1/inst/apps/PRD12111_1705ecloud05/appl/admin/PRD12111_1705ecloud05.xml -patchcontextfile=/u01/oracle/PRD122/fs2/inst/apps/PRD12111_1705ecloud05/appl/admin/PRD12111_1705ecloud05.xml -promptmsg=hide -console=off -mode=migrate -sync_mode=copy -sessionid=6 -timestamp=20171109_032332 -outdir=/u01/oracle/PRD122/fs_ne/EBSapps/log/adop/6/20171109_032332/prepare/1705ecloud05”
        [UNEXPECTED]occurred during CONFIG_CLONE Patch File System from Run File System, running command: “perl /u01/oracle/PRD122/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/oracle/PRD122/fs1/inst/apps/PRD12111_1705ecloud05/appl/admin/PRD12111_1705ecloud05.xml -patchcontextfile=/u01/oracle/PRD122/fs2/inst/apps/PRD12111_1705ecloud05/appl/admin/PRD12111_1705ecloud05.xml -promptmsg=hide -console=off -mode=migrate -sync_mode=copy -sessionid=6 -timestamp=20171109_032332 -outdir=/u01/oracle/PRD122/fs_ne/EBSapps/log/adop/6/20171109_032332/prepare/1705ecloud05”.
        [PROCEDURE] [START 2017/11/09 03:34:15] Updating status of patches
        [STATEMENT] SQL statement : ”  update ad_adop_session_patches
  set status=’F’
  where applied_file_system_base = ‘/u01/oracle/PRD122/fs1’ and
  patch_file_system_base = ‘/u01/oracle/PRD122/fs2′ and bug_number=’CONFIG_CLONE’ and status <> ‘Y’
  and appltop_id=195 and node_name=’1705ecloud05′”
        [PROCEDURE] [END   2017/11/09 03:34:16] Updating status of patches
 [STATEMENT] SQL statement : ”  update ad_adop_session_patches
  set status=’F’
  where applied_file_system_base = ‘/u01/oracle/PRD122/fs1’ and
  patch_file_system_base = ‘/u01/oracle/PRD122/fs2′ and bug_number=’CONFIG_CLONE’ and status <> ‘Y’
  and appltop_id=195 and node_name=’1705ecloud05′”
        [PROCEDURE] [END   2017/11/09 03:34:16] Updating status of patches
        [UNEXPECTED]fs_clone has failed.
        [UNEXPECTED]Error calling runPendingConfigClone subroutine.

B) Error in fs_clone logs: 

Note: Prepare Phase, creates a clone of Patch File System from Run File System, and as per the previous log, an error was during this Cloning so we need to look at clone logs.

$INST_TOP/admin/log/clone/FSCloneApplyAppsTier_<timestamp>.log

 START: Instantiating the OHS Config.
Running /u01/oracle/PRD122/fs2/FMW_Home/webtier/perl/bin/perl  -I/u01/oracle/PRD122/fs2/FMW_Home/webtier/perl/lib/5.10.0 -I/u01/oracle/PRD122/fs2/FMW_Home/webtier/perl/lib/site_perl/5.10.0 -I/u01/oracle/PRD122/fs2/EBSapps/appl/au/12.0.0/perl -I/u01/oracle/PRD122/fs2/FMW_Home/webtier/ohs/mod_perl/lib/site_perl/5.10.0/x86_64-linux-thread-multi /u01/oracle/PRD122/fs2/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkSetOHSConfig.pl cfgCloneInstance -ctxfile=/u01/oracle/PRD122/fs2/inst/apps/PRD12111_1705ecloud05/appl/admin/PRD12111_1705ecloud05.xml -archvloc=/u01/oracle/PRD122/fs1/EBSapps/comn/adopclone_1705ecloud05/FMW/OHS/ohsarchive.jar -mvpln=/u01/oracle/PRD122/fs1/EBSapps/comn/adopclone_1705ecloud05/FMW/OHS/moveplan.xml -t2plogloc=/u01/oracle/PRD122/fs1/inst/apps/PRD12111_1705ecloud05/admin/log/clone/ohsT2PApply -logdir=/u01/oracle/PRD122/fs1/inst/apps/PRD12111_1705ecloud05/admin/log/clone -console=off -promptmsg=hide
Script Executed in 37703 milliseconds, returning status 255
ERROR: Script failed, exit code 255
Error Message  :1
  [PLUGIN][OHS] – ERROR – Nov 9, 2017 03:34:15 – CLONE-26009   OHS T2P failed.
  [PLUGIN][OHS] – CAUSE – Nov 9, 2017 03:34:15 – CLONE-26009   Unable to start OS component.
  [PLUGIN][OHS] – ACTION – Nov 9, 2017 03:34:15 – CLONE-26009   Check clone log and error file and ohs log file /u01/oracle/PRD122/fs2/FMW_Home/webtier/instances/EBS_web_PRD12111_OHS1/diagnostics/logs/OHS/EBS_web_PRD12111/console~OHS~1.log for root cause.

C) Error in OHS logs: 

Note: Clone process clones, Fusion Midddleware (WebLogic & OHS) as per the previous log, the error was during this startup of OHS server so we need to look at OHS logs.

/u01/oracle/PRD122/fs2/FMW_Home/webtier/instances/EBS_web_PRD12111_OHS1/diagnostics/logs/OHS/EBS_web_PRD12111/console~OHS~1.log

 /u01/oracle/PRD122/fs2/FMW_Home/webtier/ohs/bin/apachectl startssl: execing httpd
httpd.worker: bad user name applmgr11

3. Root Cause:

Root cause as identified in previous OHS log was, wrong owner permission for OHS files

Checked httpd.conf file and wherever it is having applmgr11,  changed to applmgr user permission

 cd /u01/oracle/PRD122/fs1/inst/apps/PRD12111_1705ecloud05/appl/admin/cutover/OHS/EBS_web_PRD12111
[root@1705ecloud05 EBS_web_PRD12111]# grep applmgr11 *
httpd.conf:        User applmgr11
[root@1705ecloud05 EBS_web_PRD12111]# chown applmgr:dba httpd.conf
[root@1705ecloud05 EBS_web_PRD12111]#
   User applmgr
        #Group GROUP_TEMPLATE
</IfModule>

4. Change in Context File

Note: After updating user to applmgr in httpd.conf, OHS start-up was still failing

So checked $CONTEXT_FILE and below two entries having applmgr11 user on both run (fs1) and patch (fs2) file system and causing the issue

<appsuser oa_var=”s_appsuser” osd=”unix”>applmgr11</appsuser>
<sysadminmail oa_var=”s_sysadmin_mail” osd=”unix”>applmgr11@1705ecloud05.k21academy.com</sysadminmail>
/u01/oracle/PRD122/fs1/inst/apps/PRD12111_1705ecloud05/appl/admin/PRD12111_1705ecloud05.xml

Note: $CONTEXT_FILE, captured all configuration parameters. In Oracle R12, we are having two context file:

Application context file called the <CONTEXT_NAME>.xml contains the configuration information for the application tier & is located in $APPl_TOP/admin for each node of application tier.

Database context file called the <CONTEXT_NAME>.xml contains the configuration information for the database tier & is located in $ORACLE_HOME/appsutil/

Fix:

1. Changed user to applmgr on both run and patch file system under $CONTEXT_FILE and run autoconfig, then restart prepare phase again

<appsuser oa_var=”s_appsuser” osd=”unix”>applmgr</appsuser>
<sysadminmail oa_var=”s_sysadmin_mail” osd=”unix”>applmgr@1705ecloud05.k21academy.com</sysadminmail>
/u01/oracle/PRD122/fs1/inst/apps/PRD12111_1705ecloud05/appl/admin/PRD12111_1705ecloud05.xml

Run Prepare Phase Again:

adop phase=prepare restart=yes

The Node Manager is already shutdown

NodeManager log is located at /u01/oracle/PRD122/fs2/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1

adnodemgrctl.sh: exiting with status 2

adnodemgrctl.sh: check the logfile /u01/oracle/PRD122/fs2/inst/apps/PRD12111_1705ecloud05/logs/appl/admin/log/adnodemgrctl.txt for more information …

The prepare phase completed successfully.

adop exiting with status = 0 (Success)

Checked ADOP Status 

adop -status

ADOP (C.Delta.8)
Session Id: 6
Command: status
Output: /u01/oracle/PRD122/fs_ne/EBSapps/log/adop/6/20171109_085910/adzdshowstatus.out
===============================================================
Node Name       Node Type  Phase           Status          Started             Finished      Elapsed
————— ———- ————— ————— ——————– ——————– ————
1705ecloud05    master     PREPARE         COMPLETED       2017/11/06 22:41:13 2017/11/09 08:47:02  58:05:49
APPLY           NOT STARTED
FINALIZE        NOT STARTED
CUTOVER         NOT STARTED
CLEANUP         NOT STARTED

File System Synchronization Type: Light

adop exiting with status = 0 (Success)

Related/Further Readings

Next Task For You

We cover Oracle E-Business R12.2 Architecture & concepts in our Oracle Apps DBA For Beginners Training along with the Installation, Patching, Cloning, and Troubleshooting and also, Database upgrade to 19c  and much more including the hands-on labs you must perform to upgrade your skills and get a good job with a high package.

Begin your journey towards becoming an Apps DBA by joining our FREE Masterclass on How To Learn Oracle Apps DBA (R12) & It’s New Features.

Picture of mike

mike

I started my IT career in 2000 as an Oracle DBA/Apps DBA. The first few years were tough (<$100/month), with very little growth. In 2004, I moved to the UK. After working really hard, I landed a job that paid me £2700 per month. In February 2005, I saw a job that was £450 per day, which was nearly 4 times of my then salary.