A Troubleshooting the Upgrade

This appendix describes some common procedures for troubleshooting a failed upgrade, domain reconfiguration or server start issues.

A.1 Reviewing the Release Notes

Make sure that you review the release notes to determine if any known issues could be impacting your upgrade. You can find the release notes in the Oracle Fusion Middleware 12c (12.1.3) library.

A.2 Resolving Server Start Errors

If servers do not start, or they start in AdminMode, the cause is most likely that the setDomainEnv.sh changes from the previous environment were not reapplied to the 12c domain. Compare the setDomainEnv file from 11g to the new 12c setDomainEnv file and then add any custom changes after the upgrade.

For more information, see "Re-apply Customizations to Startup Scripts".

A.3 Recovering From a Failed Upgrade

Recovering from a failed upgrade depends on when the error(s) occurred. Review the following to determine how to recover:

  • If there are errors while running the Upgrade Assistant to upgrade _SOAINFRA schema, you must fix the errors in the schema and rerun batch jobs.

    Note that this recovery method only applies when you are running the Upgrade Assistant for the first time and you selected the Schema option.

  • If there are errors while running the Reconfiguration Wizard, you must restore from source environment and restart the upgrade from the beginning.

  • If there are errors while running the Upgrade Assistant to upgrade WebLogic Component Configurations option, then you can fix the errors and rerun the Upgrade Assistant. The second time you run the Upgrade Assistant there is no need to restore from backup and restart the upgrade process from the beginning. This process is reentrant.

  • If there are errors while running the Upgrade Assistant to upgrade schemas, and the error occurs during the upgrade phase, you will have to restore from backup, correct the issues, and then restart the upgrade from the beginning. If the error occurs during the examine phase, however, you can correct the issues and restart the Upgrade Assistant. Errors that occur prior to the upgrade phase are reentrant.

For more information on troubleshooting your upgrade, see "General Troubleshooting Guidelines" in the Upgrading with the Upgrade Assistant.

BAM Upgrades Only:

If you received the CFGFWK-60950 error, rename the BAM templates as described in "Rename the Oracle BAM templates before upgrading the 11g schemas." and launch the Reconfiguration Wizard again.

If you received this error, you will need restore your entire pre-upgrade environment, perform the necessary pre-upgrade tasks and then perform the steps in the section listed above before you can attempt the reconfiguration process again.

A.4 Recovering from a Failed BAM Upgrade

This section applies only when there are BAM servers in the domain. As part of BAM Upgrade, you can export BAM archives from 11g and import them into BAM 12c. If you receive any errors during this process, use this section to try to resolve the issues.

A.4.1 Error Handling: 11g Process Cubes to BAM 12c Star Schema Migration

You may be able to resolve common errors by rolling back the data changes and rerunning the scripts with modified options.

Rollback All Data Changes:

  1. Open a SQL session on the SOA database.

  2. Log in as the SOAINFRA schema user and run the following script to roll back any data changes:

    "<exportDir>/rollBackBPMProcessCubesMigration.sql"
    

Review the recommendations for your operating system:

A.4.2 Error Handling for UNIX Operating Systems

If any unexpected errors occurred during migration, you can try the following steps to correct the issues:

For Errors that Occurred During the Import Phase:

If the error occurred while importing archives to BAM 12c, rerun the shell script "migrateBPMProcessCubes.sh" as described in Migrate 11g Process Cubes to BAM 12c Process Star Schema (BPM Users Only), but add the "-importOnly" option. This can save time by skipping the export step.

For example:

cd $ORACLE_HOME/bam/bin
./migrateBPMProcessCubes.sh -serverUrl <BAM 12c server url> -serverPort <BAM 12c server port> -serverUserName <BAM 12c server user> -dbUrl <soa db jdbc url> -dbUserName <soainfra schema username> -exportDir <export dir> [-exportType ALL] [-importOnly]

For Errors that Occurred During the Export Phase:

If the error occurred while exporting archives from BPM Process cubes, perform the following tasks:

  1. Create a backup copy of the export directory defined as (<exportDir>)

  2. Delete the contents of the <exportDir>.

  3. Rerun the shell script "migrateBPMProcessCubes.sh" as described in Migrate 11g Process Cubes to BAM 12c Process Star Schema (BPM Users Only), but remove the "-importOnly" option.

    For example:

    cd $ORACLE_HOME/bam/bin
    ./migrateBPMProcessCubes.sh -serverUrl <BAM 12c server url> -serverPort <BAM 12c server port> -serverUserName <BAM 12c server user> -dbUrl <soa db jdbc url> -dbUserName <soainfra schema username> -exportDir <export dir> [-exportType ALL]
    

Additional Information:

You can also try the following to help resolve any issues:

  • After importing each archive to BAM 12c, review the bamcommond.log.* files located in the $ORACLE_HOME/bam/bin directory to make sure no errors occurred.

  • Review the migration logs located in the <exportDir>/MigrationLogs.* :

A.4.3 Error Handling for Windows Operating Systems

Roll back all of the data changes as described above, and then try the following:

For Errors that Occurred During the Import Phase:

Reimport the archives as described in the following sections:

For Errors that Occurred During the Export Phase:

If the error occurred while exporting archives from BPM Process cubes, perform the following tasks:

  1. Create a backup copy of the export directory defined as (<exportDir>)

  2. Delete the contents of the <exportDir>.

  3. Rerun the shell script "migrateBPMProcessCubes.sh" as described in Migrate 11g Process Cubes to BAM 12c Process Star Schema (BPM Users Only), but remove the "-importOnly" option.

    For example:

    java -cp
    
    %ORACLE_HOME%\soa\modules\oracle.bpm.runtime_11.1.1\oracle.bpm.analytics.metrics.interface.jar;
    %ORACLE_HOME%\soa\modules\oracle.bpm.runtime_11.1.1\oracle.bpm.analytics.metrics.model.jar;
    %ORACLE_HOME%\oracle_common\modules\oracle.jdbc_12.1.0\ojdbc6.jar;
    %ORACLE_HOME%\bam\modules\oracle.bam.client\bam-client.jar;%ORACLEHOME%\bam\lib\bam-schema.jar;
    %ORACLE_HOME%\soa\modules\oracle.bpm.runtime_11.1.1\oracle.bpm.analytics.metrics.dataobject.jar;
    %ORACLE_HOME%\soa\modules\oracle.bpm.runtime_11.1.1\oracle.bpm.hwfanalytics.dataobject.jar
    
    oracle.bpm.metrics.dataobject.migration.application.Migrate11gProcessCubesto12cDO -url <soa db jdbc url> -userName <soa schema user name> -exportDir <export directory path> [-exportType ALL]
    
  4. Repeat the remaining migration steps in Migrate 11g Process Cubes to BAM 12c Process Star Schema (BPM Users Only).

A.5 Rewiring Components After OWSM Cluster Upgrade

If you modified the wsm-pm targeting for OWSM, and the wsm-pm is now targeted only to the OWSM cluster post-upgrade, then you must rewire the components as described in "Using Cross-Component Wiring for Auto-Discovery of Policy Manager" in Oracle WebLogic ServerSecuring Web Services and Managing Policies with Oracle Web Services Manager.

A.6 Encryption Issues During Upgrade

If you received the following error message during the reconfiguration, you may need to apply additional policy files to the JDK and restart the upgrade from your backup:

JPS-06513: Failed to save keystore. Reason oracle.security.jps.service.keystore.KeyStoreServiceException: Failed to perform cryptographic operation

To prevent this error from reoccurring, apply the policy files before the subsequent upgrade using the information in Section 2.1.6, "Using Enhanced Encryption (AES 256)".

A.7 Troubleshooting Oracle Service Bus

If you experience post-upgrade issues with Oracle Service Bus, review the following and apply any relevant solutions.

A.7.1 HTTP 404 Error After OSB Upgrade with OHS as Cluster Frontend Host

If you configure Oracle HTTP Server (OHS) as a cluster domain frontend host, then you must add the following code to the OHS configuration file (ohs.confg):

 <Location /sbconsole>
  SetHandler weblogic-handler
  WebLogicCluster [ADMIN_SERVER_HOST]:[ADMIN.SERVER:PORT]
</Location>
<Location /servicebus>
  SetHandler weblogic-handler
  WebLogicCluster [ADMIN_SERVER_HOST]:[ADMIN.SERVER:PORT]
</Location>

Where ADMIN.SERVER:PORT is the machine name, server name and port number used for the OHS.

mymachine.us.mycompany.com:7001 as shown in this sample code example:

<Location /sbconsole>
  SetHandler weblogic-handler
  WebLogicCluster mymachine.us.mycompany.com:7001
</Location>
<Location /servicebus>
  SetHandler weblogic-handler
  WebLogicCluster mymachine.us.mycompany.com:7001
</Location>

A.7.2 HTTP 404 Error When Accessing OSB Console

Prior to 12c, the OSB console was accessed using the following URL: http://[HOST]:[PORT]/sbconsole

In 12c, the OSB Console URL has changed to: http://[HOST]:[PORT]/servicebus.

After the upgrade, if you enter http://[HOST]:[PORT]/sbconsole, it should redirect to http://[HOST]:[PORT]/servicebus.

If the redirect fails, and you receive a HTTP 404 error, try direclty entering the 12c URL: http://[HOST]:[PORT]/servicebus.

A.8 Upgrading Unsupported Domains with the Upgrade Assistant

If you receive an error from the Upgrade Assistant stating that the specified domain cannot be upgraded, contact Oracle Support for more information. Supported domain configurations are described in Understanding Domain Upgrade Restrictions.

Do not attempt to upgrade or schemas or domain configurations in an unsupported domain.

A.9 Business Rules Audit Trail Not Showing After Instance Upgrade

The audit trail for upgraded 11g instances of the Decision Service Component will not be available post-upgrade. The audit trail for new 12c instances will continue to display.

A.10 Resolving a Coherence Cache Exception

If you see the following WebLogic Cache Provider Coherence exception then it is likely that you are not following an enterprise deployment topology recommendation to specify a specific ListenAddress.

When you see this exception, you must set the ListenAddress for your managed server as shown below:

Exception:

weblogic.cacheprovider.coherence.CoherenceException:
  at weblogic.cacheprovider.coherence.CoherenceClusterManager.ensureWKAAddresses(CoherenceClusterManager.java:510)
  at weblogic.cacheprovider.coherence.CoherenceClusterManager.configureClusterService(CoherenceClusterManager.java:236)
  at weblogic.cacheprovider.CacheProviderServerService.bootCoherenceFromWLSCluster(CacheProviderServerService.java:225)
  at weblogic.cacheprovider.CacheProviderServerService.initCoherence(CacheProviderServerService.java:94)

Resolution:

  1. Log in to the WebLogic Server Console.

  2. Navigate to Servers.

  3. Locate the Managed Servers (SOA or OSB, for example).

  4. Modify the Listen Address from localhost to 127.0.0.1 or provide the actual machine name.

A.11 WSDL Generated Missing Elements for Custom Exception

If your EJBs contain custom exceptions, and you export the Web Service Description Language (WSDL) file from your EJB business service, the generated WSDL file will not have the custom exception properties in it. You will need to manually edit the WSDL file to include these custom exception properties after the upgrade.

The issue is limited only to the WSDL generation part of the file. During runtime, the custom exception thrown from the EJB will be mapped to the respective elements in the SOAP fault. The response payload will have the elements populated corresponding to the properties of the custom exception.

A.12 Connecting to the ServerSocket through Remote Clients

There is a change in behavior in which the ServerSocket is created when you upgrade from Oracle Release 11g to Release 12g. Because of this, remote clients might not able to connect to the ServerSocket when the host name is configured as localhost. As a workaround, the localhost should be changed to host name.

For more information, see "Configuring Oracle Socket Adapter" Understanding Technology Adapters.