I was recently testing out the performance of rulesets that were attached to forms, and ran into an issue that had me scratching my head. Certain rulesets were not running or even appearing on forms that they were attached to in the Planning application. Turns out, the rules had been corrupted in the transfer between Calc Manager and the Planning application More...

This post covers the basics of building a Planning app. Building a Planning application can be a straightforward process; however, there are some pitfalls to watch out for. One or two mistakes can lead to some major headaches in trying to decipher what went wrong. Whether a beginner or a seasoned vet looking for a quick refresher, this guide will outline the steps necessary to successfully create/deploy a Planning application. More...

This section will talk about how to dive into debugging critical issues with Oracle EPM.

Start a Problem Log

The most useful habit to develop during issue resolution is to start a detailed log about the issue. Some problems can take days or weeks to resolve and require trying hundreds of different prospective resolution attempts. It is easy for a “small” problem to become a long winded issue. Consequently, it is hard to foresee when the issue will resemble the analogous onion: keep peeling off layers and finding more and more to fix. If the problem log is created initially, all the important details can be captured. Additionally, it is much easier to bring others up to speed (management) and create support tickets when all of the information is documented. This log should include the error as the end user sees it, the error from any logs you are able to capture, screenshots, timestamps, and things that you have tried along with the results.

Reproduce the Issue

The first thing to find out is whether the issue is reproducible. It is very difficult to solve an issue that is not reproducible. Many errors are simply ‘glitches’ and may have been caused by a very improbable event, such as a database hiccup. For instance, a database problem propagates into the Oracle EPM system, forcing it into a bad state. Such a problem may never produce itself again. Consequently, an initial step toward resolution is to restart the Oracle EPM services to bring them back into a ‘known state’. If the problem is not immediately reproducible after the restart, go back to the problem log and record everything you can. This type of issue will need to be profiled over a period of time to try and discover patterns if it occurs again.

The Numerous Logs

Once it is discovered that the issue is not a simple glitch, it is time to start digging. As mentioned previously, the first place to track down the cause of an issue is in the logs. The logs come in various forms. Here is a general breakdown of the log types:

 General

Log Type

Description

Windows Event Viewer

This is helpful for general system related messages. Also some modules built on Windows Technology (DCOM) will log messages here. For example, Financial Management (HFM) and Financial Data Quality Management (FDQM).

Application Logs

The application logs are actually generated by the Hyperion code itself. These often contain the most useful information.

Application Server Logs

This type of log pertains to a Java based Web Application. Most of the Hyperion modules with a web based front end have Application Server Logs. The Application Server Logs run within the WebLogic, Tomcat, or WebSphere container.

Web  Server logs

The web server controls the handoff of web requests between the Hyperion Modules. The best way to use this log is to look for error codes (404, 401… etc) in the web log and review the corresponding URL that was used to ensure it is correct. Sometimes it might be obvious that the URL in the web log has the wrong domain, points to the wrong server, or cannot resolve the context.

 

Start by reviewing the log for the product where the error is occurring. The Application Logs and Application Server Logs will be most useful at first. The goal is to find a useful error message that can be used in the next process to find a resolution to the problem.

Common Log Locations:

Unfortunately, the actual log locations change drastically between recent versions of Oracle/Hyperion products. As stated before, searching for *.log might be useful.

Example Application Server Logs:

Essbase Admin Services

Svr2

/Oracle/Middleware/user_projects/domains/EPMSystem/servers/EssbaseAdminServices0/logs

Workspace

Svr1

/Oracle/Middleware/user_projects/domains/EPMSystem/servers/FoundationServices0/logs

Financial Reporting

Svr1

/Oracle/Middleware/user_projects/domains/EPMSystem/servers/FinancialReporting0/logs

Analytic Provider Services

Svr2

/Oracle/Middleware/user_projects/domains/EPMSystem/servers/AnalyticProviderServices0/logs

Web Analysis

Svr1

/Oracle/Middleware/user_projects/domains/EPMSystem/servers/WebAnalysis0/logs

 

Example Application Logs

Reporting and Analysis Core

Svr3

/Oracle/Middleware/user_projects/epmsystem1/diagnostics/logs/ReportingAnalysis/

 

/Oracle/Middleware/user_projects/epmsystem1/diagnostics/logs/ReportingAnalysis/stdout_console_default.log

 

Essbase

Svr4

/Oracle/Middleware/user_projects/epmsystem1/diagnostics/logs/essbase/

 

 

Sifting Through the Logs:

It helps to know which modules depend on each other in order quickly pick out the respective log files to analyze. The basic idea is to determine which products are interacting and to review each log in detail for messages. It is important to review the logs of the product not only during runtime (as it is happening), but also during startup. Sometimes the fastest way to cut out the fluff is to stop the services, move or delete all the existing logs and start the environment back up. This ensures any log messages are relevant to the issue. Alternatively, one has to sift through potentially large logs looking for timestamps to ensure relevance, which can be daunting.

Product

Depends On

Shared Services

Relational Database, MSAD/LDAP

Lifecycle Management (LCM)

Shared Services, LCM Source/Target applications

Essbase

Shared Services

Hyperion Planning

Shared Services, Essbase, Business Rules, Relational Database per App

Business Rules

Shared Services, Hyperion Planning, Essbase, Relational Database (single database)

Hyperion Financial Management

Shared Services, Relational Database (single database), DCOM (Event Viewer)

Financial Data Quality Management (FDM)

Shared Services, Relational Database per App, Adapters for Essbase, Planning, HFM…etc, DCOM (Event Viewer)

Strategic Finance

Shared Services, Relational Database (optional)

Data Relationship Management

Shared Services, Database Client, Adapters, DCOM (Event Viewer)

 

Found an Error Message!

After discovering the error message, the first thing to ask is does this message make any sense? Try to use it within the context of your problem to solve the issue. Often, it is necessary to use external resources to resolve the issue. Use resources like Google, the Oracle Support Knowledgebase, and the Oracle Forums to further research the issue. Most often there will be information regarding your issue.

Note: If possible do not searching using end user messages, i.e. what the user sees when encountering the error.  Rather, find a detailed message in the logs. The end user messages are usually very generalized and can provide misleading information because of the vast number of issues which might match the general error message.

If there is still a struggle to discover a useful error message, most of the Hyperion modules use a logging mechanism that can be changed into debug mode. The actual method will differ based on product, for instance, most modules use log4j and there is often a .properties file you can change the logging level from “ERROR” or “WARN” to “DEBUG”. For instance, to enable debugging in Hyperion Planning: log into a Planning application, go to Administration -> Manage Properties, Select the System tab, Add the property DEBUG_ENABLED with a value of True. After changing the logging level, the service will need to be restarted to reflect the changes. Turning on application debugging should provide more context clues around what the product is doing at the time and help pinpoint the error.

 

Nothing Found…

If these resources do not help, an Oracle Support Ticket may be required. Additionally, the Oracle Forums can be a good place to post a question.  When creating a support ticket and posting to a forum, please include as much information as possible. This is where the Problem Log will come in handy.

This is a good time to look for updates and patches to the product. Check for patches and updates on http://support.oracle.com. Read the release notes for anything matching your problem. Even if there is nothing coming up, some obscure errors can be solved by simply applying a patch. Not all bugs will be in the release notes for the patch. Oracle’s hpatch process is pretty straight forward, but older environments might take some time to apply the patch. Always read the entire release notes and installation instructions before applying a patch. Also, sometimes patches are not as proven as the initial installers. This is because some patches may have just been release and only tested with a handful of clients. So ensure there is a good backup process in case the patch causes unintended problems. The Oracle hpatch process has a back out feature, but it is not always useful if the patch is half way installed and failed.

Finally, the last part of troubleshooting is intuition. As more problems are encountered and resolved, one can become more confident in resolving upcoming issues. There is no way to have encountered every issue and know the resolution, so the best that you can do is arm yourself with a good knowledge of the architecture, have a set of best practices, and lots of patience for problem solving.

"Validation"

In installment #3 of this series we installed and configured the 11.1.x software.  In this installment we will discuss what Infrastructure Architect will do before the environment is turned over to the development or migration teams.

It is quite frustrating to the developers if the environment is not fully functional when they start using the system.  Additionally, it is very frustrating for the installation architect to have users in the environment as debugging of issues is occurring.  Each installation and configuration project plan should include at least a day or two to review an environment, restart it a few times, check out the logs, and then test the functionality of all installed components.  The number of items to validate depends on the products used and licensed by the client but it should start with the following and adjust as necessary.

  • Shared Services
  • Essbase
  • EAS and Business Rules
  • Planning
  • Financial Reporting
  • Web Analysis
  • Interactive Reporting
  • SQR
  • Workspace
  • Smart View and Provider Services
  • Financial Management
  • Financial Data Quality Management
  • Oracle Data Integrator
  • Data Relationship Management
  • Strategic Finance

The Installation Architect will test the use functionality of each of the installed product to ensure there are no errors.  This activity takes a combination of functional and technical ability.  The installation architect must know how the application works from the interface as well as understanding what any potential errors mean and how they may be corrected.  I’m not suggesting the infrastructure engineer know how to create a P&L report or design a Planning application, but the ability to navigate the user interfaces and test functionality eliminates the problems of encountering them after development has begun.

Early in my exposure to these applications, I’d spend a lot of time with a developer or functional user of the applications to show me how to test the system.  I’d ask them to tell me the first thing they try to do when they get a new environment.  It is always useful to know more about how the applications are used. 

Some of the common problems that occur include the following.
EPMA dimension server does not resolve in Workspace
Shared Services doesn’t find users in Active Directory
Cannot create Planning Application
Cannot create FDM Database
ODI repositories are not available
Common Essbase commands do not work

The solutions to some of these problems may range anywhere from Database Access Permissions, Windows Security Rights, DCOM Settings, or incorrect Active Directory Setup.  Over the past few years working on dozens of installation, I’ve seen all of these.  From encountering many of these, the pre-installation requirements covered in installment #2 have been improved.  Some of these problems don’t arise until functionality is tested.  It’s important to test each installation and environment.  I’ve had situations where the development environment will test out fine and the QA Environment will have issues.  Each installation is usually different from each prior installation because of server settings, security policies, database settings, firewalls, or some other nuance.

If there are problems with the functionality there are a number of resources available to assist in troubleshooting.  I find the Oracle Technology Network Forum to be very useful.  I recommend anyone looking to work in this space, get an ID, and get involved.  You may also find some real useful things on blogs like this or a number of other very experienced bloggers.  There is a wealth of information at Oracle Support in the knowledge base.  In addition, if you have a support agreement with Oracle, register here and you can get support from Oracle.

Assuming everything is functioning as expected, the environment is turned over to the appropriate support person, or maybe support falls on the same individual that did the installation.  Either way, there is a lot of information that needs to be collected.  In the next installment, we’ll look at the information that should be compiled to capture the state of the environment as it was at the end of the installation as well as information that is useful to those that will be using the system.

“Pre-Installation Requirements”

In installment #1 of this guide, we reviewed the architecture considerations and defined a simplistic architecture to use as a reference moving forward.  I recommend you read the previous post before you pick up this one.  I also recommend reading

Oracle Hyperion Enterprise Performance Management System Installation Start Here Release 11.1.1.2.pdf (128 pages)” from the Oracle Documentation Library.

To reiterate our general approach, the Hyperion architecture establishment and installation activities in our organization cover the following five areas.

  1. Defining an Architecture – Work with the client to define the hardware, software, and the distribution of Hyperion components
  2. Provide Pre-Installation Requirements – Provide the client with a detailed list of activities prior to the installation
  3. Installation – Running the installation and configuration utilities
  4. Validation – Perform all functional activities necessary to validate the environment readiness
  5. Documentation – Provide the client with all the details of the environment as it is configured.

In this post, I will go through step 2 that the Hyperion architect, should deliver.  Steps 3-5 will be available in the coming weeks.  For the sake of simplicity I will be using the example of a common installation, primarily Hyperion Planning, Hyperion Financial Management (HFM), and the core BI applications.

As part of any installation, some items need to take place before the Fusion Installer is started.  I like to create a checklist of things that need to be done.  Often times these things are out of my control and I will rely on Database Administrator, Network Administrators, and other System Administrators.  This checklist contains the following elements.

  • Web Application Server Specifications
  • Relational Repository Information
  • General System Administration
  • Network Information
  • Additional Components
  • DCOM Configuration
  • IIS and .NET Configuration

I’ll start with the Web Application Server Specification.  Once the web application server platform is chosen from the table below, the installation and configuration often falls on System Administrators.  Items such as clustering, system account management, and JVM setting are managed outside of the Hyperion installation.  Other times, I’ll get admin access and manage it myself.  The first item to do is to validate the application server is certified.  This is directly from Oracle Enterprise Performance Management System - Supported Platforms Matrices “Oracle Enterprise Performance Management System, Fusion Edition Release 11.1.1.2)” in the Oracle document library.  I recommend reviewing this document.  It can change from release to release.

 

Server

Notes

Oracle Application Server 10g (10.1.3.3.x) a

If Oracle Application Server is used as the Web application server, Oracle HTTP Server is also required.  Profitability and Cost Management supports only Oracle Application Server 10.1.3.x.

Oracle WebLogic Server 9.2 (MP1 minimum) / 9.2.xb

Shared Services requires WebLogic Server patch CR283953” for all platforms. You can obtain this patch at the BEA web site.

IBM WebSphere 6.1.0.17 / 6.1.x C

 


Embedded Java container d

 


a Supports these editions: Java, Standard One, Standard & Enterprise. Includes support for Oracle Application Server Single Sign-On.

b WebLogic Express is supported for each supported version of WebLogic Server; non-base versions are supported only via manual deployment.

c WebSphere Express, ND, and XD Editions are supported for each supported version of WebSphere; ND and XD are supported only via manual deployment.

d For this release, Apache Tomcat 5.5.17 is the embedded Java container that is installed automatically on all platforms. Apache Tomcat is supported only in this capacity. If future EPM System releases embed different Java application servers, Apache Tomcat will no longer be supported. For deployments that require high availability or failover, Oracle recommends using a commercially supported Web application server that supports high availability and failover.

I request the URL and authentication information since this will be needed during the deployment.  If I am doing a manual deployment, I will request contact information from the web application server administrator and work in collaboration on the deployment.

The next item on my checklist is to get the Relational Repositories Information set up.  This is mostly straightforward.  In general, I like to create a tablespace/database for each component ((Hyperion Foundation, Essbase Admin Services / Business Rules, EPMA, Planning, Financial Management, etc).  A distinct tablespace/database for each component makes it easier to manage in my opinion.  Although it may not be strictly necessary, the documentation does not seem to be clear on the matter.  I say ‘better safe than sorry’.  For the installation and deployment, I’ll need credentials for each table.  Based upon some Q&A, I’ll make initial size recommendations. 

The target installation servers have a General System Administration checklist containing the information that I’ll need to execute the installation.  This is made of the following items.

  • Operating Systems version/build
  • Account on each server to run the Hyperion services and account requirements
  • External Authentication information (MSAD, LDAP, or OID if applicable)
  • Drive/Volume information identified for installation of the Hyperion software.
  • DCOM and .NET account information if HFM or FDM are to be installed

Next, I identify the Network Information necessities for appropriate communication between servers.  This includes IP addresses, DNS information, validation of name resolution, trace between servers, subnet configuration, etc.  This is vital so the components can communicate via Fully Qualified Domain Name, Short Name, and IP address.  Some components use different variations of name resolution probably because the components were developed separately and have not been fully standardized.

In addition to the Hyperion Software, Web Application Servers, and Relational Repositories there are a few Additional Components that need to be installed.  A PDF writer is needed for Reports Server to render .pdf reports in Workspace.  This can be GhostScript or Acrobat distiller.  I suggest referring to the “Start Here” documentation to see what is currently supported but we often go with GhostScript due to its cost.

For the Windows Administration, we provide the DCOM Configuration information needed to support FDM, EPMA, and FDM.  This includes the DCOM account information, permissions, and authentication information.  Although this is spelled out in detail in the “Start Here” manual, I like to provide step-by-step information with screen shots since DCOM is often confusing…well it is to me at least.

The last thing we review is the IIS and .NET Configuration.  IIS is often not installed as part of a standard OS build.  We make sure this requirement is specified, ensuring .NET is installed, and validate it is the appropriate version.

As with any installation, I recommend the Installation Architect read, and re-read, the Hyperion Manuals on there own rather than relying on this information or intuition.  It can always change and your installation may have some caveats that I have not covered.  For our purposes, with all the above activities completed and validated, we should be ready to start laying out the binaries and start the Hyperion Installation.  We will review the Fusion Installer and Hyperion Configuration Utility in our next installment. 

 

Hyperion Enterprise Performance Management System Installation and Configuration Guide Release 11.1.1.2 (252 Pages) and Oracle Hyperion Enterprise Performance Management System Manual Deployment Guide Release 11.1.1.2 (160 Pages) are a wealth of knowledge.  I know because I have read a great deal of it and keep them quite close during each engagement.  Most of what you need to know is available in the documentation and a good deal of additional knowledge can be found on the support forums.  I do not want to rehash the specifics of the installation and configuration.  There are just too many variables to cover everything without reposting most of the already prepared material.  What I hope to provide here is just a primer on architecture considerations and general installation activities with this release.

The Hyperion architecture establishment and installation activities in our organization cover the following five areas.

  1. Defining an Architecture – Work with the client to define the hardware, software, and the distribution of Hyperion components
  2. Provide Pre-Installation Requirements – Provide the client with a detailed list of activities prior to the installation
  3. Installation – Running the installation and configuration utilities
  4. Validation – Perform all functional activities necessary to validate the environment readiness
  5. Documentation – Provide the client with all the details of the environment as it is configured.

In this post, I will go through step 1 that you, as the Hyperion architect, should deliver.  Steps 2-5 will be available in the coming weeks.  For the sake of simplicity I will be using the example of a common installation, primarily Hyperion Planning, Hyperion Financial Management (HFM), and the core BI applications.

Defining the architecture is somewhat difficult, especially if you do not have a good understanding of your organizations’ usage patterns or platform preferences.  Below are the core questions that need to be answered to define the necessary architecture.

How many servers will be needed?

The number of servers depends on a lot of things.  Below is a sample of some of the questions I typically ask when recommending the server infrastructure.

  • How many users do you have?
  • Will you be leveraging an existing Relational Database Platform?
  • What products did you purchase?
  • What products do your users use most frequently?
  • Will you be using loadbalancing?

Let’s assume the answer is 100 users, the client has an existing Oracle database I can leverage, the client purchased HFM and Planning, and the client uses a lot of Financial Reports.

With these assumptions, you will want to make sure the Workspace and Reporting platforms are a decent size and the reporting application services reside on a separate server from the application servers.  In addition, since you are going to leverage an existing relational database, you can plan on having a stand-alone data tier for Planning and Essbase.  Just to further simplify the matter, let’s assume there is no load balancing and clustering.  For our example discussion, I will assume 3 servers will be included in the architecture.

What would be the system specifications and Operating System?

Once again, the answer to this question can vary widely based on the answers below.

  • What operating systems does your organization support?
  • How long do you intend to use this hardware?
  • Will you be using virtual machines?
  • Will you be using shared or local disks?
  • How much data do you have now and what does your growth look like?

Keep in mind that HFM requires a Windows Server so I will need at least one of those.  It may be simpler from an administrative perspective to have a homogeneous OS infrastructure but it is not necessary.  If you have no preference, this is what I would typically go with.  If you have a propensity for some flavor of UNIX, then I would go that route where applicable.  Many organizations keep hardware in place for up to 3 years.  With that in mind, I think the best approach is to buy the most robust and powerful hardware you can afford.  The cost of migrating in 24 months will often be greater than the hardware spend.  VMware is quickly growing in popularity and I see deployments in virtual environments quite often with success.  Although Essbase does not require a physical server, I recommend that it not reside on virtual machines due to higher disk I/O.  It is also the application that is expecting the most growth.

What Web Application Server should we use?  Should it be Oracle Application Server 10g, WebSphere, WebLogic, or the Embedded Java container (Apache Tomcat 5.5.17)?

  • What does your organization support?
  • Do you have a license for anything?
  • Will you need clustering?

First, some comments in favor of the Apache Tomcat server.  It is free and Hyperion does do some development on that platform.  So far, I have found it to be stable in most situations.  If your organization has a Web Application group, it may not be standardized on a version that is required by Hyperion.  In these instances, it may make sense to deploy Tomcat as opposed to a version of WebLogic or WebSphere that your organization does not support.  Additionally, if the organization is upgrading its application server farm in the future, Hyperion may be an outlier.

Now for the downside, there isn’t really any help desk you can call for Tomcat outside of Hyperion.  Unless your organization has a few other Tomcat servers, there is likely to be a learning curve in the administration and support.  Then, there is always the question of scalability.  Clustering with Tomcat is quite a bit more complex.  Our common recommendation on application servers is that you should use what you know.  If you do not know anything about application server administration and do not have a huge user base, Tomcat is probably your best choice due to its ease of installation.

One rule of thumb is that you can expect a single Tomcat instance to handle 25-30 concurrent threads comfortably.  The practical maximum is 50 per instance.  If you are using other sorts of Java servers, it is a good idea to review with those administrators their experience with free servers.  In a development environment, one could benchmark how Hyperion EAR and WAR files behave in comparison to those of other Java apps.

With all these assumptions, I deliver a Viso diagram detailing which components go on which servers with general recommendations for server sizing and operating systems. 

In our 2nd installment, I will review the pre-installation activates that need to occur before the installation begins.