There has been an issue discovered when installing on the AIX machine. This is a known issue with respect to running an Installshield Multi-Platform installer on an AIX 5.3 machine.
If the process creates a core dump when executing the installation of the agent on your AIX 5.3 machine. Here is the important information from the core dump of the java process:
SIGSEGV received at 0x362220e4 in /opt/JDEdwards/Install/ismp001/libaixppk.so. Processing terminated.
If you see the above in the dump file then use this link to solve the problem. Follow the instructions closely and this should allow you to install the agent. It is a fix to the libaixppk.so for ISMP. Here is the link:
http://knowledge.macrovision.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=Q111262&sliceId=
Thursday, March 20, 2008
Thursday, March 13, 2008
iSeries EnterpriseOne Enterprise Server - Change Component processing by the Management Agent
The management of an EnterpriseOne Enterprise Server is handled on all supported platforms. This tips section will speak to the iSeries specific Change Component process. As most know, the process for doing anything on the iSeries (AS400) machine is quite different than any other platform. I will talk to the steps (commands) that are used during the process of changing out a tools release using Server Manager.
Processing that occurs when you choose to register a pre-8.97 version of the Enterprise Server with Server Manager and then go to upgrade your server tools code to 8.97, Server Manager will automatically save the pre-8.97 version into a software component. That software component could subsequently be used to change from the current version back to the pre-8.97 version. So, Server Manager will allow you to roll back to that version that was registered when you first installed and setup your management domain. This is not specific functionality for the iSeries, but is good information to note as we understand what processing is going on in Server Manager. (NOTE: All work for any instance is done inside the agents jde_home\targets\"instancename" directory. Remember this note throughout the processing of the change component.)
Now, what happens during the processing to change out an iSeries EnterpriseOne Enterprise Server's tools code? First thing to note is that logging is fairly detailed as to the actual commands that are issued to the iSeries. Additionally, the return messages, even when successful, are logged to the agents log file. This is good for all troubleshooting. Second thing to note, as with any platform, the processing of changing out a tools release does a backup of the release being changed out as a first step into a temporary folder. This is for rollback purposes. If a step in the process of changing out the tools release fails a rollback will be executed. Now that you have these notes in the back of your mind, we can talk about the steps in the process.
First in processing is the backup itself. A save file is created for both the QSYS and IFS file system related to the EnterpriseOne code base, CRTSAVF commands. Next, if a printqueue directory exists in the IFS, a temp directory is created (MKDIR command) and the contents "moved" to the temp dir (MOV command). The reason for the quotes around the moved is based on the fact that the contents are not copied as this operation could take very long if the printqueue directory is large. The iSeries can "move" ifs objects much more efficiently than copy them. Next, a temp directory is created for the ini directory (MKDIR) and the ini file is "moved" to the temp directory just created (MOV command). Moved for consistency with the printqueue more than anything. Next, the actual save of the system library (i.e. - E812SYS) and the ifs directory (i.e. - E812SYS) is executed (SAVLIB command). Next each of the save files is copied into the IFS jde_home\targets\\ using the copy stream file command (CPYTOSTMF command). Now the backup of the existing system is complete.
Second in processing is replacing the old with the new. First we extract the software components contents into a temp directory. The contents that are important to Server Manager are scf-manifest.xml, system(SAVF) and krnlspec(SAVF). Next, the system library (E812SYS) is deleted (DLTLIB command). An important note here is that if any jobs have a lock on this library at this time, the DLTLIB will fail and the change will throw an Exception. So, make sure no locks are present before beginning this process. Next, a save file for the system library is created in a job QTEMP library (CRTSAVF command). Next, the extracted save file is copied into the newly created save file for the system library using copy from stream file (CPYFROMSTMF commands). Now we have the new qsys system save file that can be restored. Next, the system libarary is restored (RSTLIB command). A note here is that the library the system is restored into is the same name as the original library. Next, the creation of a subsystem (SBSD) is executed (CRTOWSBS command). Next, changing the data area BUILD_VER (DTAARA) in the system library is necessary (CHGDTAARA command). This matches the restored library name with the data area information. Next, delete the IFS E812sys directory (Java IFS file system utilities) . Next, a save file for the system folder is created in a job QTEMP library (CRTSAVF command). Next, the extracted save file is copied into the newly created save file for the system folder using copy from stream file (CPYFROMSTMF commands). Now we have the new ifs folder save file that can be restored. Next, the ifs folder (i.e. - E812SYS) folder is created. Next, the ifs folder changes owner to ONEWORLD (CHGOWN command). Next, the ifs folder is restored (RST command). Next, is the move of the printqueue and ini folders and contents. First make a directory for printqueue (MKDIR command). Move the contents into the new directory from the backup location (MOV command). Then change ownership to ONEWORLD (CHGOWN command). Second make a directory for ini (MKDIR command). Move the contents from the backup directory into the new directory, should just be jde.ini (MOV command). Then change ownership to ONEWORLD (CHGOWN command). Next, get the version value from the PTF.LOG file and set it for the instance. This is so we know the version of EnterpriseOne based on the value in this file.
So, if all the above steps complete successfully, we now have a new version of the EnterpriseOne Enterprise Server ready to run. Otherwise, at any point in the process that fails the backup system will be restored based on where in the process the failure occurs. Hence, there is as much level of commitment control going on as possible. Basically, the IFS and QSYS steps are tracked and then if all these are past then a full rollback should be attempted. I will not go through all possble scenarios with the rollback, but you will see the commands run to execute the rollback the same as the above steps.
Processing the iSeries EnterpriseOne Enterprise Server changeComponent is quite different than any other platforms Enterprise server, but the model is the same. Meaning that Server Manager will use common models for install, change and uninstall throughout the processing of all the EnterpriseOne server types.
Processing that occurs when you choose to register a pre-8.97 version of the Enterprise Server with Server Manager and then go to upgrade your server tools code to 8.97, Server Manager will automatically save the pre-8.97 version into a software component. That software component could subsequently be used to change from the current version back to the pre-8.97 version. So, Server Manager will allow you to roll back to that version that was registered when you first installed and setup your management domain. This is not specific functionality for the iSeries, but is good information to note as we understand what processing is going on in Server Manager. (NOTE: All work for any instance is done inside the agents jde_home\targets\"instancename" directory. Remember this note throughout the processing of the change component.)
Now, what happens during the processing to change out an iSeries EnterpriseOne Enterprise Server's tools code? First thing to note is that logging is fairly detailed as to the actual commands that are issued to the iSeries. Additionally, the return messages, even when successful, are logged to the agents log file. This is good for all troubleshooting. Second thing to note, as with any platform, the processing of changing out a tools release does a backup of the release being changed out as a first step into a temporary folder. This is for rollback purposes. If a step in the process of changing out the tools release fails a rollback will be executed. Now that you have these notes in the back of your mind, we can talk about the steps in the process.
First in processing is the backup itself. A save file is created for both the QSYS and IFS file system related to the EnterpriseOne code base, CRTSAVF commands. Next, if a printqueue directory exists in the IFS, a temp directory is created (MKDIR command) and the contents "moved" to the temp dir (MOV command). The reason for the quotes around the moved is based on the fact that the contents are not copied as this operation could take very long if the printqueue directory is large. The iSeries can "move" ifs objects much more efficiently than copy them. Next, a temp directory is created for the ini directory (MKDIR) and the ini file is "moved" to the temp directory just created (MOV command). Moved for consistency with the printqueue more than anything. Next, the actual save of the system library (i.e. - E812SYS) and the ifs directory (i.e. - E812SYS) is executed (SAVLIB command). Next each of the save files is copied into the IFS jde_home\targets\
Second in processing is replacing the old with the new. First we extract the software components contents into a temp directory. The contents that are important to Server Manager are scf-manifest.xml, system(SAVF) and krnlspec(SAVF). Next, the system library (E812SYS) is deleted (DLTLIB command). An important note here is that if any jobs have a lock on this library at this time, the DLTLIB will fail and the change will throw an Exception. So, make sure no locks are present before beginning this process. Next, a save file for the system library is created in a job QTEMP library (CRTSAVF command). Next, the extracted save file is copied into the newly created save file for the system library using copy from stream file (CPYFROMSTMF commands). Now we have the new qsys system save file that can be restored. Next, the system libarary is restored (RSTLIB command). A note here is that the library the system is restored into is the same name as the original library. Next, the creation of a subsystem (SBSD) is executed (CRTOWSBS command). Next, changing the data area BUILD_VER (DTAARA) in the system library is necessary (CHGDTAARA command). This matches the restored library name with the data area information. Next, delete the IFS E812sys directory (Java IFS file system utilities) . Next, a save file for the system folder is created in a job QTEMP library (CRTSAVF command). Next, the extracted save file is copied into the newly created save file for the system folder using copy from stream file (CPYFROMSTMF commands). Now we have the new ifs folder save file that can be restored. Next, the ifs folder (i.e. - E812SYS) folder is created. Next, the ifs folder changes owner to ONEWORLD (CHGOWN command). Next, the ifs folder is restored (RST command). Next, is the move of the printqueue and ini folders and contents. First make a directory for printqueue (MKDIR command). Move the contents into the new directory from the backup location (MOV command). Then change ownership to ONEWORLD (CHGOWN command). Second make a directory for ini (MKDIR command). Move the contents from the backup directory into the new directory, should just be jde.ini (MOV command). Then change ownership to ONEWORLD (CHGOWN command). Next, get the version value from the PTF.LOG file and set it for the instance. This is so we know the version of EnterpriseOne based on the value in this file.
So, if all the above steps complete successfully, we now have a new version of the EnterpriseOne Enterprise Server ready to run. Otherwise, at any point in the process that fails the backup system will be restored based on where in the process the failure occurs. Hence, there is as much level of commitment control going on as possible. Basically, the IFS and QSYS steps are tracked and then if all these are past then a full rollback should be attempted. I will not go through all possble scenarios with the rollback, but you will see the commands run to execute the rollback the same as the above steps.
Processing the iSeries EnterpriseOne Enterprise Server changeComponent is quite different than any other platforms Enterprise server, but the model is the same. Meaning that Server Manager will use common models for install, change and uninstall throughout the processing of all the EnterpriseOne server types.
Thursday, January 31, 2008
SM Automatically Shutsdown on Signout
Hi All
We have recently taken a few instances where the Server Manager process stops when you sign in and out of the Server Manager machine, or you connect remotely to the Server Manager machine and then disconnect from it. This issue has been SARd under 8688882.
SOLUTION:
This is actually a known Oracle issue with OC4J as documented in Metalink document 245609.1. For Server Manager, there are two methods to correct this issue. The first method is recommended since it does not involve editing the registry manually and it also ensures that the Server Manager install script gets modified so if it is rerun in the future, it will automatically add the service correctly. Method 2, however, is likely the quickest work around.
Method 1:
1) Make the following change in the installManagementConsoleService.bat which is located in your JDE_HOME\bin directory of the Server Manager machine:
from:"--StartParams=-Xmx512m;-Djde.home=%JDE_HOME%;-jar;oc4j.jar"
to:"--StartParams=-Xmx512m;-Xrs;-Djde.home=%JDE_HOME%;-jar;oc4j.jar"
Note the addition of -Xrs. This change REQUIRES that -Xrs come just after -Xmx512m.
2) Ensure that the Server Manager service is currently stopped.
3) Open a command prompt, and go to your JDE_HOME\bin directory.
Run:uninstallManagementConsoleService.bat
4) After the service uninstalls successfully,
Run:installManagementConsoleService.bat PASSWORD
where PASSWORD is your original jde_admin password from the server manager installation.
5) Start the service. It should now remain running after you log out.
Method 2:
1) Open the registry editor
2) Locate the following registry key:HKEY_LOCAL_MACHINE\SOFTWARE\Apache Software Foundation\Procrun 2.0\SCFMngmtConsole1\Parameters\Start
where SCFMngmtConsole1 is the last part of the display name of the service
Set the "Params" value to:
-Xmx512m
-Xrs
-Djde.home=C:\jde_home
-jaroc4j.jar
(note the addition of -Xrs)
This change REQUIRES that -Xrs come just after -Xmx512m.
3) Start the service. It should now remain running after you log out.
We have recently taken a few instances where the Server Manager process stops when you sign in and out of the Server Manager machine, or you connect remotely to the Server Manager machine and then disconnect from it. This issue has been SARd under 8688882.
SOLUTION:
This is actually a known Oracle issue with OC4J as documented in Metalink document 245609.1. For Server Manager, there are two methods to correct this issue. The first method is recommended since it does not involve editing the registry manually and it also ensures that the Server Manager install script gets modified so if it is rerun in the future, it will automatically add the service correctly. Method 2, however, is likely the quickest work around.
Method 1:
1) Make the following change in the installManagementConsoleService.bat which is located in your JDE_HOME\bin directory of the Server Manager machine:
from:"--StartParams=-Xmx512m;-Djde.home=%JDE_HOME%;-jar;oc4j.jar"
to:"--StartParams=-Xmx512m;-Xrs;-Djde.home=%JDE_HOME%;-jar;oc4j.jar"
Note the addition of -Xrs. This change REQUIRES that -Xrs come just after -Xmx512m.
2) Ensure that the Server Manager service is currently stopped.
3) Open a command prompt, and go to your JDE_HOME\bin directory.
Run:uninstallManagementConsoleService.bat
4) After the service uninstalls successfully,
Run:installManagementConsoleService.bat PASSWORD
where PASSWORD is your original jde_admin password from the server manager installation.
5) Start the service. It should now remain running after you log out.
Method 2:
1) Open the registry editor
2) Locate the following registry key:HKEY_LOCAL_MACHINE\SOFTWARE\Apache Software Foundation\Procrun 2.0\SCFMngmtConsole1\Parameters\Start
where SCFMngmtConsole1 is the last part of the display name of the service
Set the "Params" value to:
-Xmx512m
-Xrs
-Djde.home=C:\jde_home
-jaroc4j.jar
(note the addition of -Xrs)
This change REQUIRES that -Xrs come just after -Xmx512m.
3) Start the service. It should now remain running after you log out.
Friday, January 25, 2008
Server Manager Logging - Part 3
Managed Home Agent Logging
OK, now on to the good stuff. The managed home agent, herein referred to as simply the agent, is responsible for a majority work performed by Server Manager (SM). This includes:
- registering/installing E1 managed instances
- registering and managing IBM WebSphere and Oracle Application Server
- managing the configuration file(s) for E1 managed instances
- starting/stopping E1 servers and the J2EE servers
- performing tools release upgrades/downgrades for E1 servers
- discovering and sending log files to the management console for viewing
You get the idea. The agent does most of the work. If anything goes wrong during one of these items, or your just an inquisitive person, the place you'll want to look to see what's going on are the agent's log files.
Logging Overview
Before we dig too far into the log files themselves let's get an understanding of the logging system used. Server Manager makes use of the standard java logging framework. This framework is different than the logging engine used by most other E1 software components which are based on the jdelog.properties configuration file.
The java logging framework exposes, and we make use of, seven levels of logging as outlined below in descending order
The division between SEVERE and WARNING is a little muddy. You may see messages that appear as SEVERE that really should be qualified as WARNING based on the above descriptions, and less frequently there may be a WARNING message that should have been classified as SEVERE.
Level Description SEVERE A critical error has occurred from the perspective of the agent. Critical errors are non-recoverable errors and require immediate attention. An example would be a critical problem when initializing the agent that would prevent it from starting or functioning properly. WARNING Denotes an abnormal or unexpected result occurred that is recoverable, from the perspective of the agent. An example would include a failure while changing the tools release of an E1 server. It is a significant problem, however, the agent will recover so is considered a warning. INFO Denotes informative messages providing contextual information as to what the agent is doing. An E1 server that is started using SM would have a log message indicating so at the INFO level. CONFIG Not commonly used in SM, a message at the configuration level is simply a means for logging information particular to that installation, such as the platform of the server. FINE A lower level message still intended as human readable that provides insight into what the agent is doing. This can be thought of as a standard "debug" message. FINER An even lower level trace of debug message. Messages are classified at this level rather than FINE if they are very frequently occurring and either less likely to be of interest. FINEST The lowest level of logging. These messages may be very frequent, verbose, or cryptic and may only be meaningful to Oracle development.
The default logging level for the agent is FINE, which should be fine for nearly all troubleshooting needs. The agent is much less verbose than some of the other E1 products. Keeping the logging level at FINE, FINER, or even FINEST will have negligible impact on performance. There just aren't that many messages emitted to cause a problem.
Agent Log Location
The agent log files are located in the directory <agent_install>logs, where <agent_install> refers to the install location supplied during the managed home installer. The log files are named e1agent_#.log. The logging mechanism will automatically split log files into approximately 10 MB chunks, and up ten log files will be retained. That means the most amount of disk space needed for the agent logs is 100 MB. The '#' in the log name is the index of the log file with zero being the most recent and nine being the oldest. When the current log file, e1agent_0.log, reaches approximately 10 MB the last chunk (e1agent_9.log) will be deleted and all the log files will be renamed with the index incremented. The e1agent_0.log is now e1agent_1.log and a new e1agent_0.log file will be created. Another file, 'e1agent_0.log.lck' may also be present. This file is created by Java as a lock file and may be ignored.
The log files for the agent may be viewed directly through the management console directly. From the management dashboard (start page) select the managed home of interest.
At the bottom of the page for the selected managed home is a log files section.
Selecting a log file will transfer the log from the remote machine to the management console so it may be viewed using the integrated log viewer, as shown below.
Each log message consists of two or more lines. The first line contains the timestamp, originating java class, and the name of the current method from which the message was written. The second line (or multiple lines) contains the actual log message.
Changing Logging Levels
The default level for agent logging is FINE, which is appropriate for most occasions. It may be desirable to change the level to FINER or FINEST to troubleshoot an issue, or move it to a higher level, such as INFO, for some reason. If the agent is running and connected to a management console you may change the level directly from the console. Located on the left hand side of the page for the management agent is a section entitled 'Managed Home Details'. In there is a dropdown for 'Agent Log Level'. The current value will be shown.
To change the level simply select the desired log level from the drop down list. The change will immediately take effect. Changing the log level through this dropdown is not permanent; when the agent is restarted it will resume logging at the FINE level.
If you wish to permanently change the logging level, or you will to set the level to something other than FINER during agent startup, you may add the following line to the <agent_install>/config/agent.properties file.
log.level=FINER
You may use any of the log levels mentioned in the table above, or OFF to prevent all logging.
I do not recommend permanently keeping the log level at OFF, SEVERE, WARNING, CONFIG, or INFO. You never know when you may need the additional information provided by the FINE level. The agent will automatically remove old log files so there should be no concern of log files filling a disk.
Embedded Agent Logging
The E1 enterprise server and E1 HTML server all contain variants of the management agent. In the embedded form the logging messages generated by the management agent are not typically logged. It may be desired to enable the same logging for the embedded management agents. To do so simply add the following line to the <agent_install>/config/agent.properties file.
log.embedded.instances=true
After adding this line and restarting the E1 managed instance series of log files will appear in the <agent_install>/logs directory. The filename of the log files will be the instance name of the E1 server.
It is very rare to need to enable this logging; that said it may be useful to troubleshoot why the embedded agents may not be communicating with the management console.
This form of logging is not available to an E1 web server if it is deployed to a federated (e.g. network deployment) node in IBM WebSphere.
Final Thoughts
Stack Traces: some log messages may contain stack traces. This information is useful for identifying the source of the message. A stack trace is not always indicative of a problem. The level of the message is much more indicative. Stack traces for INFO, FINE, FINER, or FINEST messages are included to provide more information and is not an error.
Server Manager Logging - Part 2
Part 2 in this series about the Server Manager logs will be a short one and is all about the log files created when installing a management agent. I've already posted an article Troubleshooting Agent Communications that covers in detail the troubleshooting process for a successfully installed agent that refuses to communicate and appear in the management console. This article will simply look at the installation logs created by the management agent installer.
In a future post I'll get to the good stuff -- the log files created by a running management agent. That will be where you'll wan to look for troubleshooting most issues in Server Manager.
The Management Agent Installer
During the running of the installation program the log file created, named smha_is_install.log, is located in platform specific temporary directory. The
InstallShield installer creates a temporary directory named something like 'ismp01' underneath the system temporary directory. On UNIX based platforms and the iSeries this temporary location is usually /tmp and on Windows the temporary directory is identified by the %TMP% environment variable, usually Documents and SettingsusernameLocal Settingstemp.
After the installation is complete the smha_is_install.log is copied to the installation location supplied during the installation wizard.
Compared to the installer for the management console discussed in part one of this series the management agent installer is very trivial. The installer is responsible for performing the following:
- Creating the managed home directory structure
- Installing the Java based managed home agent and any configuration it needs
- Creating the Windows service used to start and stop the agent [Windows only]
- Creating the startup and shutdown scripts
- Installing the Java JDK used by the managed home agent
- Installing the Oracle Configuration Manager agent [non-iSeries platforms]
The installer itself is actually quite dumb. The installer program does not actually contain any of the components it will install; instead it obtains the components to install directly from the management console. This is the reason the management console must be running at the time the management agent is installed.
So let's take a look at the smha_is_install.log from a Linux installation to see what is going on. After prompting for the management console name and HTTP port the installer will retrieve a list of items that need to be downloaded/installed. If the supplied server name or port are wrong, or the management console is not running, you will see an error here and the user will be re-prompted for valid credentials. Upon successful connection to the console you will see something like:
The list of components to be downloaded are saved as a text file to <agent_install>/List.txt. The contents of the file will vary slightly for each platform. The next step is to download and extract the components contained within.(Jan 25, 2008 12:53:44 PM), Install, com.ibm.wizard.platform.linux.LinuxProductServiceImpl, msg1, installing Get Scf File List (getScfList)(Jan 25, 2008 12:53:44 PM), Install, com.oracle.e1.install.scfha.ISMP.product.GetScfFileList, dbg, Connection opened: http://denlcmwn5.mlab.jdedwards.com:7000/manage/agents/Linux/i386/2.6.9-42.ELsmp?serviceId=null(Jan 25, 2008 12:53:44 PM), Install, com.oracle.e1.install.scfha.ISMP.product.GetScfFileList, dbg, Writing http://denlcmwn5.mlab.jdedwards.com:7000/manage/agents/Linux/i386/2.6.9-42.ELsmp?serviceId=null to /u02/jdedwards/agentdemo/List.txt as text(Jan 25, 2008 12:53:44 PM), Install, com.ibm.wizard.platform.linux.LinuxProductServiceImpl, dbg.install, JVM memory after installing Get Scf File List (getScfList): free=5134496 total=17588224
The remaining steps performed by the installer are related to setting some permissions on the installed JDK (UNIX based platforms) and to start the management console. On windows you would see an additional step to invoke the installManagementAgent.bat script to create the Windows service for the managed home agent.(Jan 25, 2008 12:53:44 PM), Install, com.installshield.scfha.event.InstallScript, dbg, Entering Function : Expand Remote Jars(Jan 25, 2008 12:53:44 PM), Install, com.installshield.scfha.event.InstallScript, dbg, Expand Remote Jars : File List to download:/u02/jdedwards/agentdemo/List.txt(Jan 25, 2008 12:53:44 PM), Install, com.installshield.scfha.event.InstallScript, dbg, Expand Remote Jars : Destination :/u02/jdedwards/agentdemo(Jan 25, 2008 12:53:44 PM), Install, com.installshield.scfha.event.InstallScript, dbg, Extracting remote jar 'http://denlcmwn5.mlab.jdedwards.com:7000/manage/scfagent/scfagent.jar?serviceId=null' to '/u02/jdedwards/agentdemo'(Jan 25, 2008 12:53:45 PM), Install, com.installshield.scfha.event.InstallScript, dbg, Copying remote file : http://denlcmwn5.mlab.jdedwards.com:7000/manage/agent/config/agent.properties?serviceId=null(Jan 25, 2008 12:53:45 PM), Install, com.installshield.scfha.event.InstallScript, dbg, Header Field: attachment; filename=config/agent.properties(Jan 25, 2008 12:53:46 PM), Install, com.installshield.scfha.event.InstallScript, dbg, Extracting remote jar 'http://denlcmwn5.mlab.jdedwards.com:7000/manage/agent/ccr/ccr-Production-10.2.5.0.0-Linux.zip?serviceId=null' to '/u02/jdedwards/agentdemo'(Jan 25, 2008 12:53:48 PM), Install, com.installshield.scfha.event.InstallScript, dbg, Extracting remote jar 'http://denlcmwn5.mlab.jdedwards.com:7000/manage/agent/jdks/linux_jdk.jar?serviceId=null' to '/u02/jdedwards/agentdemo'(Jan 25, 2008 12:54:01 PM), Install, com.installshield.scfha.event.InstallScript, dbg, Copying remote file : http://denlcmwn5.mlab.jdedwards.com:7000/manage/agent/agent-misc/linux/bin/startAgent?serviceId=null(Jan 25, 2008 12:54:01 PM), Install, com.installshield.scfha.event.InstallScript, dbg, Header Field: attachment; filename=bin/startAgent(Jan 25, 2008 12:54:01 PM), Install, com.installshield.scfha.event.InstallScript, dbg, Copying remote file : http://denlcmwn5.mlab.jdedwards.com:7000/manage/agent/agent-misc/linux/bin/stopAgent?serviceId=null(Jan 25, 2008 12:54:01 PM), Install, com.installshield.scfha.event.InstallScript, dbg, Header Field: attachment; filename=bin/stopAgent(Jan 25, 2008 12:54:01 PM), Install, com.installshield.scfha.event.InstallScript, dbg, Copying remote file : http://denlcmwn5.mlab.jdedwards.com:7000/manage/agent/agent-misc/linux/bin/restartAgent?serviceId=null(Jan 25, 2008 12:54:01 PM), Install, com.installshield.scfha.event.InstallScript, dbg, Header Field: attachment; filename=bin/restartAgent(Jan 25, 2008 12:54:01 PM), Install, com.installshield.scfha.event.InstallScript, dbg, Copying remote file : http://denlcmwn5.mlab.jdedwards.com:7000/manage/agent/agent-misc/rte.jacl?serviceId=null(Jan 25, 2008 12:54:02 PM), Install, com.installshield.scfha.event.InstallScript, dbg, Header Field: attachment; filename=bin/rte.jacl
The final step is to remove the List.txt file downloaded earlier and start up the managed home agent. Pretty simple, eh?
Thursday, January 24, 2008
Server Manager Logging - Part 1
Server Manager has log files. A lot of 'em in fact. There, I said it.
So where do you go when something in SM isn't working just right? We spent considerable time attempting to log often and use meaningful messages within SM. That said it is a distributed architecture with components contained on the console machine, agents installed on remote machines, and more agents built into the E1 products themselves. This post will help you identify which logs you should be looking at, where they are, how to configure them, and what to look for. For the sake of being able to finish the article eventually I'll just focus on the SM logs.
So where do you go when something in SM isn't working just right? We spent considerable time attempting to log often and use meaningful messages within SM. That said it is a distributed architecture with components contained on the console machine, agents installed on remote machines, and more agents built into the E1 products themselves. This post will help you identify which logs you should be looking at, where they are, how to configure them, and what to look for. For the sake of being able to finish the article eventually I'll just focus on the SM logs.
Throughout this post I'll be referring to some common paths.
<console_install> - Refers to the location supplied during the installation of the management console.
Installation Logs - Management Console
We begin with looking at the log files generated during the installation of the management console. These can be helpful if there are problems installing the management console application. Once the console is successfully installed and you are able to sign-in to the web application these logs are no longer relevant for any purpose and may be removed, if desired.
The management console is installed using an InstallShield based installer. This installer is responsible for creating the <console_install> directory, extracting all the component files that make up SM, configuring the integrated J2EE server, and deploying the management console web application.
The primary log created by the installer is located in <console_install>logssmmc_install.log. This is the first place you should look if you are having problems installing the management console. This log will probably go down as the most cryptic of the log files to analyze but somewhere in there is good information. One thing to note is if something goes wrong during the installation the installer will roll back the changes already made. This may result in error messages and other possibly confusing messages in the log file. In the case of a failed installation you want to find the first thing that went wrong in the log file before it started rolling back the installation.
The bulk of the messages in the smmc_install.log file will detail the extraction of files from the installer application to the <console_install> location. Near the end the installer performs the configuration of the J2EE container and deploys the management console application. The first task is to configure the administrative user for the J2EE container using the credentials provided during the installer wizard. The smmc_install.log will contain the following lines:
(Jul 31, 2007 12:40:14 PM), Install, com.installshield.wizard.platform.win32.Win32ProductServiceImpl, dbg.install, JVM memory before installing Exec Action (execJaznJar): free=9241864 total=23748608
(Jul 31, 2007 12:40:14 PM), Install, com.installshield.wizard.platform.win32.Win32ProductServiceImpl, msg1, installing Exec Action (execJaznJar)
(Jul 31, 2007 12:40:16 PM), Install, com.installshield.wizard.platform.win32.Win32ProductServiceImpl, dbg.install, JVM memory after installing Exec Action (execJaznJar): free=9833152 total=23748608
At the same time two new log files will appear in <console_install>logs: jazn_stdout.log and jazn_stderr.log. These are the stdout and stderr output from running commands contained in the jazn.jar java application delivered with the J2EE container. The stderr file should be empty. Any content in the file is indicative of an error and will cause the installation to fail. If the jazn_stderr.log file is not empty you should investigate the cause of any errors detailed within.
The next step performed is to install the Windows service used to start and stop the management console. This is performed by running the delivered script <console_install>bininstallManagementConsoleService.bat. The following lines in the smmc_install.log file indicate when this batch file is called:
(Jul 31, 2007 12:40:16 PM), Install, com.installshield.wizard.platform.win32.Win32ProductServiceImpl, dbg.install, JVM memory before installing Exec Action (execInstService): free=9809880 total=23748608(Jul 31, 2007 12:40:16 PM), Install, com.installshield.wizard.platform.win32.Win32ProductServiceImpl, msg1, installing Exec Action (execInstService)(Jul 31, 2007 12:40:17 PM), Install, com.installshield.wizard.platform.win32.Win32ProductServiceImpl, dbg.install, JVM memory after installing Exec Action (execInstService): free=9552496 total=23748608(Jul 31, 2007 12:40:17 PM), Install, com.installshield.scfmc.event.InstallScript, msg1, Beginning the onInstalledOc4jExecComp method.
The installManagementConsoleService.bat creates a log file named <console_install>logsinstallManagementConsole.log. The contents will look something like:
[2008-01-24 14:50:41] [1206 prunsrv.c] [debug] Procrun log initialized[2008-01-24 14:50:41] [496 prunsrv.c] [debug] Installing service...[2008-01-24 14:50:41] [info] Service SCFMngmtConsole3 name SCF Management Console [D:servermanagerdemosystem] - SCFMngmtConsole3[2008-01-24 14:50:42] [538 prunsrv.c] [debug] Setting service description SCF Management Console[2008-01-24 14:50:42] [info] Service SCFMngmtConsole3 installed[2008-01-24 14:50:42] [info] Procrun finished.
Now that the J2EE server has been configured with the administrative password and the Windows service created its time to start the container. The batch file <console_install>binstartManagementConsole.bat is invoked and you will see lines along the lines of:
(Jul 31, 2007 12:40:17 PM), Install, com.installshield.scfmc.event.InstallScript, msg1, Command to run is: d:servermanagerdemosystem/bin/startManagementConsole.bat(Jul 31, 2007 12:40:20 PM), Install, com.installshield.scfmc.event.InstallScript, msg1, Return code is: 0
The output of the startManagementConsole.bat is written to the log file <console_install>logsMCDeploy_stdout.log. The first two lines of the file will contain something like:
The SM Management Console [D:servermanagerdemosystem] - SCFMngmtConsole4 service is starting.The SM Management Console [D:servermanagerdemosystem] - SCFMngmtConsole4 service was started successfully.
It may take a while for the J2EE container to fully start up. To ensure the container is actually running before the management console application is deployed the installer will repeatedly run a command line program that will check if the server is actually running. You will see this in the log file:
(Jul 31, 2007 12:40:30 PM), Install, com.installshield.scfmc.event.InstallScript, msg1, Command to run is: d:servermanagerdemosystem/jdk/bin/java.exe -jar d:servermanagerdemosystem/targets/oc4j/j2ee/home/admin_client.jar deployer:oc4j:localhost:24795 oc4jadmin ******** -validateURI(Jul 31, 2007 12:41:24 PM), Install, com.installshield.scfmc.event.InstallScript, msg1, Return code is: 0
The installer will be looking for a success message emitted by the program run. If not found it will wait a bit and re-run the command, eventually giving up. The responses are appended to the MCDeploy_stdout.log. The success message looks like
URI deployer:oc4j:localhost:24795 is valid and connected
Once the J2EE container is validated as running the management console application is deployed to it. The deployment is performed through a command line program, the running of which is shown below:
(Jul 31, 2007 12:41:24 PM), Install, com.installshield.scfmc.event.InstallScript, msg1, Command to run is: d:servermanagerdemosystem/jdk/bin/java.exe -jar d:servermanagerdemosystem/targets/oc4j/j2ee/home/admin_client.jar deployer:oc4j:localhost:24795 oc4jadmin ******** -deploy -file d:servermanagerdemosystem/_stage/ManagementConsole_WAR.ear -deploymentName ManagementConsole -bindAllWebApps(Jul 31, 2007 12:42:00 PM), Install, com.installshield.scfmc.event.InstallScript, msg1, Return code is: 0(Jul 31, 2007 12:42:00 PM), Install, com.installshield.scfmc.event.InstallScript, msg1, Deployment succeeded for MC.
The output of the deployment operations are also appended to MCDeploy_stdout.log and should look something like:
07/07/31 12:41:26 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:26 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:27 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:27 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:27 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:27 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:28 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:28 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:28 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:29 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:29 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:29 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:29 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:30 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:30 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:30 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:30 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:31 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:31 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:31 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:31 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:32 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:32 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:32 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:33 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:33 Notification ==>Uploading file ManagementConsole.ear ...07/07/31 12:41:33 Notification ==>Application Deployer for ManagementConsole STARTS.07/07/31 12:41:33 Notification ==>Copy the archive to D:servermanagerdemosystemtargetsoc4jj2eehomeapplicationsManagementConsole.ear07/07/31 12:41:33 Notification ==>Initialize D:servermanagerdemosystemtargetsoc4jj2eehomeapplicationsManagementConsole.ear begins...07/07/31 12:41:33 Notification ==>Unpacking ManagementConsole.ear07/07/31 12:41:34 Notification ==>Done unpacking ManagementConsole.ear07/07/31 12:41:35 Notification ==>Unpacking ManagementConsole_WAR.war07/07/31 12:41:44 Notification ==>Done unpacking ManagementConsole_WAR.war07/07/31 12:41:44 Notification ==>Initialize D:servermanagerdemosystemtargetsoc4jj2eehomeapplicationsManagementConsole.ear ends...07/07/31 12:41:44 Notification ==>Starting application : ManagementConsole07/07/31 12:41:44 Notification ==>Initializing ClassLoader(s)07/07/31 12:41:44 Notification ==>Initializing EJB container07/07/31 12:41:44 Notification ==>Loading connector(s)07/07/31 12:41:44 Notification ==>Starting up resource adapters07/07/31 12:41:44 Notification ==>Initializing EJB sessions07/07/31 12:41:44 Notification ==>Committing ClassLoader(s)07/07/31 12:41:44 Notification ==>Initialize ManagementConsole_WAR begins...07/07/31 12:41:44 Notification ==>Initialize ManagementConsole_WAR ends...07/07/31 12:41:44 Notification ==>Started application : ManagementConsole07/07/31 12:41:44 Notification ==>Binding web application(s) to site default-web-site begins...07/07/31 12:41:44 Notification ==>Binding ManagementConsole_WAR web-module for application ManagementConsole to site default-web-site under context root /manage07/07/31 12:41:45 Notification ==>Initializing Servlet: com.jdedwards.mgmt.web.ConsoleInit for web application ManagementConsole_WAR07/07/31 12:41:58 Notification ==>Initializing Servlet: oracle.cabo.servlet.UIXServlet for web application ManagementConsole_WAR07/07/31 12:41:59 Notification ==>Binding web application(s) to site default-web-site ends...07/07/31 12:41:59 Notification ==>Application Deployer for ManagementConsole COMPLETES. Operation time: 25813 msecs
Any errors or problems that occur during the startup, running validation, and deployment will appear in the log file <console_install>logsMCDeploy_stderr.log. The installer will check if there are any errors contained within this file and if so terminate and rollback the installation, as shown below:
(Jul 31, 2007 12:42:00 PM), Install, com.installshield.wizard.platform.win32.Win32ProductServiceImpl, msg1, installing Log File Error Check (logFileErrCheck)
(Jul 31, 2007 12:42:00 PM), Install, com.oracle.e1.install.common.ISMP.product.LogFileErrorCheck, msg1, Beginning stderr log checks.
(Jul 31, 2007 12:42:00 PM), Install, com.oracle.e1.install.common.ISMP.product.LogFileErrorCheck, msg1, Checking for errors in: D:servermanagerdemosystemlogsMCDeploy_stderr.log
(Jul 31, 2007 12:42:00 PM), Install, com.oracle.e1.install.common.ISMP.product.LogFileErrorCheck, msg1, No errors in the stderr logs, continuing the installation.
In case you forgot the information you entered, such as the HTTP port to use for the management console, a readme.txt file is created in the <console_install> directory. After creating the readme.txt file the installation is essentially complete.
Next Time
Now that we have the management console installed it's time to start installing some management agents, and looking at their log files.
Wednesday, January 16, 2008
Some Kind Words on 8.97
I'm always willing to publish kind words about Server Manager. Check out this post from Andrews Consulting Group:
Oracle Delivers Cool Tools for JD Edwards EnterpriseOne
Oracle Delivers Cool Tools for JD Edwards EnterpriseOne
Tuesday, January 8, 2008
Monitors in Server Manager
Monitors
Most of my posts thus far have been about installation, troubleshooting, and other server manager basics. Today begins a series of posts outlining the new or enhanced capabilities provided by SM.
Monitors are the mechanism by which administrators can be alerted through e-mail when an event of interest occurs. Much of this functionality is a direct carryover from that provided by the SAW SMC infrastructure in previous tools releases with some significant enhancements to boot.
As you may be aware by now Server Manager is a complete replacement for SAW and SMC. Among the other benefits, such as deployment and configuration management, we wanted to enhance and make easier to use the functionality provided by the SAW application.
While evaluating the SMC monitoring capabilities we identified the need to improve it in the following ways:
* Simplify the setup required to monitor servers and configure events of interest
* Enhance the monitored events to include some key items of interest, such as a user being unable to login to the E1 HTML Server
* Permit configuration of the hours in which alert e-mails should be sent for sites that make use of multiple administrators that are responsible for particular times of the week
* Maintain a history of past events and record the e-mail messages that were sent
We also changed the mechanism by which the events of interest are obtained. Beginning with 8.97 our server products contain an embedded variant of the management agent that provides server manager with the runtime information about the servers. Using this mechanism to obtain events provided two primary benefits: many of the events are reported to SM immediately upon occuring and events can be obtained from clustered or multi-JVM configurations for our web based products.
Currently monitoring is supported for our enterprise server and HTML server products only.
To get started select the monitors link from the quicklinks section. Note you must be signed into the management console as the jde_admin user or another user that as been granted the 'monitorConfig' permission to make changes to the monitoring configuration.
SMTP Configuration
The first step is to configure the SMTP mail server that will be used by server manager to send emails. Simply supply the mail server name, TCP/IP port to use, and sender email to use as the 'from' address. Some SMTP servers may require the sender email be from the same domain the mail server is configured to use. Note: SMTP servers that require authentication to send emails is not currently supported.
After making any changes you may supply an email address to test the settings. Server Manager will send an email to the supplied address to ensure the mail server configuration is correct.
Getting Started
The next step is to create a new monitor. You may have as many monitors as you wish. For example you may wish to create multiple monitors that listen for different events and each have different email recipients. Enter a name for the new monitor and select the 'Create' button. You will be redirected to a page used to configure the newly created monitor.
The first option in the general settings controls how often the monitor should poll for events. Some events will be detected immediately; when they occur a notification is sent to the management console and then to each running monitor. If this event is enabled for a particular monitor an email will be sent immediately. Other events are polled on a periodic basis. For example checking the free disk space on an enterprise server occurs on this period poll. You can change the frequency in which the monitor will check for these events.
Checking for monitored events is a low impact activity. That said if you have a large number of monitors it may be advisable to increase this interval from the default of 30 seconds.
Secondly you may configure whether this monitor should be automatically started when the management console application is started. Regardless of this selection an authorized user may start and stop monitors at any time using the previous page.
Instance Selection
The next step involves selecting the managed instances that this monitor should observe. Simply move the desired instances from the available options list to the selected options. Note that any changes made here on a running monitor will take effect immediately; the monitor need not be restarted.
Event Selection
Now that we have selected which managed instances we should monitor we now need to select which events we wish to observe. You do so by simply selecting the events of interest in the next section of the page. Each event has a help box next to it describing what the event is and when it may occur.
Some events may have threshold values that allow you to define a limit that, once reached, will trigger an email notification. The example below shows the limits for simultaneous users.
Once a threshold limit is reached on an enabled event a email notification will be sent. Notifications will not be resent unless the threshold goes higher. Consider the simultaneous users event. If we set the threshold to 50 we would receive a notification once 50 users are on at the same time. If two users sign off and two new users sign back on we are back at 50 simultaneous users. An email will not be sent; an email for 50 users has already been sent. If another user signs on, so we are at 51 sessions, and email will be sent; we have gone higher then the highest threshold reached.
I won't go into what all the events are in this post; they are documented with online help within the application.
Notification Hours
For a particular monitor you may specify which hours in the day and which days of the week email notifications should be sent. This may be helpful for those who administer in shifts. Those interested in events on weekdays may be different then those interested in weekend events, for example.
When you create a new monitor the default will be to enable notifications for all hours of all days. You can change this by modifying the times for each day using 24 hour notation. To disable events for an entire day simply set the start time and end time to both be 00:00.
The management console will use the clock and time zone information provided by the JVM on which it runs. That is the times should be considered to be the times as known to the management console machine.
Email Recipients
Finally we specify the email recipients that should receive notifications. You may add as many recipients as you wish. Any changes made to this list will take effect immediately; you need not restart the monitor.
Emails are sent individually to each recipient defined for a monitor using the from address configured previously. The subject and content of the email will contain details of the event. The mail format is plain text and is suitable for email, pager, and SMS mailboxes.
If an email could not be sent for any reason the failure will be recorded in the monitor's history, as discussed below.
Monitor History
Server Manager maintains a history for each monitor. Each start of a monitor will be listed in the monitor history.
You may view the history of a particular monitor to see all the events that occurred and the emails sent by clicking the appropriate icon in the grid row.
Each event that occurred will be listed along with the same type if information that was contained in the email sent. A grid will contain a listing for each email recipient of the monitor showing the successful sending of the email, an email that wasn't sent because it was outside the notification hours configured, or a email that failed to send for some reason such as an invalid recipient.
You may delete the monitor history if you no longer wish to view it. You may not delete the history for an actively running monitor.
Cloning Monitors
We have made it easy to clone an existing monitor. Simply select the corresponding icon in the 'Create Duplicate' column in the list of available monitors.
All the settings, selected managed instances, events, notification hours, and email recipients from the selected monitor will be copied to a new monitor definition. This makes setting up monitors for shifts much easier; the events and other setup need not be configured multiple times.
Summary
Hopefully you see that setting up and using monitors in Server Manager is much easier than previous solutions and the added events make administering your E1 servers much easier. Dig in, play with monitors, and enjoy.
UPDATE: I think the issue with missing images has been resolved.
Wednesday, January 2, 2008
Using NFS partitions on AIX
Unless you are running an E1 enterprise server on an NFS partition on the AIX platform, you can probably skip this posting.
So what does this have to do with E1 and Server Manager? The second step of performing a tools release change involves deleting the existing tools release. The caching of the shared libraries, and thus the presence of the .nfs#### files in the $EVRHOME/system/lib directories will prevent the removal of the system directory. This will cause the tools release change to fail, and the previous tools release will be restored. Even root cannot delete this .nfs files directly.
Still here? Ok. This post outlines a potential problem with changing the tools release of an enterprise server when it is running on a NFS partition on AIX. It pertains to that combination only.
The AIX operating system has a feature that keeps shared libraries in memory even when the program that loads them terminates. Subsequent loads of that or any other program using the same library would be faster because the library is already in memory.
This behavior can cause some problems when the shared library is located on an NFS partition. Consider the case when Server Manager is performing a tools change for an enterprise server. The management agent will 1) stop the enterprise server, 2) delete the existing tools release, 3) extract and replace it with the new tools release.
So where's the problem? After stopping the enterprise server the E1 shared libraries may be cached by AIX even though no active processes are using them. AIX maintains open file handle to the shared library. On UNIX based platforms you are able to delete a file that is open by another process; although it will immediately disappear from the file system directory listings it will not actually be removed once the last handle to that file is closed. This behavior is done within the filesystem implementation.
The remote nature of the NFS file system requires a special implementation. When an open file is deleted on a NFS partition it will appear as a .nfs##### file in the same directory, where #### refers to a number randomly assigned. This file cannot be removed directly; it will disappear as soon as the last process holding the originally deleted file closes that handle.
So what does this have to do with E1 and Server Manager? The second step of performing a tools release change involves deleting the existing tools release. The caching of the shared libraries, and thus the presence of the .nfs#### files in the $EVRHOME/system/lib directories will prevent the removal of the system directory. This will cause the tools release change to fail, and the previous tools release will be restored. Even root cannot delete this .nfs files directly.
What can be done is to stop the enterprise server using Server Manager then sign on as root and run the command 'slibclean'. This will instruct AIX to unload/uncache any shared libraries that are no longer being used by an active process. You may then change the tools release using Server Manager without any issue.
Subscribe to:
Posts (Atom)
Posts
