Peoplesoft - How to Increase or Decrease Oracle Tuxedo IPC Resources on Windows

For: Microsoft Windows x64 (64-bit)

Below is an example of common settings that are increased to support multiple application server domains on one Windows machine.

Go to Control Panel - Oracle Tuxedo 10gR3 with VS2008 Administration - IPC Resources tab

Uncheck Use Default IPC Settings

Click next to where it says IPC Resources it is a little box. When cursor is over the box it will show, New (Insert)

Type in the word Custom and hit enter

Now everything on the right turned from greyed out to white

Change the values you need as per requirements..

Hit Apply

The ORACLE ProcMGR V10gR3 with VS2008 Service will then need to be restarted. Make sure you shut down all domains and schedulers that are maintained by Tuxedo before doing this.

Peoplesoft Reports Posting Issue.Tuning Tuxedo IPC Resources Message queues

I had to tune the tuxedo message queue and message segment parameters to fix ths issue and to avoid the issue in future.

Issue: http://psoftdba.blogspot.sg/2013/11/peoplesoft-reports-struck-in-posting.html

PSPRCSRV.4360 (0) [11/26/13 23:59:31](1) (NET.333): The service PostReport could not be sent because of a blocking condition within Tuxedo on the client.  
PSPRCSRV.4360 (0) [11/26/13 23:59:31](1) (NET.333): The service PostReport could not be sent because of a blocking condition within Tuxedo on the client.  
PSPRCSRV.4360 (0) [11/26/13 23:59:31](1) (NET.333): The service PostReport could not be sent because of a blocking condition within Tuxedo on the client.  

Messages of up to 100KB are typical in PeopleTools. Where possible, I suggest increasing the maximum
message size to at least 128KB and possibly as far as 256KB if sufficient memory is available. There are
always a few messages that don’t fit in the queues. Ad hoc queries that are run online in the PIA and that
generate large result sets can easily exceed this limit.

I have tuned the below parameters to solve the issue.

Maximum Message Queue Size to be increased to 196608 from 65536

Size of Message Segment to be increased to 1024 from 68

Please note that the memory needs to be considered when you tune this.

Happy Troubleshooting :-)

Peoplesoft Reports Struck in Posting Status - The service PostReport could not be sent because of a blocking condition within Tuxedo on the client

There are multiple issues/scenario for Posting issue. Below post is specific to Error below
===============================================================
PSPRCSRV.4360 (0) [11/26/13 23:59:31](1) (NET.333): The service PostReport could not be sent because of a blocking condition within Tuxedo on the client.  
PSPRCSRV.4360 (0) [11/26/13 23:59:31](1) (NET.333): The service PostReport could not be sent because of a blocking condition within Tuxedo on the client.  
PSPRCSRV.4360 (0) [11/26/13 23:59:31](1) (NET.333): The service PostReport could not be sent because of a blocking condition within Tuxedo on the client.  
PSPRCSRV.4360 (0) [11/26/13 23:59:31](1) (NET.333): The service PostReport could not be sent because of a blocking condition within Tuxedo on the client.  
PSPRCSRV.4360 (0) [11/26/13 23:59:31](1) (NET.333): The service PostReport could not be sent because of a blocking condition within Tuxedo on the client.  
PSPRCSRV.4360 (0) [11/26/13 23:59:31](1) (NET.333): The service PostReport could not be sent because of a blocking condition within Tuxedo on the client.  
PSPRCSRV.4360 (0) [11/26/13 23:59:31](1) (NET.333): The service PostReport could not be sent because of a blocking condition within Tuxedo on the client.  
PSPRCSRV.4360 (0) [11/26/13 23:59:31](1) (NET.333): The service PostReport could not be sent because of a blocking condition within Tuxedo on the client.  
PSPRCSRV.4360 (0) [11/26/13 23:59:31](1) (NET.333): The service PostReport could not be sent because of a blocking condition within Tuxedo on the client. 



Reference from the Oracle Support only but i have verified it and worked well for my issue: 

Case 1:

To remove a Tuxedo POSTREPORT blocking condition (without having to recreate the Process Scheduler);

1. Bring down the affected Process Scheduler(s).

2. Identify the report that is causing the blocking condition by running the following queries:

A) SELECT * from PS_CDM_TRANSFER where TRANSFERINSTANCE = 'lowest transferinstance value'.
B) SELECT * from PS_CDM_LIST where TRANSFERINSTANCE = 'value from (A)' [this will give you the process instance number or PRCSINSTANCE].

C) Go to Process Monitor and verify this process instance is still in a POSTING status.

D) From sql B, take note of the CONTENTID value.

E) Go to the Report Repository location and search for a directory named [CONTENTID from above].

If this directory exists, it means the output has already been posted.

F) Issue a DELETE FROM PS_CDM_TRANSFER where TRANSFERINSTANCE='value from A'

G) UPDATE PS_CDM_LIST SET DISTSTATUS = '4' where TRANSFERINSTANCE = '###'

H) 
Tools 8.4x -8.43

UPDATE PSPRCSRQST SET RUNSTATUS '12' where PRCSINSTANCE = '###'

Tools 8.44 and above

UPDATE PSPRCSRQST SET DISTSTATUS ='4' where PRCSINSTANCE = '###'

I)
Tools 8.4x -8.43

UPDATE PSPRCSQUE SET RUNSTATUS = '12' where PRCSINSTANCE = '###'

Tools 8.44 and above

UPDATE PSPRCSQUE SET DISTSTATUS = '4' where PRCSINSTANCE = '###'

J) On the affected Process Scheduler machine, go to the PSHOME/appserv/prcs/<domain>/log_output directory. If there is a directory for the process instance number from step B, delete the directory.

3. Restart the Process Scheduler(s),

Note: You may need to repeat this series of events depending upon the nature of the blocking condition.  Meaning, there could be multiple instances causing the blocking, so it's best to make these changes one instance at a time.

Case 2:

There are more entries in PS_CDM_TRANSFER and so many reports struck in posting ..

Have deleted all the entries from PS_CDM_TRANSFER and set the reports to not posted. Then Manually resend the reports solved the issue.

DELETE FROM PS_CDM_TRANSFER;
UPDATE PSPRCSRQST SET DISTSTATUS=4 WHERE DISTSTATUS=7;
UPDATE PSPRCSQUE SET DISTSTATUS=4 WHERE DISTSTATUS=7;

UPDATE PS_CDM_LIST SET DISTSTATUS=4 WHERE DISTSTATUS=7;

If this is repeating and want to avoid in future. Tuxedo message parameters need to be tuned.

To help avoid this issue in the future:

1.  Increase Max Instance number of distribution agents for the Process Scheduler.  The following setting can be set to a maximum value of 5 in psprcs.cfg.

Values for config section - PSDSTSRV
    Max Instances =1
    Recycle Count=0
    Allowed Consec Service Failures=0
    Max Fetch Size=0

2.  Increase the default 'Maximum Message Queue Size' for Tuxedo.

Generally, this blocking condition in Tuxedo occurs when the IPC Resource are insufficient on the server. This problem has been tracked primarily to the value of "Maximum Message Queue Size" parameter. The default on Windows for this is 65536, which is the maximum size of a queue (in bytes). This parameter and others can be tuned through the Settings->Control Panel ->BEA Tuxedo Administration. Other Tuxedo parameters that might be tuned are "Number of Message Segments" and "Size of Message Segment". The corresponding UNIX kernal parameters are MSGMNB, MSGSEG, MSGSSZ.

Refer: http://psoftdba.blogspot.sg/2013/11/peoplesoft-reports-posting-issuetuning.html


IPC_PARAMETERS Reference: 

UNIX MESSAGE Queue Reference

Rest Next :-)

Entrust Entelligence Group Share for securing PeopleSoft Report Repository Folders

Some customers may want to encrypt their report repositoty as the report folders has the sensitive data which they do not want to expose to security threats.

With 3rd party application Entrust Entelligence Group Share this can be obtained and the report repository will be encrypted automatically and decrypt while accessing from online.

The detail configuration wihch I will post later. Also can be obtained from the Entrust Entelligence Group Share documentation.

entrust.com


Rest Next :-)

Remote desktop use windows button by default and locks itself and minimizes windows. How to issue Ctrl-Alt-Del in remote desktop.

I know its irrelevant to Peoplesoft. But providing this detail just to save your time of engaging Wintel teams and service desk.  :-)  I had to surf and with no results found I called service desk but no luck.

Issue:
========
When you connect to remote desktop sometimes it locks itself when you type key 'l' or 'L'  and minimize all windows when you type 'm or 'M' .

Reason: Somehow the remote desktop issues the windows key combinations and thats causeing the issue.


Solution:

Issue Ctrl-Alt-Del and cancel. but in remote desktop you cannot issue as it will apply the key combination to your desktop instead remote desktop.

In order to send Ctrl-Alt-Del keystrokes to remote computer connected via Remote Desktop client,

Ctrl+Alt+End

Rest Next :-)

psconfig.sh /usr/lib/hpux64/dld.so: Unable to find library 'libgpnet.sl'.

Error: /usr/lib/hpux64/dld.so: Unable to find library 'libgpnet.sl'. occurs when trying to create or manage application server domains.

Issue could be with environment variables not set properly.

For 8.51 below is the variables are mandatory to invoke psadmin and manage domains.

Make sure below varialbel are set and its better to save it in seperate <envname>.sh or update these values in psconfig.sh.

PS_HOME=<Update PS_HOME here>;export PS_HOME

TMP=/tmp/;export TMP

TUXDIR=<Update TUXEDO installation directory here>;export TUXDIR

PATH=$PS_HOME/bin:$TUXDIR/bin:<update Oracle client HOME Path here>/bin:$PATH;export PATH

ORACLE_HOME=<update Oracle client HOME Path here>;export ORACLE_HOME

PS_CFG_HOME=<Update PS_HOME or the path you want to customize here>;export PS_CFG_HOME

LD_LIBRARY_PATH=$TUXDIR/lib:$ORACLE_HOME/lib:/lib:/usr/lib;export LD_LIBRARY_PATH

SHLIB_PATH=$TUXDIR/lib:$ORACLE_HOME/lib:/lib:/usr/lib;export SHLIB_PATH

cd $PS_HOME

Weblogic Webserver crashes with error BEA-100025 - java.lang.OutOfMemoryError: unable to create new native thread

Error: 

<Error> <HTTP Session> <Machine> <PIA> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1383913301180> <BEA-100025> <An unexpected error occurred in HTTP session timeout callback while deleting sessions.
java.lang.OutOfMemoryError: unable to create new native thread


Possible Cause: 

This problem is caused by a lot of Jolt threads being spawned on the web server. This can happen if users are closing their browsers without logging off, or if new sessions are rapidly being created because of cookie problems or interference by other network devices.

Solution: 
 
First reported on HPUX, about 1800 threads were being spawned, each thread was taking 512K memory. Along with a heap of 700M this accounted for almost 1600MB of space, which did not leave much for the JVM and its libraries from the 2GB addressable space.

The following tuning should be done to optimize and timeout these threads:

1. Decrease heap size to 512m, or less to JAVA_OPTIONS in setEnv.sh (setEnv.cmd). If running as a service on Windows, delete and reinstall the service.

2. Set tuxedo_network_disconnect_timeout in your web profile. Set this to be more than the highest "Service Timeout" value in psappsrv.cfg on the AppServer, but less than the Session Timeout period in your web profile.


3. Reduce Jolt Client Cleanup Timeout in the Application Server configuration from 60 minutes to 15 minutes. You will have to reconfigure the Application Server Domain for this change to take effect.


4. Add "-Xss256k" to JAVA_OPTIONS in setEnv.sh (setEnv.cmd). If running as a service on Windows, delete and reinstall the service.

Before attempting this resolution, make sure that the error is not caused by a real shortage of kernel threads, check your OS documentation on how to configure and monitor kernel threads.
See Note 608218.1 E-AS: Application Server and Web Server Kernel Parameters for UNIX and Linux Operating Systems 

How to find the table size in Oracle.

Sometimes you may have to find the list of tables size or specific table size to do cleanup or know the  growth.

In this scenario I want to find the backup tables residing in the database and size of each tables occupied in the database. 

Query: 

select a.owner,a.table_name,b.created,c.bytes/1024/1024 MB from dba_tables a, dba_objects b, dba_segments c 

where a.owner='SYSADM' and table_name like '%BACKUP%' 

and a.table_name=b.object_name and a.table_name=c.segment_name and segment_type='TABLE' order by c.bytes desc

You can add or modify the query according to your needs. 

PeopleSoft - What Is a Patch, a Fix, a Bundle, a Maintenance Pack, a Service Pack, a Feature Pack, a Report ID, an Update ID, a Project (file), an ICE, an Incident, a Resolution, a Bug, a Defect, a POC, a One-off?

When searching for Knowledge Documents in the Knowledge Base or during emails exchange or phone contacts with a Support engineer, soon or later you will come across one of the following terms:

• Patch
• Fix
• Bundle
• Maintenance Pack
• Service Pack
• Feature Pack
• Report ID
• Update ID
• Project (file)
• ICE
• Incident
• Resolution
• Bug
• Defect
• POC
• One-off

All these terms have something to do with code changes that we have made in order to correct the application and make it work how it was intended to work. However do all these terms mean and what is the difference between them? An example below shows how all these terms are related to one another.

This knowledge Documents provides the following information:

• a definition of all the above terms,
• an example showing the relationship between all the above terms
• where to find instructions in order to download and install a Fix/Patch or a Bundle or a Maintenance Pack.

SCOPE

This Knowledge Document is intended to any person who is involved in the system maintenance, who is communicating with the Oracle Support department, or who has access to the Oracle Support Portal and can search the knowledge base.

DETAILS

1) Definitions

Patch & Fix

• A Patch or Fix is a the direct solution of a specific issue. The code changes to install have been made to fix that specific issue only. We usually do not post a Patch/Fix on the Oracle Support Portal unless it has been made to fix a production critical issue and should be available for customer to download as soon as possible. The terms 'Patch' and 'Fix' are not official terms.

Bundle

• A Bundle is group of Patches/Fixes. Depending on the release, they can be product specific (for example Inventory only) and include a few Patches/Fixes, or product line specific (for example SCM) and include hundreds of Patches/Fixes. The term 'Bundle' is an official term.

Maintenance Pack (MP)

• A Maintenance Pack is a group of Bundles. They are usually available in 2 versions: a delta one which includes the Bundles which have been posted on Oracle Support Portal since the previous Maintenance Pack and acumulative one which includes all Bundles that have posted since that specific release went GA. The term 'Maintenance Pack' is an official term.

Feature Pack (FP)

A Feature Pack contains all of the code included in a standard major release (such as PeopleSoft Financials and Supply Chain Management release 9.1) plus a roll-up of capabilities and updates previously delivered in Bundles and Maintenance Packs. If you are upgrading to the release (say 9.1) for which the Feature Pack has been made available, we have recertified the upgrades and integrations to the Feature Pack. If you are implementing PeopleSoft applications for the first time, you can implement the Feature Pack version of the release to reduce the amount of time spent on loading maintenance. Feature Packs do not replace the existing delivery mechanisms of Bundles and Maintenance Packs, so if you have already deployed release 9.1, we encourage you to continue using Bundles or Maintenance Packs to obtain the latest code updates and feature enhancements. The term 'Feature Pack' is an official term.

Update ID & Report ID

• This is how an individual Patch/Fix, a Bundle, or a Maintenance Pack is available to customers on the Oracle Support Portal. Both terms mean the same thing. They refer to a 6 digits numbers (i.e. 810741). The terms 'Update ID' and 'Report ID' are official terms.
• The Report ID is to be found when searching for a Patch/Fix/Bundle/Maintenance Pack on the Oracle Support Portal (see the instructions which are available in the Knowledge Document 1432368.1).
• After searching and finding a Patch/Fix/Bundle/Maintenance Pack on the Oracle Support Portal and clicking on it, a page displays with the Update ID of that particular Patch/Fix/Bundle/Maintenance Pack. Same thing when clicking on a Bundle or Maintenance Pack using the Maintenance Schedule Knowledge Documents (for example the Knowledge Document 1276035.1 (PeopleSoft Enterprise SCM Maintenance Schedule for 2011-all).

Project (file)

The code changes for a particular Patch/Fix, a Bundle, or a Maintenance Pack, which are made available to customers on the Oracle Support Portal via an Update ID/Report ID (see the definition of these terms above), are usually available in a Project (file), which name is PRJ + Update ID/Report ID (i.e. PRJ742695). This Project (file) can be loaded into the application using Application Designer (select 'Tools > Copy Project > From File...' in the top menu). If you start a release 8.9 Application Designer and search for the Project called PRJ742695 (select 'File > Open' in the top menu and then select 'Project' in the 'Definition' drop down list), you will get all the objects which have been modified by the SCM 8.9 Bundle #25. The term 'Project (file)' is an official term.

ICE & Incident & Resolution ID

All 3 terms are used by Oracle internally. These terms are coming from a system called ICE that Development used to track and resolve issues. ICE stands for 'Incident, Change and Enhancement'.

• An ICE (usually a number of up to 10 digits ending with '000' i.e 1756357000) was usually created by a Support Engineer when there was a need to report an issue to Development. An ICE number is most of the time reporting one issue only. It is focusing on the issue itself meaning that we do not state at the ICE level for which product and release the issue occurs. ICE's were also created by Development to work on Bundles and Maintenance Packs.
• An Incident ID is derived from an ICE. It has the same number as an ICE except for the last 3 digits which are used as a counter. So the first Incident linked to the ICE # 1756357000 has 1756357001 as number, and the second Incident 1756357002. There's always an Incident number per product and release impacted by the issue reported at the ICE level. So when reporting an issue to Development for General Ledger (GL) and Order Management (OM) releases 8.9 and 9.0, four Incidents were created: one for GL release 8.9, one for GL release 9.0, one for OM release 8.9, and one for OM release 9.0. The same rule applies for Bundles. The only difference with an issue is that a Bundle is created for a specific release. For example, the release SCM 8.9 Bundle #25 has 10 Incidents: one for Purchasing, one for Inventory, one for Manufacturing, ... all for release 9.0.
• A Resolution ID is a 6 digit number which is created and linked to an Incident ID when the Incident ID is resolved by Development. A Resolution ID is also created if the changes made by Development for a particular Resolution ID require translation. The code changes made for a particular Resolution ID are usually not posted on the Oracle Support Portal unless they have been made to fix a production critical issue and should be available for customer to download as soon as possible. In that case the Resolution ID is the same as the Update ID/Report ID (see the definition of these terms above). A Resolution ID is also created for a Bundle or a Maintenance Pack and linked to all the Incidents which were created for the Bundle or the Maintenance Pack. The Resolution ID for a Bundle is also linked to the Incident ID's which code changes are included in that Bundle. When the Bundle or the Maintenance Pack are posted on the Oracle Support Portal, the corresponding Update ID/Report ID (see the definition of these terms above) are the same as the Resolution ID.

Bug

• This term is by Oracle internally. These terms are coming from a system called BugDB which Development is currently using instead of ICE (see the definition of this term above) to track and resolve issues. A Bug (usually a number of up to 8 digits) is product and release specific. So you have 2 different Bugs when an issue has to be fixed in 2 different releases.
• Incidents for release 8.8 and above have been migrated to BugDB and have been assigned to Bug numbers.
• There is no rule between Bug numbers similar to what we used to have with Incidents. 2 Incident numbers for the same ICE are almost the same, except for the counter. 2 Bugs reporting the same issue for 2 different releases could have numbers which differ a lot (for example Bug 12834762 is reporting a release 9.0 Inventory issue and Bug 12849890 is reporting the same issue but for release 9.1).

Defect

This term is by Oracle internally. A Defect is a term used in the application used by the Support Engineer to work on Customer's Service Requests. A Defect used to be linked to an ICE. It is now linked to a Bug.

POC

A POC (understand Proof Of Concept) is a Patch/Fix that we are providing to customers in partnership with them and with their understanding that although the Patch/Fix has been unit tested by Development, it has not yet completed the rigorous quality assurance testing that all posted Patches/Fixes pass through.

Customers, in accepting and installing this customization, acknowledges that we are delivering this Patch/Fix fix through an atypical method of delivery, and as such, may encounter problems in its content or installation. Customers have been made aware of the risks involved in accepting a Patch/Fix in this manner. A Patch/Fix delivered in this manner (and until the posted versions are applied) is considered a customization; any problem or error conditions customers may receive as a result of installation of this Patch/Fix is not supported.

It is recommended to perform a full system backup prior to the installation of this Patch/Fix in the event there is a need to return to the prior database version.

We cannot guarantee that the version of the file and/or changes documented or provided in a POC match the source file or object version residing at the customer site. As such, it is recommended the code application should not be directly applied from this medium, but instead customers should enter any revisions via manual changes. Problems that arise from the application of the custom code changes are not supported by Oracle.

One-off

A One-off is a Patch/Fix provided by Development in an attempt to resolve a performance issue. Performance issues are often fixed using a trial and error method as these issues are usually linked to customer's setup and data. So it is very difficult fro Development to know in advance how efficient the Patch/Fix is going to be. Depending on the result of a One-off, Development could provide another one in order to further improve the performance of the application. Customers should be ready to accept and test any one-off that Development may provide.

It is recommended to perform a full system backup prior to the installation of a One-off in the event there is a need to return to the prior database version.

We cannot guarantee that the version of the file and/or changes documented or provided in a One-off match the source file or object version residing at the customer site. As such, it is recommended the code application should not be directly applied from this medium, but instead customers should enter any revisions via manual changes.