If you’ve ever manned a Siebel HelpDesk, you’ve received the frantic calls from users whose data or workflow has been lost when the system disconnected them. Of course, you do what you can to help recover their work or confirm that a process was executed or, if not, where it terminated – users come first – but what you really want to do is identify and correct the cause of their problems so that the disconnect issues don’t affect other users. Unfortunately, that’s easier said than done, unless you have a powerful Application Performance Monitoring solution like Germain UX. Today, we’d like to share with you the most common causes for these annoying Disconnects to help you get to the root cause of the problem.
The Most Common Causes of User Disconnects in Siebel
In our experience, and as confirmed by our clients using Siebel, these are the most common causes of Siebel disconnects:
- Network failure
- TCP/IP timeout
- Siebel Object Manager crash
- Built-in Memory Recycle and management mechanisms
- Problem with Siebel Enterprise Configuration
Working through that list to diagnose a disconnection issue would take hours, if not days or weeks, of your time scouring huge log files, reviewing lines and lines of code, tracing user interactions, and drilling through configuration files. With Germain UX, this is a much faster process: its deep visibility into the Siebel application layers and monitoring of the infrastructure on which it resides enables system engineers to quickly perform root cause analysis (RCA) for all Siebel issues.
I’ve provided a short summary of the troubleshooting steps to diagnose a Siebel user disconnect, noting where and how Germain UX (real-time user, process and technology monitoring) & germain CRT (automatic performance review of eScript, Repository Objects and DB Indexes) can streamline the process.
Step 1 – Locate the Disconnect
On the Germain dashboard, user disconnects display in black – they’re easy to spot. Another screen produces trend analysis of disconnects to quickly identify whether you are dealing with an emerging problem or a one-off system hiccup. Without an APM solution, you must plow through log files to find the disconnect, files that might be hundreds of thousands of lines of code. Once the disconnect is located, you can proceed to perform RCA.
Step 2 – Check the Network
This most frequent cause of Siebel user disconnects should be eliminated first. The manual RCA process requires you to trace the network status back to the time the disconnect occurred, and then locate the network branch that was affected. A review of those log files may uncover problems with network or other issues. The process is faster in Germain UX as the RCA Screen provides a high-level view of the network environment at all times and can easily be dialed back to the exact moment at which the disconnect occurred. A visual alert will display red if the network had problems, and clicking that alert identifies the affected branch. Within seconds, you’ve located the disconnect, if there was one.
Step 3 – Object Manager May Be the Culprit
With the network eliminated as the problem’s source, most engineers would next check for a crash of the Object Manager during the specific user transaction that was disconnected. Prepare to spend a lot of time reviewing details of the Object Manager log files for both system components, such as SCBroker and SRBroker, and for user-driven components, including eCommunicationObjmgr_enu, SCCobjmgr_enu, and others, to identify a crash linked to the user transaction. If a crash did occur, research its crash file and the associated object manager log files and fdr file to understand its root cause.
With Germain UX, on the other hand, it’s a breeze to troubleshoot the crash and debug it. A quick glance at the Germain dashboard tells you which users were disconnected due to a crash, their logins, and what transactions they were executing at the time of the crash. If an issue created by a business scenario is the root cause of the crash, Germain UX will tell you that too.
An Object Manager crash may be caused by a specific sequence of actions by the Siebel user or it may involve systemic failures such as might result from an overload of users or faulty integration. Germain UX’s Advanced RCA screen displays all relevant statistics for each crash on a single page allowing the technician to easily understand the crash’s root cause and, importantly, be able to reproduce the crash in a test environment.
Step 4 – Drill, Baby, Drill.
If you’ve sloshed through this troubleshoot without an APM utility thus far, I wouldn’t want to be sharing a cubicle with you. You had to be frustrated when the problem wasn’t an Object Manager crash. You’ve taken many hours in a manual troubleshoot to reach a point Germain UX would have had you in a few minutes, and you’re still looking at an uphill battle.
If your problem is not a network issue or Object Manager crash, you have to drill deep. For example, a broken Memory Recycle mechanism will start recycling Siebel processes. Without Germain UX, you have to trace individual siebmtshmw processes or related files to assess the amount and type of processing being performed in each one and separate internal from user-processes, etc. Their configurations must be reviewed – they can be acceptable one day yet be unacceptable two weeks later.
Similarly, if you suspect that the user disconnected is due to a misconfiguration of an MT Server, a manual troubleshoot requires that you analyze that user scenario and related files to confirm the diagnosis. Or, if you have my luck, it would be a Siebel enterprise configuration issue – for example, a process may recycle when the Minimum and Maximum amount of MultiThreaded Server (or MTServer represented by the siebmthsmw system process) aren’t set to the same value.
I could go on. The point is: had you deployed Germain UX this entire process is almost fully automated. Siebel disconnects, with their related data loss or decreased productivity, are very frustrating to the user community. If you are dealing with a lot of them, we hope this article helped you find the root cause(s) of the problem quickly.
Topic for Discussion: What has been your experience in dealing with Siebel user disconnects? Please share your tips and tricks in the comments section.
