Finding Out What is Wrong
- Jason Andrew
Analytics
Checking for Hardware crashes
Did (any one of) the server machine(s) crash?
This will indicate how long the machine has been up. If this is a short period of time it is likely that the outage has been caused by machine problems. This should be reported to the system administrator to investigate further.
Checking for failed processes/start up
Has the system as a whole failed to start?
If one or more of the key processes fails to start up then the whole server will not start up. Additionally, if one of the non-key processes does not start then the rest of the system may appear to be running smoothly, but may in fact require investigation.
Has any ServiceOptimizer server process crashed? Are there any recovery files on the server?
First, check whether the ServiceOptimizer processes are still present.
In most cases they will be found to still be running, as the system monitor is designed to check the state of all its child processes and, on finding any one missing, to automatically start it up again (see System Monitor for further details). Only with a very serious problem would a process not restart and would you find not all processes present.
There should be 1 spsysmon process, 1 APIAgent process, 1 (or more) GanttRouter process(es) and then 1 process for each FRU that is active.
Irrespective of whether the system is now working OK, it is advisable to follow up on the cause of the issue:
Checking Communications
Has Connection from Client/Interface Processes Failed?
This could be due to network traffic issues.They are usually short term and correct themselves. The actual effect on the ServiceOptimizer system will depend on the nature of the incident, and whether ServiceOptimizer is set up in a distributed form.
A system administrator will be needed for diagnosis!
The following information should be collected and passed on to the system administration team to determine if the problem is, in fact, caused by network issues:
The time of the outage – as accurately as possible so it can be compared to times in the logs (note that times in log files are defined by the tz_Default value in sp083_system_parameters table in the database. Normally this will be set to the time zone of the machine location).
Is all of SERVICEPower unavailable or just some functions? In the latter case which of the main functions are affected? i.e. - appointment offering, appointment booking, job status change, job cancel, Gantt, SERVICEManager, SERVICEPlanner or SERVICEAnalytics. In the case of applications, are all or just some of their functions affected?
Is the unavailability permanent or intermittent?
Are the affected functions unavailable to all users as far as can be determined, or only to some?
Are users of any other services on the same machine having difficulty getting through?
Checking System Resource/Filestore Usage
Is there evidence in the O/S logs to show a lack of any of the resources required for the instance?
O/S knowledge will be required to follow up on this, but Resources may give some pointers.
Checking for Database Errors
There are 2 main types of database error that can occur:
Has the application attempted to do any illegal functions?
Is the a problem on the database server?
The ServiceOptimizer processes have some capability to withstand temporary database issues, however, any lasting issue will cause the affected process to restart. ServiceOptimizer needs to ensure the integrity of the data it is dealing with in memory and hence cannot risk the side effects of an error. It therefore takes the safe route of a restart. Any database errors that occur will be recorded in the failed process’s log file. Reading Log files gives details of what to look for and what it means.