More useful Solaris commands

Here is a list of Solaris 10 commands that I found helpful, to monitor and optimize the performance of the system:

1. To find the top 10 largest files on the file system:
du -a /var | sort -n -r | head -n 10

2. Sort the ls command to show the largest files first:
ls -l | sort -nk5

3. Find out the top processes taking up the CPU:
prstat -s cpu

4. Find out the top processes taking up the memory:
prstat -s size

5. Find out the number of mails in the mail queue:
mailq |wc -l

6. Get the information about a specific mail in mail queue
cat /var/spool/mqueue/qf

Solaris 10 commands: How to see Documentum Server logs

Checking documentum server logs while a workflow is running is very useful, also while debugging the content server methods. Here are the steps:

1. Login to Solaris 10 (From windows, I use putty or SSH client, both are good)
2. Type bash to change the shell (I find bash is very convenient shell than ksh)
3. Change to documentum folder: type cd $DOCUMENTUM/dba/log
4. Open the log: type tail -100f <<docbase_name>>.log

The above tail command lists out last 100 lines from the log file. As and when the log file is updated by the server, the refresh happens automatically enabling to see last 100 lines always. This is very useful when the server is throwing lot of log messages.

Solaris 10 commands: How to clear Memory segments and Semaphores

We have a Solaris 10 box which hosts a Documentum docbroker and around 22 docbases. Frequently, due to high memory usage, docbases go down and everything stops working. This also happens when the backend database goes down.

To resolve this, I normally restart all the docbases and docbrokers after clearing the processes, memory segments and semaphores.

Here is the step by step process that I follow to restart the Documentum Server.

STEP 1: Kill all the docbase processes first.
1. Type ps -ef grep documentum which will list out all the processes along with the process ids.
2. Copy the output(from console) of this command and paste it into an Excel sheet. (I use SSH Secure Shell to connect to Solaris from Windows)
3. Extract just the process ids and arrange them with spaces between them. There are many ways to do it, I use the Text Import Wizard in Excel to achieve this. You should have your process ids in format like this:
12345 234 12333 12356 123213
4. Type kill -9 and paste the above process id string and hit enter.
kill -9 12345 234 12333 12356 123213
5. All your documentum processes are killed now.

STEP2: shutdown the docbroker

STEP3: List all the semaphores and memory segments with command ipcss -a.
1. This will list out both the semaphores and memory segments.
2. Copy the output as mentioned in STEP1 to an excel sheet.

STEP4: Kill the memory segments first.
3. Copy only the memory segment ids. Leave out the memory segments for root.
4. Prefix ipcrm -m to each id. Your list should look like this:

ipcrm -m 435348
ipcrm -m 23465
ipcrm -m 23122
ipcrm -m 93747

5. Copy the above list and paste it into Solaris Console.
6. Memory segments are now killed.

STEP5: Kill the semaphores now
1. Copy only the semaphore ids. Leave out the semaphores for "root".
2. Prefix ipcrm -s to each id. Your list should look like this:

ipcrm -s 435348
ipcrm -s 23455
ipcrm -s 99888
ipcrm -s 435345

3. Copy the above list and paste it into Solaris Console.
4. Semaphores are now killed.

STEP6: Start the docbroker
STEP7: Start the docbases