Problem

I have an HP-UX server and IBM WebSphere Application Server is installed on this server. Some applications are running on this application server. One day suddenly one JVM of IBM WebSphere Application Server stopped serving the requests (JVM was running but not processing any request).

After analysing the logs with IBM and HPE support team it was found that the node was not processing any request because of the below error in the Syslog.

# cat /var/adm/syslog/syslog.log
.
.
.
Feb 21 13:20:15 DBAPPWEB syslog: Java: Number of open files: 4096
Feb 21 13:20:15 DBAPPWEB syslog: Java: Maximum number of allowed open files: 4096

Reason

As you can see the error in Syslog, the maximum number of allowed open files 4096 which is very low for a system on which applications like IBM WebSphere Application Server are running. To resolve the above error we need to increase the limit of the number of allowed open files. It can be done by changing the value of maxfiles_lim kernel parameter.

You may see and verify the current usage of maxfiles_lim by using the command kcusage, command output showing that it has reached its maximum value.

# kcusage |grep maxfiles_lim
maxfiles_lim               4096 / 4096

You can see more details about the maxfiles_lim by using the below command.

# kctune -v maxfiles_lim
Tunable             maxfiles_lim
Description         Hard maximum number of file descriptors per process
Module              fs
Current Value       4096 [Default]
Value at Next Boot  4096 [Default]
Value at Last Boot  4096
Default Value       4096
Constraints         maxfiles_lim >= 32
                    maxfiles_lim <= 1048576 maxfiles_lim >= maxfiles
Can Change          Immediately or at Next Boot

Solution

The default value of maxfiles_lim is 4096. As suggested by the IBM WebSphere Application Server support team we need to change the value of maxfiles_lim from its default value to 20000.

If you change the value of this parameter then it will be immediately come in effect.

Here are the steps to change the value of maxfiles_lim:

• First, check the value of the maxfiles_limit

  # kctune -v maxfiles_lim
  Tunable             maxfiles_lim
  Description         Hard maximum number of file descriptors per process
  Module              fs
  Current Value       4096 [Default]
  Value at Next Boot  4096 [Default]
  Value at Last Boot  4096
  Default Value       4096
  Constraints         maxfiles_lim >= 32
                      maxfiles_lim <= 1048576 maxfiles_lim >= maxfiles
  Can Change          Immediately or at Next Boot

• Now change the value of maxfiles_limit to 20000

  # kctune maxfiles_lim=20000
       ==> Update the automatic 'backup' configuration first?
               yes     - Update backup before every change
               once    - Update backup now; ask for each future change
               no      - Don't update backup now; ask for each future change
               disable - Don't update backup before any change
       ==> Update the automatic 'backup' configuration first? yes
         * The automatic 'backup' configuration has been updated.
         * Future operations will update the backup without prompting.
         * The requested changes have been applied to the currently
           running configuration.
  Tunable                 Value  Expression  Changes
  maxfiles_lim  (before)   4096  Default     Immed
                (now)     20000  20000
  #

• Now verify the current usage and limit by running the below command, here you can see that the limit changed from 4096 to 20000 and the current usage is 5011.

  # kcusage |grep maxfiles_lim
  maxfiles_lim               5011 / 20000
  

 

Now, the problem resolved and all the JVMs started processing the requests.

Last Updated: August 10, 2020