Problem
Certain older versions of OpenSSL contain a bug that causes Apache to fail while starting on systems that have a large number of VirtualHosts, at least one of which is used for SSL access.

Red Hat® and CentOS 4.6+ and 5.x have been patched for this issue. If your system is running one of those distributions and is up-to-date, you should not encounter this bug.

If you are running a different distribution and encounter segmentation faults when starting up Apache 2.0 or 2.2, you may need to contact your Linux distribution vendor and inquire about this problem.

Apache 1.3 is not affected by this issue.

Detailed Analysis
The core problem is that versions of OpenSSL before 0.9.7k and 0.9.8c have a fault that appears when certain functions are used after FD_SETSIZE (generally 1024) file descriptors have been opened. For more information, see:

http://marc.info/?l=openssl-dev&m=114301777619250&w=2

With Apache 1.3, initialization of mod_ssl occurs before httpd.conf is processed. With Apache 2.0 and 2.2, most initialization of mod_ssl occurs after httpd.conf has been processed, meaning that all the VirtualHost logfiles will be open when ssl_init_Module() is executed. The functions in OpenSSL affected by this issue currently are:

RAND_status(), called by ssl_init_Module() -> ssl_rand_seed()

RSA_generate_key(), called by ssl_init_Module() -> ssl_tmp_keys_init() -> ssl_tmp_key_init_rsa()

It does not appear that there are any faults once initialization of mod_ssl is completed on Apache 2.0 and 2.2. Starting Apache with a trimmed down httpd.conf file, copying the complete httpd.conf file over it, and then restarting Apache does not cause a segfault because the ssl_init_Module() is not executed again.

How do you know whether an install of Apache 2.x suffers from this problem?

A quick way of checking for this specific problem is by capturing the output of the httpd startup via the strace program. The OpenSSL bug manifests at the end of the output as:

1. Reading from /dev/urandom.
2. Closing /dev/urandom.
3. Getting the current time.
4. Seg Fault.

A more positive identification can be done by disabling all SSL Vhosts in httpd.conf and restarting Apache. If Apache restarts without error, then the problem lies with mod_ssl/OpenSSL.

To pinpoint the exact function call causing the segmentation fault, Apache must be recompiled with debugging symbols so that a usable backtrace can be created with GDB. In order to recompile Apache with debugging symbols, you must simply set the appropriate CFLAGS and then run EasyApache, like so:

• export CFLAGS=”-g”
    /scripts/easyapache –build

How Can This Bug Be Fixed?
Unfortunately, the fix provided by Red Hat is only in CentOS/RHEL 5 and Fedora® at this time. Older versions are left with the following options:

1. Update Your Operating System, if Possible — Granted, this is no trivial task. But it will give you the latest OpenSSL without this OpenSSL bug.
2. Use Only Apache 1.3 – Tests with Apache 1.3 show that it is not susceptible to this bug, due to the way it performs some module initialization before processing httpd.conf, while Apache 2.x does the module initialization after processing httpd.conf.
3. Patch Apache 2.x – At this time, cPanel patches Apache 2.0 and 2.2 mod_ssl to avoid the call to RAND_status() for older versions of OpenSSL. On some systems, this is sufficient to avoid any problems. Reordering the call to RSA_generate_key() is more involved, since that function must not run before other initialization functions have executed. It is possible that other function calls during mod_ssl’s initialization also cause segmentation faults under some circumstances.

After applying a patch, if you choose that option, please be sure to test Apache again to see whether it is still suffering from the issue.

A note about the /opt/openssl workaround
We previously recommended that you use /opt/openssl to work around this issue. However, that method does not ensure that you have the latest patched version of OpenSSL, as /opt/openssl is not updated by either EasyApache or the system package manager.

This can be a security concern, and requires you to manually track and update the OpenSSL installation in /opt/openssl. In addition, when using the /opt/openssl workaround, multiple copies of libssl can be loaded into memory. When a function is called by a program looking to interact with libssl, there may be two potentially different copies of that function loaded into memory.

There’s no real way to determine which function is being called, and therefore, troubleshooting issues with a /opt/openssl implementation is difficult to manage.

For these reasons, we do not recommend using /opt/openssl to work around this bug.

Bug: Apache 2.2 Child Processes in “G” Status
Problem
Apache 2.2 has a bug affecting some systems, wherein a graceful restart does not fully clean up any running processes.

The result is that child processes are stuck in “G” status, and more file descriptors are used, which can push Apache over its limit and crash it.

Fix
Apache is aware of this bug and patches have been submitted; however, they are all problematic. For more information, see http://issues.apache.org/bugzilla/show_bug.cgi?id=42829.

Apache is attempting to deal with the problem upstream, but the fix is still not finalized. For more information, see http://svn.apache.org/viewvc/httpd/httpd/trunk/
server/mpm/prefork/prefork.c?view=log&pathrev=613260.

If you experience this issue on a server, your best and only supported option is to run Apache 2.0 instead of 2.2. This is because the patches are not ready for production, as they have known issues and need further refining by Apache.