Understanding and Addressing Stuck Threads in Crowd 3.2.1 and Later

The warning message you encountered indicates that a thread in your Crowd application has been active for an extended period, exceeding the configured threshold for stuck thread detection. Specifically, the log entry shows that the thread handling a request to authenticate a user has been active for over 67 seconds, while the threshold is set at 60 seconds. This situation may lead to performance issues if not addressed.

Causes of Stuck Threads

Resource Pooling Issues: The thread may be waiting for a resource, such as a database connection, which can occur if the connection pool is exhausted or improperly configured. This is often seen with connection pools like C3P0.
Database Performance: Slow database queries can cause threads to hang as they wait for responses. This can be exacerbated by high load or inefficient queries.
High Load: If the application is experiencing a high volume of requests, threads may queue up, leading to longer processing times.
Deadlocks: Threads may be waiting on each other to release resources, causing them to become stuck.

Suggested Solutions

To address the stuck thread issue, consider the following actions:

Review Resource Pool Configuration: Check the configuration of your connection pool (e.g., C3P0) to ensure it can handle the expected load. For example, you might want to increase the maximum pool size. Example configuration in hibernate.cfg.xml:

  <property name="hibernate.c3p0.max_size">60</property>
  <property name="hibernate.c3p0.timeout">300</property>

Optimize Database Queries: Analyze the queries being executed to identify and optimize any that are slow. Ensure that your database is indexed appropriately to improve performance.
Increase Thread Pool Size: If the application is under heavy load, consider increasing the maximum number of threads available to handle requests. This can be configured in server.xml for Tomcat. Example configuration:

  <Connector port="8095" protocol="HTTP/1.1"
             connectionTimeout="20000"
             redirectPort="8443"
             maxThreads="200"/>

Investigate Deadlocks: Use monitoring tools to identify potential deadlocks in your application. Refactor code as necessary to resolve these issues.
Adjust Stuck Thread Detection Threshold: If the current threshold is too low for your application’s requirements, consider increasing it. This can be done in server.xml:

  <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="120"/>

Conclusion

The stuck thread warning indicates potential performance issues that should be addressed to maintain the stability and responsiveness of your Crowd application. By reviewing your resource pool configuration, optimizing database performance, increasing thread pool sizes, investigating deadlocks, and adjusting detection thresholds, you can mitigate these warnings and improve overall application performance.

Citations:
[1] https://jira.atlassian.com/browse/CWD-3692
[2] https://community.atlassian.com/t5/Confluence-questions/Thread-may-be-stuck-318-threads-are-monitored-by-this-Valve-and/qaq-p/683210
[3] https://confluence.atlassian.com/display/CROWD/Crowd%2B3.2%2BUpgrade%2BNotes
[4] https://community.atlassian.com/t5/Crowd-questions/Crowd-server-stopping-unexpectedly/qaq-p/654052
[5] https://confluence.atlassian.com/display/JIRAKB/How%2Bto%2Bread%2Bthe%2BTomcat%2BStuck%2BThreads%2Blog%2Bor%2BStuckThreadDetectionValve%2Bin%2BJira
[6] https://confluence.atlassian.com/crowd/crowd-3-2-release-notes-949250099.html
[7] https://stackoverflow.com/questions/32203272/how-to-configure-stuck-thread-handling-for-only-one-new-restful-web-service
[8] https://stackoverflow.com/questions/76384599/tomcat-stuck-thread-detection-in-spring-boot
[9] https://gist.github.com/victorhooi/6709897
[10] https://github.com/hazelcast/hazelcast/issues/160
[11] https://docs.oracle.com/cd/E17904_01/apirefs.1111/e13952/taskhelp/tuning/TuningExecuteThreads.html
[12] https://docs.diffusiondata.com/docs/latest/manual/html/administratorguide/tuning/threadpools/c_threadpools.html
[13] https://forum.joomla.org/viewtopic.php?start=90&t=829729
[14] https://github.com/cptactionhank/docker-atlassian-confluence/issues/53
[15] https://ndma.gov.in/sites/default/files/PDF/Reports/managingcrowdsguide.pdf
[16] https://www.reddit.com/r/jira/comments/sogu1p/threads_getting_stuck_slown_and_unresponsive/