Quartz Trigger Not Firing – CentOS

I have a java web application which runs on tomcat.
I am using Quartz 2.3.0
(Updated to 2.3.2, still not working)
I am initializing quartz with Properties as follows:

Properties properties = new Properties();
properties.put("org.quartz.scheduler.instanceName", hostname);
properties.put("org.quartz.scheduler.instanceId", hostname);
properties.put("org.quartz.threadPool.threadCount", "5");
properties.put("org.quartz.jobStore.class", "org.quartz.impl.jdbcjobstore.JobStoreTX");
properties.put("org.quartz.jobStore.isClustered", "true");
properties.put("org.quartz.jobStore.misfireThreshold", "600000");//until 10 minutes, dont count as a misfire
properties.put("org.quartz.jobStore.clusterCheckinInterval", "100000");
properties.put("org.quartz.jobStore.driverDelegateClass", "org.quartz.impl.jdbcjobstore.StdJDBCDelegate");
properties.put("org.quartz.jobStore.dataSource", "QCP");
properties.put("org.quartz.dataSource.QCP.connectionProvider.class", "mypackage.quartz.QuartzConnectionProvider");
properties.put("org.quartz.plugin.shutdownhook.class", "org.quartz.plugins.management.ShutdownHookPlugin");//This plugin catches the event of the JVM terminating (such as upon a CRTL-C) and tells the scheuler to shutdown.
properties.put("org.quartz.plugin.shutdownhook.cleanShutdown", "false");
properties.put("org.quartz.scheduler.skipUpdateCheck", "true");

Scheduler quartzScheduler quartzScheduler = new StdSchedulerFactory(properties).getScheduler();     

quartzScheduler.start();

then I add a job:

JobDetail runnableDetail = JobBuilder.newJob(RunnableJob.class)
                .withIdentity("runnable_JOB", "runnable")
                .storeDurably(true)
                .requestRecovery(false)
                .build();
quartzScheduler.addJob(runnableDetail , true);

and then create a trigger for it like this:

String uuid = UUID.randomUUID().toString();
Trigger trigg = TriggerBuilder.newTrigger()
                .withIdentity("runnable_TRIGGER_"+uuid, "runnable")
                .forJob("runnable_JOB", "runnable")
                .withSchedule(SimpleScheduleBuilder.simpleSchedule().withMisfireHandlingInstructionFireNow().withRepeatCount(0))
                .startAt(DateBuilder.futureDate(startDelaySeconds , IntervalUnit.SECOND)) //startDelaySeconds is between 1-300
                .build();

quartzScheduler.scheduleJob(trigg);

This is my RunnableJob class:

public class RunnableJob implements Job {
    @Override
    public void execute(JobExecutionContext context) throws JobExecutionException {
        try {
            //some code
        } catch (Exception e) {
            //log the exception
        };
    }
}

But the trigger never fires and when I see the QRTZ_TRIGGERS table my MySQL database, I can see that the trigger and the job has been added successfully, but when the right time comes to fire the trigger, I see that after some seconds the NEXT_FIRE_TIME column changes.

I have already read this and checked these values:

quartzScheduler.isStarted();
quartzScheduler.isInStandbyMode();

which was: true and false meaning that the scheduler is in the right state and also my Trigger state is WAITING.

The stranger thing is that it works fine on my system but it fails when I deploy it on the production system.

I have Windows 10 on my system and Centos and Tomcat 9 on the server and I’m using Java 14.

UPDATE:
After some changes(not to the way triggers and jobs are created), now the NEXT_FIRE_TIME doesn’t update and I checked catalina.out and It’s filled with this exception:

31-Aug-2020 11:04:56.986 INFO [QuartzScheduler_demo123.domain.co-demo123.domain.co_ClusterManager] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [META-INF/services/javax.naming.spi.InitialContextFactory]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [META-INF/services/javax.naming.spi.InitialContextFactory]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
        at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1385)
        at org.apache.catalina.loader.WebappClassLoaderBase.findResources(WebappClassLoaderBase.java:985)
        at org.apache.catalina.loader.WebappClassLoaderBase.getResources(WebappClassLoaderBase.java:1086)
        at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1197)
        at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1222)
        at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1266)
        at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1301)
        at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1386)
        at java.naming/javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:697)
        at java.naming/javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:305)
        at java.naming/javax.naming.InitialContext.init(InitialContext.java:236)
        at java.naming/javax.naming.InitialContext.<init>(InitialContext.java:184)
        at mypackage.quartz.QuartzConnectionProvider.getConnection(QuartzConnectionProvider.java:25)
        at org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:108)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.getConnection(JobStoreSupport.java:780)
        at org.quartz.impl.jdbcjobstore.JobStoreTX.getNonManagedTXConnection(JobStoreTX.java:71)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.doCheckin(JobStoreSupport.java:3307)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.manage(JobStoreSupport.java:3920)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.run(JobStoreSupport.java:3957)
- ClusterManager: Error managing cluster: Failed to obtain DB connection from data source 'QCP': java.lang.NoClassDefFoundError: java/lang/Thread
org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'QCP': java.lang.NoClassDefFoundError: java/lang/Thread [See nested exception: java.lang.NoClassDefFoundError: java/lang/Thread]
    at org.quartz.impl.jdbcjobstore.JobStoreSupport.getConnection(JobStoreSupport.java:789)
    at org.quartz.impl.jdbcjobstore.JobStoreTX.getNonManagedTXConnection(JobStoreTX.java:71)
    at org.quartz.impl.jdbcjobstore.JobStoreSupport.doCheckin(JobStoreSupport.java:3307)
    at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.manage(JobStoreSupport.java:3920)
    at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.run(JobStoreSupport.java:3957)
Caused by: java.lang.NoClassDefFoundError: java/lang/Thread
    at mypackage.quartz.QuartzConnectionProvider.getConnection(QuartzConnectionProvider.java:25)
    at org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:108)
    at org.quartz.impl.jdbcjobstore.JobStoreSupport.getConnection(JobStoreSupport.java:780)
    ... 4 more

The problem did not solve with restarting the tomcat server.

UPDATE 2:

The problem seems to be from the data source which I’m trying to get a connection from.
My tomcat instances are defined in server.xml file with different <Host> tags
and their resources are defined in the <Context> tag for each instance of my application:

<Resource name="jdbc/mysql/hamkelasi" auth="Container" type="javax.sql.DataSource"
    
    initialSize="1" maxTotal="50" maxIdle="1" 
    maxWaitMillis="20000" removeAbandonedOnBorrow="true" removeAbandonedTimeout="600"
    validationQuery="select now();" timeBetweenEvictionRunsMillis="1800000"

    username="user" password="pass" driverClassName="com.mysql.jdbc.Driver"
    url="..."
    />

And in my ContextListener I instantiate a javax.naming.InitialContext and then use .lookup(poolLookupString). I tried different approaches here by caching the InitialContext or caching data sources but didn’t succeed.

Still, the problem is only on the production server with the same configurations and the same libraries.

The error happens when I start the QuartsScheduler and when it tries to get a connection from the database. The server and the application are both working fine and only quartz fails with a really weird error saying:

java.lang.NoClassDefFoundError: java/lang/Thread

And also I’m only using the QuartzScheduler on one of my applications (for test porpuses)