Uploads or exports stuck on the on-screen log with no progress? Background jobs not completing? This guide walks through how to check and recover the three worker layers involved: god (the process monitor), resque (the background job workers), and puma (the application server).
After triggering an upload or export, the on-screen log should update continuously until Worker process completed appears. If it stalls with no progress, work through the sections below in order.
god is the process monitoring service that supervises all the other workers. If god itself isn't running, nothing else will be either — so this is always the first thing to check.
Run the following in the console as dradispro:
$ god status
The expected output is:
cic-puma: up dradispro-puma: up nginx: up resque: cic-resque-1: up dradispro-resque-1: up
If any process shows as down, or if the god command itself fails, try restarting god:
$ sudo /etc/init.d/god restart $ god status
If you've recently rebooted the instance and god didn't come back up cleanly, force it to reload its configuration:
$ god load /etc/god.d/dradispro-puma.god $ god load /etc/god.d/cic-puma.god $ god restart
Still not right? Check the god log for more detail:
$ vi /var/log/god.log
If the god command returns command not found, your terminal session may not have loaded the correct environment. See the SSH Environment section of the instance troubleshooting guide.
Resque workers handle background jobs — uploads, exports, migrations, and project tasks. A stuck resque process will cause uploads and exports to stall on the progress log even though god reports everything as up.
The first thing to try is a targeted resque restart via god:
$ god restart dradispro-resque-1
If you're unsure which worker is stuck, or if the above doesn't resolve it, a full god restart will also cycle the resque workers:
$ god restart
Retry the upload or export in the web app. If it's still stalling, move on to clearing the jobs queue.
In Dradis v4.18 or later, you can inspect the queue directly in the UI before going to the console. Click the cogwheel icon and select Background Jobs, or navigate manually to /pro/admin/jobs in your browser. The Queues tab shows any pending jobs, and the Failed Jobs tab shows anything that is stuck or has errored out.
To kill stuck resque processes from the console, run the following as dradispro:
$ cd /opt/dradispro/dradispro/current $ ps aux | grep resque
A stuck worker will show output like this, with the job it is stuck on:
dradisp+ 12838 66.0 6.3 358416 130092 ? Rl 20:08 0:00 resque-1.27.4: Processing dradis_export since 1547687319 [ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper]
Note down the process ID (the second column — 12838 in the example above) and kill it:
$ kill -9 12838
Run the grep command again and repeat the kill -9 for any remaining stuck processes. You're done when all resque output shows Waiting rather than Processing:
dradisp+ 8747 0.0 5.5 324868 114892 ? Sl Jan14 0:24 resque-1.27.4: Waiting for dradis_export,dradis_upload,dradis_migrate,dradis_project
Once the workers are all in a waiting state, retry the upload or export in the web app.
If the restarts above resolve the stall but the problem keeps coming back, the resque logs are the place to look for the underlying cause. Check the following files as dradispro:
$ vi /var/log/god/dradispro-resque-1.log
Any recurring errors or exceptions in these logs are worth capturing and sending to the Support team.
Puma is the application server that serves both Dradis itself and the CIC (the Dradis Administrator Console). While resque handles background jobs, Puma needs to be healthy for the web interface to respond at all. A down Puma process can manifest as a 404 from nginx or a completely unresponsive UI even when god reports a clean status.
Check how many Puma master processes are running:
$ ps aux | grep "[p]uma master"
You should see two processes — one for the CIC and one for Dradis. If either is missing, try starting it via god:
$ god start dradispro-puma
Or restart both:
$ god restart dradispro-puma $ god restart cic-puma
If Puma is failing to start or keeps going down, the log files will usually tell you why. Check the following as dradispro:
Dradis:
CIC:
$ vi FILEPATH
Still stuck after working through the steps above? Gather the Support bundle from /setup/support in your instance and send it to our Support team — we'll give you a hand. If you aren't able to reach the endpoint above to download the support bundle, you can get it from a terminal session to Dradis with:
$ /opt/dradispro/bin/dp-support-bundle -o > /tmp/support-bundle.tar.gz
Next help article: My tool output won't upload! →
Your email is kept private. We don't do the spam thing.