Solid Queue: The New Database-Backed Jobs for Rails

 

With Rails 8 introducing Solid Queue as the default queuing backend for Active Job, this modern background processing system deserves a proper examination. In this post, I’ll share what I’ve learned about this new job solution from Rails team, how it compares to other solutions like Sidekiq, and why you might want to consider it for your next project.

What is Solid Queue?

Solid Queue is a database-backed Active Job backend. It’s designed as a modern option compared to older tools like Delayed Job. Because it’s built into Rails’ Active Job framework, it provides a full solution for background job processing without needing extra infrastructure like Redis.

The main features that make Solid Queue stand out include:

• Regular job processing: The standard features you expect for putting jobs in and running them.
• Delayed jobs: Schedule jobs to run at specific times in the future.
• Concurrency controls: Limit how many jobs run at the same time.
• Recurring jobs: Set up jobs that repeat on a schedule.
• Queue pausing: Temporarily stop processing for specific queues.
• Job priorities: Set numeric priorities for each job.
• Queue ordering: Prioritize based on the order of queues.
• Bulk enqueuing: Efficiently schedule many jobs at once with enqueue_all.

Technical Requirements

Before you start, make sure your setup meets these needs:

• Rails 7.1+ (though it’s the default in Rails 8)
• Ruby 3.1.6+
• A supported database:
  • PostgreSQL (9.5+ recommended)
  • MySQL (8.0+ recommended)
  • SQLite (for smaller applications)

The database recommendation is important as Solid Queue works best with MySQL 8+ or PostgreSQL 9.5+. This is because these versions support the FOR UPDATE SKIP LOCKED clause, which is key for efficient job handling.

Setting Up Solid Queue

Getting started with Solid Queue is simple:

1. Add the gem to your Gemfile :
   gem "solid_queue"
2. Install dependencies :
   bundle install
3. Run the migrations:
   bin/rails solid_queue:install:migrations
bin/rails db:migrate
4. Configure your application:
   #config/application.rb or appropriate environment file
config.active_job.queue_adapter = :solid_queue
5. Start the job processor :
   bin/rails solid_queue:work

However, If you want to switch gradually, you can always set the queue_adapter at the job level instead of a global configuration.

# app/jobs/my_job.rb
class MyJob < ApplicationJob
  self.queue_adapter = :solid_queue
  # ...
end

Database Configuration

Solid Queue can use your main application database. However, it’s often better to use a separate database for job processing in production. Here’s how to set this up in your database.yml:

production:
  primary: &primary_production
    <<: *default
    database: app_production
    username: app
    password: <%= ENV["APP_DATABASE_PASSWORD"] %>
  queue:
    <<: *primary_production
    database: app_production_queue
    migrations_paths: db/queue_migrate

Then, tell Solid Queue which database connection to use:

# config/application.rb or an initializer
config.solid_queue.connects_to = { database: { writing: :queue } }

This setup creates a separate database for your job queue while using the same connection details as your main database.

Database Architecture

Let us now take a look at the database changes for implementing Solid Queue. Understanding the data model helps you use it effectively. The system organises job processing through ten distinct database tables, each serving a specific purpose:

• solid_queue_jobs: Serves as the central repository for all job data.
• solid_queue_ready_executions: Contains jobs prepared for immediate execution.
• solid_queue_scheduled_executions: Stores future jobs with predetermined execution times.
• solid_queue_recurring_executions: Manages jobs designed to run repeatedly on a schedule.
• solid_queue_claimed_executions: Holds jobs currently undergoing processing.
• solid_queue_blocked_executions: Maintains jobs waiting due to concurrency limitations.
• solid_queue_failed_executions: Captures jobs that encountered errors, along with detailed failure information.
• solid_queue_semaphores: Facilitates concurrency control across job processing.
• solid_queue_pauses: Records queues that have been temporarily suspended.
• solid_queue_processes: Monitors and tracks all supervisor processes.

Solid Queue Job’s Life-cycle Through Database Tables

When working with Solid Queue, jobs follow a clear progression through these database tables:

1. Creation Phase: Initially, every job is registered in the solid_queue_jobs table.
2. Scheduling Phase: From there, the job takes one of two paths:
   • For immediate processing, it moves directly to the solid_queue_ready_executions table.
   • Alternatively, for delayed execution, it’s placed in the solid_queue_scheduled_executions table.
3. Preparation Phase: When the scheduled time arrives, the Dispatcher transfers jobs from scheduled_executions to ready_executions, making them available for processing.
4. Processing Phase: Next, Workers claim available jobs from ready_executions, temporarily moving them to claimed_executions during processing.
5. Completion or Exception Phase: After processing, one of several outcomes occurs:
   • Successfully completed jobs are simply removed from the tables.
   • However, if a job fails, it’s transferred to failed_executions along with comprehensive error details.
   • Furthermore, if concurrency rules prevent immediate execution, the job is directed to blocked_executions until resources become available.
6. Recurring Jobs: Meanwhile, the Scheduler continuously monitors recurring_executions, automatically creating new job instances according to their defined schedules.
7. Supervision: Throughout this entire process, the Supervisor oversees all operations, tracking process activity in the solid_queue_processes table.

This well-structured approach ensures reliable job processing while providing clear visibility into each job’s current status and history.

How Solid Queue Works Under the Hood

Solid Queue gets its impressive speed from a clever database technique called FOR UPDATE SKIP LOCKED. Here’s exactly how this process unfolds:

1. First, when a worker searches for jobs, it requests available tasks from the database using this special command.
2. Next, the database immediately marks these jobs as “in use” exclusively for this worker.
3. Meanwhile, other workers simultaneously looking for jobs will automatically skip these marked tasks and instead find different ones.
4. As a result, this system effectively prevents multiple workers from accidentally processing the same job, and consequently, nobody needs to wait.

This innovative approach is significantly better than traditional methods where workers would inevitably get stuck waiting when another process was already handling a job. Therefore, your application benefits from faster processing and ultimately makes better use of your server resources

The Solid Queue Architecture

Solid Queue strategically divides job processing among four essential components:

• Workers: Initially, these components process the actual jobs from the ready queue, forming the foundation of the system.
• Dispatchers: Additionally, these components continuously watch for scheduled jobs and promptly move them to the ready queue when it’s time to run them. Furthermore, they help effectively manage how many jobs run simultaneously.
• Scheduler: Moreover, this component handles recurring jobs, automatically creating new instances based on schedules you establish.
• Supervisor: Finally, this overarching component diligently oversees everything, constantly monitors the health of all processes, and quickly restarts any that stop working.

Together, these well-coordinated components create a remarkably reliable system that efficiently handles everything from immediate tasks to complex scheduled jobs. As a result, this thoughtful design provides both exceptional performance and complete visibility into your background job operations.

Default Configuration

Here’s a full set of configurations that you can change to fit your needs:

# config/initializers/solid_queue.rb
Rails.application.config.solid_queue.polling_interval = 1.second
Rails.application.config.solid_queue.poll_jitter = 0.15
Rails.application.config.solid_queue.connects_to = { database: { writing: :primary } }
Rails.application.config.solid_queue.max_batch_size = 500
Rails.application.config.solid_queue.concurrency_limit = nil
Rails.application.config.solid_queue.concurrency_maintenance_interval = 5.minutes
Rails.application.config.solid_queue.heartbeat_interval = 5.seconds
Rails.application.config.solid_queue.process_timeout = 10.minutes
Rails.application.config.solid_queue.shutdown_timeout = 25.seconds

For production environments, you’ll often want a more specific setup:

# config/solid_queue.yml
production:
  dispatchers:
    - polling_interval: 1
      batch_size: 500
      concurrency_maintenance_interval: 300
  workers:
    - queues: "*"
      threads: 3
      polling_interval: 2
    - queues: [real_time, background]
      threads: 5
      polling_interval: 0.1
      processes: 3
  scheduler:
    polling_interval: 60

This configuration does the following:

• A dispatcher checks for scheduled jobs every second.
• A general worker handles all queues with 3 threads.
• A special worker for high-priority queues has more threads and checks more often.
• A scheduler checks for recurring jobs every minute.

Solid Queue vs. Sidekiq: A Comparison

Many Rails developers are already familiar with Sidekiq. Therefore, it’s helpful to understand how Solid Queue compares to this popular solution:

Performance:

• First, Sidekiq uses Redis, which typically operates faster than SQL databases for queue operations.
• Consequently, Solid Queue might not match Sidekiq’s speed when handling extremely high volumes (specifically 10,000+ jobs per minute).
• However, for most common applications, you likely won’t notice any significant performance difference in practice.

Infrastructure:

• Most importantly, Solid Queue simply uses your existing database, so you don’t need to maintain any additional services.
• In contrast, Sidekiq requires Redis, which means you must set up and manage this extra component.

Features:

• Remarkably, Solid Queue includes numerous advanced features right out of the box.
• Meanwhile, Sidekiq reserves many of its advanced capabilities exclusively for paid Pro/Enterprise versions.

Data Integrity:

• Naturally, Solid Queue benefits from your database’s built-in transaction safety features.
• Similarly, Sidekiq can achieve good data integrity, but this requires additional configuration work.

Cost:

• Best of all, Solid Queue is completely free and open source with no hidden costs.
• On the other hand, Sidekiq charges for access to its advanced features through paid licenses.

Overall, for most typical applications, Solid Queue offers an excellent balance of useful features, straightforward setup, and solid performance. Furthermore, it achieves this without requiring extra infrastructure or costly license fees, making it an attractive choice for many Rails projects.

Monitoring with Mission Control

Managing background jobs effectively is crucial for any robust Rails application. Fortunately, there’s a handy gem that simplifies this process: mission_control-jobs. While Solid Queue doesn’t have its own dashboard, it works well with Rails Mission Control. This gives you a way to watch and manage your jobs. First and foremost, adding this gem to your Gemfile brings a dedicated Rails-based interface directly to your fingertip

gem "mission_control-jobs"

This adds a Rails-based interface that lets you:

• See the status of jobs across all queues.
• Track jobs that failed and their error messages.
• Pause and restart queues.
• Retry jobs that failed.
• See how well workers are performing.

Conclusion

Solid Queue represents a modern and compelling approach to handling background jobs within Rails applications. Indeed, it stands out as a strong alternative to more traditional Redis-based solutions. Its key strengths lie in its tight integration with the Rails ecosystem, its comprehensive set of features, and its leveraging of your existing database. Therefore, for many projects, Solid Queue presents itself as a very viable and sensible choice.

Looking ahead, with Rails 8 poised to make Solid Queue the default job backend, we can anticipate a growing adoption and the development of even more supporting tools around it. Now, for teams deeply invested in Redis and Sidekiq, an immediate switch might not be essential. However, for new projects or those seeking to streamline their infrastructure, Solid Queue is definitely worth serious consideration.

Have you had the opportunity to try Solid Queue in your own Rails applications? I’d be very interested to read about your experiences in the comments below!

Happy coding!

---

 

References

• rails/solid_queue – The official GitHub repository for Solid Queue.
• Solid Queue Job Lifecycle – A blog post detailing the lifecycle of jobs in Solid Queue.
• Solid Queue – A blog post with more details on UPDATE SKIP LOCKED.
• rails/mission_control-jobs – The official GitHub repository for Mission Control Jobs.