Background Processing With Resque

Mike Cochran

@vongrippen



http://vongrippen.com/memrb-2013-04-resque

Quick Info on Resque

  • Built by Github
    • https://github.com/blog/542-introducing-resque
  • Uses Redis
    • Fast key-value store
  • Pure Ruby, Rails not required
  • Named queues
  • Queued data is pure JSON
  • Simple, easy to use
    • Create a class with a "perform" method.
      That's pretty much it.
  • Uses process forks for workers
    • Avoids the GIL in MRI Ruby
  • Bundled Web UI



the Problem

Simulating an investment portfolio

(Average Case)

  • Invest for 30 years
  • Rebalance investments after each year
  • Simulate 205 times, incrementing the starting quarter
    • Start in Dec 1932, invest for 30 years
    • Start in Mar 1933, invest for 30 years
    • Start in Jun 1933, invest for 30 years
    • ... etc
  • Save each simulation (called a "period")
  • Collect the summary of each as a "Case Result"

Without Resque

Controller
                def create
  client_case.input = filtered_params
  client_case.calculate
  # 3 minutes later...
  client.save
  # Render HTML
end



The result?

Timeout Error

Here's the Slow down

Model 
                    def calculate
  periods = 0..(periods_to_run-1)
  summaries = [ ]
  periods.each do |period|
    # Do a bunch of fun math
    years_in_period = calculate_period(input, period)
    self.save_period(period, years_in_period)
    summaries << years_in_period.last
  end
  self.summary = summarize(summaries)
end



Let's Add Resque

With Resque

Controller
                    def create
  client_case.input = filtered_params
  client_case.save
  Resque.enqueue(CaseWorker, client_case.id)
  # Render HTML
end
CaseWorker
                class CaseWorker
  @queue = :cases

  def self.perform(case_id)
    client_case = Case.find(case_id)
    client_case.calculate
    client_case.save
  end
end



The result?

No Timeout!
(But it still takes 3 minutes to calculate)

Here's the Slow down (Still)

Model 
                    def calculate
  periods = 0..(periods_to_run-1)
  summaries = [ ]
  periods.each do |period|
    # Do a bunch of fun math
    years_in_period = calculate_period(input, period)
    self.save_period(period, years_in_period)
    summaries << years_in_period.last
  end
  self.summary = summarize(summaries)
end



Let's Add (More) Resque

With (More) Resque

Controller
                        def create
  client_case.input = filtered_params
  client_case.save
  (0..client_case.periods_to_run-1).each do |period|
    Resque.enqueue(PeriodWorker, case_id, period)
  end
  # Render HTML
end
CaseWorker
                    class CaseWorker
  @queue = :cases
  def self.perform(case_id)
    client_case = Case.find(case_id)
    client_case.summary = summarize(client_case.summaries)
    client_case.save
  end
end

With (More) Resque

PeriodWorker
                    class PeriodWorker
  @queue = :periods

  def self.perform(case_id, period)
    client_case = Case.find(case_id)
    years_in_period = calculate_period(input, period)
    client_case.save_period(period, years_in_period)
    client_case.summaries << years_in_period.last
    client_case.save
    if client_case.periods == client_case.periods_in_case
      Resque.enqueue(CaseWorker, case_id)
    end
  end 
end




The result?

No Timeout!
Plus, I can now throw more machines at it.
With 30 "workers" running, 25 seconds.



What did we just do exactly?




  • We separated the calculation from our web app itself
  • We broke down our code into smaller bits
  • We ensured that our code could be run in parallel on multiple machines



Workers run

Anywhere,
Anytime



Run as many
as you want



Q & A

Resources


Project:
https://github.com/resque/resque

Screencast:
http://railscasts.com/episodes/271-resque

Sidekiq

sidekiq.org

  • Uses threads instead of forks
  • Easily extensible
  • Has a lot of extra functionality out of the box
  • Only loads a single copy of Rails in memory (if using Rails)
  • Smaller footprint

CAVEATS

  • You code must be thread-safe
  • Any gem you use must be thread-safe
  • More processor intensive