Job Queue

A queueing system to schedule jobs and have multiple rate-limited consumers.

API

Setting up a Job Queue

jobQueue = new JobQueue consumers, limit, period

Parameter	Type	Description
consumers	Function	A function that will accept a job as the only parameter.
limit	Integer	The maximum number of jobs the consumer should process in period milliseconds.
period	Integer	The number of milliseconds limit applies to.

Adding Consumers

jobQueue.addConsumers consumers, limit, period

Function signature for addConsumers is same as the above constructor.

Enqueuing a Job

jobQueue.enqueue job

Parameter	Type	Description
job	Anything	The job can either be any type (Object, Function, Number, ...). Its type depends on what the consumer takes as its argument.

Getting Count of Pending Jobs

pendingJobs = jobQueue.pendingJobs

Example Usage

JobQueue = require "job-queue"

makeConsumer = (consumerId) ->
    (job) ->
        console.log "Consumer #{consumerId} processing job #{job.id}"
        job.process consumerId

# Create a process queue with 5 consumers where each supports up to 5 requests per second
processQueue = new JobQueue [1..5].map(makeConsumer), 5, 1000

# Add another 5 consumers to the process queue where each supports up to 80 jobs per minute
processQueue.addConsumers [6..10].map(makeConsumer), 80, 60 * 1000

# Adding 10k jobs to the process queue
for jobId in [1..10000] then do (jobId) ->
    processQueue.enqueue
        id: jobId
        process: (consumerId) ->
            # The rate limited section (e.g. some API call)
            console.log "Job #{jobId} processed by consumer #{consumerId}"