Open-sourcing our Node.js, MongoDB-backed background jobs processor

Piotr Jakubowski, Staff Software Engineer

February 19, 2026 8 min read

Over the past two years, we've used our MongoDB-backed library to run billions of Node.js background jobs. It continues to run ~40M jobs per week (and growing) with throughput topping around 850 jobs/second. We open-sourced the library for others to use.

The following is the story of why (and how) we built it.

Introduction

Background jobs let you offload heavy or time-consuming processing from the interactive HTTP code path to another process (assuming you don't need the background job result in your HTTP response). The libraries add a record of the job to a data store, where another process later picks it up and executes it. Background jobs let you do asynchronous processing when you don't want or need to roll out a full message broker system. Even if you have a message broker, jobs can ensure you send messages only when you should.

Database-backed background jobs are not a new concept. You may have stumbled upon Sidekiq, Celery, or Oban. And if your software engineering career was taking off before Snapchat and when Instagram was still Burbn, you may have even used Delayed::Job, which seems to have started it all.

Yet the Node.js ecosystem doesn't treat background jobs as highly esteemed as those in communities like Ruby, Elixir, or Python. We wanted something native to our persistence layers, Postgres and MongoDB. For Postgres, we built around pg-boss. However, we couldn't find a library meeting our MongoDB criteria.

Why do we want background jobs in MongoDB?

"Why not use a Redis-based solution with, for example, BullMQ?" That's a good question and a popular choice. There's a reason why Sidekiq took the Ruby world by storm back in 2012-2013 (just around the time when Facebook coughed up $1B for Instagram), with part of the way paved by Resque, to dethrone Delayed::Job. Building performant queues in Redis is much simpler than SQL (or NoSQL-but-kinda-almost-relational) databases. Or maybe it's just harder to shoot yourself in the foot.

Unfortunately, Redis-backed queues have a couple of drawbacks for our use case:

  1. There is yet more infrastructure you must maintain, keep track of, update, and not misuse. That said, Redis deployments are ubiquitous, and we already use it for caching.
  2. Atomicity. When scheduling a background job with other system changes, we want the job to get scheduled along with the other database changes. They should succeed or fail together. If you schedule a job before your database changes and the changes fail, you've scheduled a job for something that never happened. If you schedule it after and it fails to schedule, your job never runs.

The lack of atomicity was a deal breaker. We want eventual consistency and not perpetual inconsistency. The cleanest solution is to use transactions, but Redis doesn't support them. We couldn't find a MongoDB-based background jobs library, so we decided to roll up our sleeves and build one.

What do we care about?

A few things were essential for us to get right from the start.

Observability

Without tracing and observability, it's hard to verify that things work as intended. The library:

  1. Traces the jobs themselves. It doesn't matter how good your library is if client code breaks. We had to ensure we could trace job performance and tie the execution of those jobs to the enqueued context, all the way back to where they originated.
  2. Tracks internal library performance. We want to know whether there are any issues with the queries used to acquire jobs (described below).
  3. Tracks execution delay (the time between when the job gets scheduled and when it actually runs), the number of jobs awaiting execution, and the number of failures and retries for the general system health.

Performance

Many people on the team had experience working with database-based background job libraries. Often, they would have many niche features, but the moment you get hundreds of thousands of jobs in the queue, they would struggle to process anything.

Since we're supporting high-traffic use cases, we must make it resilient to spikes in queue size. That means simple, indexed queries that are fast and harder to mess up. No conditional branches in queries, no ORs, no in-memory sorting. That means a constrained feature set. For example, job priorities introduce unnecessary complications and aren't necessary. If something is critical, you may want to isolate it for separate consumption.

Trials and tribulations

We put the first implementation to the test once we got it up and running. The effort we put into observability paid off, as we could quickly learn whether our assumptions held up in practice. While we were right about a few things, we definitely hit a few bumps along the way.

Dealing with long queues

One problem that became apparent early on was that our solution wasn't very good at handling situations where a single queue suddenly received a massive influx of jobs. While acquiring jobs was fast, regardless of the number of pending jobs, the consumer would get hung up on processing all the jobs from a single queue, causing all other queues to wait patiently.

A hiccup in one part of the system affecting everything else is not a great situation. We fixed it with a fair queue consumer strategy. It keeps track of which queues have jobs ready to be processed and cycles through round-robin style, so every queue keeps moving, even if one queue resembles the Fifth Avenue Apple Store sidewalk on iPhone release day.

Minimizing execution delay

Our first implementation used a standard periodic polling mechanism to retrieve new jobs to work on. We didn't want to shorten the period and hammer the database with useless queries. We quickly found that the polling was introducing artificial delays. If a job landed right after a poll, it would sit idle until the next one. While it's not the end of the world, we didn't want to introduce unnecessary latency. If someone is waiting for a message with a one-time password, we don't want to keep them waiting longer than needed.

Fortunately, instead of playing the game of "what's the proper polling period," we used MongoDB change streams to react to new jobs almost instantly. The execution delay metric was handy to track this down.

Handling CPU-bound workloads

Node.js's single-threaded architecture can be frustrating at times. If someone writes code that hogs the CPU for a significant amount of time, everything else stops. There are worker_threads that we could leverage, but so far, it's not a big enough issue to address.

Juggling fast and slow jobs

Balancing fast, time-critical workloads with long-running ones was a challenge. If a queue with slow jobs had a spike, it could affect all other jobs, even with our fair consumption strategy. We could use a consumption strategy to prevent a single queue from clogging the entire process. So far, however, we've succeeded with the more straightforward approach of separating fast and slow queues in separate processes.

Giving back to the community

We've happily been running our MongoDB background jobs solution for about 2 years. We've seen it smoothly chug along with over 1M jobs pending in a queue, fail once or twice due to poor code in a job, and iteratively improve it along the way.

Now it's time to put it out into the wild so the community can benefit from our struggles. Check our Quick Start guide if you're using MongoDB and have a use case for background jobs. Hopefully, it treats you well, and when it doesn't, pull requests are welcome!