The concept of background tasks is well-known in software engineering. They are used when there is an action that takes time to complete. The classic example would be sending your customer an email after a successful online purchase. If we do it straight away, it will block the user interface, and the customer will have to wait. It introduces a bad user experience. What you want to do instead is update the web page immediately.
In that case, sending an email can be done asynchronously. We schedule an email in the background and our user can interact with the web page again. Many popular web frameworks offer these capabilities out of the box or have a dedicated library like Celery for Python. It is easy to use them without thinking about the internals.
Yet many engineers face certain difficulties. The pitfalls below are common and take plenty of time to resolve. Many developers go through these challenges and reinvent the solution every time.
Let’s go through the most popular issues and make sure you avoid them next time.
Issue with Arguments
You have your background job ready. We have completed the logic and implemented unit tests. Everything looks good and the tests are green. However, when we launch the task in production it fails. What can you spot wrong in our simple implementation?
@app.task
def send_email_task(user: User, subject: String, text: String):
EmailService.send_email(user.email, subject, text)
I bet many engineers miss the problem. There is also a high chance that the issue is unspotted in the code review.
In reality, the issue is simple and makes many engineers constantly forget about it.
Did you pay attention to the attributes that we passed to our background task send_email_task? What do you think about the parameter user? It has a type User.
Without proper serialization of the parameter user the job will fail before even being started. The reason is that the job doesn’t know how to interpret the type User.
Long story short: background tasks accept parameters only of the basic types such as integer, float, and string. What would be the solution here?
There are a few things you can do.
The first one is to use only parameters of a simple type. We can pass the user’s email as a string instead of the class. Or we can pass the user’s ID as an integer and later retrieve the record from the database.
The second possible solution is to add object serialization. The class User should be serializable to a string so we pass it as a parameter. For example, representing the object user as a JSON string will work because the string is a valid argument type.
Progress Tracking
Launching a simple background job is usually a trivial assignment. The task is completed in five minutes. You can easily restart it if something goes wrong.
But what happens if the task needs to run for a few hours? Or days? Or weeks?
You can have a huge dataset that you want to iterate and update. Adding more computational resources is not always feasible. And we just cannot throw more resources pretending it solves the problem immediately.
Instead, we want to ensure that the task runs uninterrupted. We want to have an option to restart the task from the position where it stopped. In this scenario, progress tracking is needed. The configuration of the worker might not be enough.
Let’s look at the SQL database with the autoincrement ID column. We can use it to evaluate the total volume and actual progress. This information has to be persisted to make our task deterministic. So if the task must run on another worker it will know the previous state and continue executing.
In the Ruby world, there is a library job-iteration from Shopify. It allows us to achieve the desired behavior. Check out if there is such a library for your programming language.
How to Test
Testing background jobs is another challenge. And here is why.
How confident are you that your job does what is expected without any harm? If the job updates or deletes records from the database, how can we verify it operates on the right entries?
Unit tests can cover multiple cases. However, unit tests work only on the defined dataset. It can be only a few records you considered for a happy path scenario and possible errors.
The reality demonstrates that unit tests do not always cover all possible cases. Do you want to rely solely on unit tests while making irreversible changes to the sensitive data?
What we want instead is to have a test drive of the background job. Just to verify it works as expected. To achieve that we can tell the job to run in a test mode.
@app.task
def send_email_task(email: String, subject: String, text: String, dry_run: bool):
if dry_run:
logger.info(f"Dry run: send email to {email}")
else:
EmailService.send_email(user.email, subject, text)
Passing a parameter dry_run will allow us to execute our background task in a test mode without making any changes. We can collect some metrics, evaluate the performance of the function, and do load testing before manipulating the data.
Conclusion
Background asynchronous job is not a new approach in software engineering. It is a well-known concept and is being heavily utilized. Despite being that long, many developers still make mistakes and do not always follow best practices.
I collected the most common challenges with the background tasks in this article that software engineers face. Hopefully, you and your teammates will avoid them from now on.
Do you want to know how to grow as a software developer?
What are the essential principles of a successful engineer?
Are you curious about how to achieve the next level in your career?
My book Unlock the Code offers a comprehensive list of steps to boost
your professional life. Get your copy now!