A Guide To Django Performance Tips For Optimization & Testing

In web-application development, performance optimization is an endless yet quintessential task. This not only helps in increasing the number of users but also helps in increasing the usability of the applications. Today, we will discuss Django Performance Tips.

Django is based on the Python Web Framework. Django is an easy to go solution for developing, releasing and maintaining a program. It is used by several big houses such as Google, Instagram, Youtube, NASA, etc.

Django provides middleware such as ConditionalGetMiddleware and GzipMiddleware which you can use to enhance the performance of the requests.

Django is based on MVT (Model-View-Template) architecture. Django’s data can be handled by “model”, whereas “view” helps to see how the website or web application would look to the user, and the template helps in UI/UX designing of the website by showing us the wide range of layouts which are present in Django.

Top-notch Python Web Development Company suggests that Django, along with python, provides you with high-performance web application but you can further optimize the Django Performance based web application by implementing these tips:

Django is utilized by 12% of developers, offering a versatile framework for building scalable web applications, tied with Spring Boot.

Caching: A Powerful Technique

Typically, if you are going to the same value repetitively, it might be a good idea to save it so that every time a client requests, we do not have to recompute it again and again.

This caching of data is a powerful technique that can provide you huge benefits while serving a high number of requests.

High Performance Django has a strong caching framework in which you can save the dynamic content so that you don’t have to compute the data again and again. It gives you granular options to either cache the whole page or some selected fragments or some specific output.

You can use the cached-property decorator on a function so that next time the function is called, you will get the saved value instead of recomputing the function again.

from django.core.cache import cache

 

@cache_page(60 * 15)  # Cache for 15 minutes

def my_view(request):

    # Expensive computations here

    return render(request, 'my_template.html', context)

Laziness: Avoids Computation

Laziness and Caching go hand-in-hand. As Caching prevents the computation of the values, again and again, Laziness helps you by executing a certain function only when the value is required.

Laziness avoids computation in the first place and it is done only when it is required. This helps since the value that was computed earlier might not be even used in the future and we will save this computation.

The operations in High Performance Django are lazy. For instance, QuerySets are lazy. QuerySets will not fetch data when you create, pass, or combine them. Only when the values in QuerySets are being used, those values will be fetched from the database.

One more way you can optimize the performance is by forcing delayed executions of fetching data from the database. Certain operations force the QuerySets to prematurely fetch the data. You should try to avoid such operations.

You can also add keep_lazy () decorator to the function. This decorator forces Django to evaluate the function only when the values from those are required. Typically you should use this decorator for functions that are computation extensive.

Database Optimization

One of the main factors slowing the performance of the web applications is the requests done to the database. So, the database operations and queries have to be molded for a function to work optimally. With SQL, you need to take care of certain aspect of database queries such as:

• Database triggers,
• stored views,
• functions,
• and stored procedures.

The database operation can be optimized if you fetch all the data required in one query. QuerySets are lazy and fetch only required data which may cause multiple fetch operations on the database as the function gets executed. You can see from the below code snippet that when we get the authors list again from the database, it fetches the data from the cache.

# Get authors of an entry, but avoid redundant database queries
entry = Entry.objects.get(id=1)

# Using .all() twice may perform redundant queries. Instead, use select_related for efficiency.
authors = entry.authors.all()  # First query to fetch authors
authors = entry.authors.all()  # This results in another query; avoid this

Another way to optimize the database is to get only the required fields. Typically, Django gets all the fields from the table irrespective of the size of the table.

But, if you require only a few fields and it brings all, then the effort of bringing the other irrelevant fields is wasted. This can be done by using defer ( ) and only( ).

Defer command tells what all fields should not be fetched whereas only () command specifies the field which should be fetched.

Also Read: MongoDB vs MySQL: Which One You Should Opt For?

HTTP Performance

Django provides middleware such as ConditionalGetMiddleware and GzipMiddleware which you can use to enhance the performance of the requests.

The ConditionalGetMiddleware enhances the performance by conditionally sending GET responses. It adds Etag and Last-Modifier header to the requests.

Next time it gets a request, if the header value is If-None-Match or If-Modified-Since, it will not process the GET request and send HttpResponseNotModified response.

The GzipMiddleware zips all the responses from the browsers that understand gzip compression. You should add this middleware once the response is completely formed and all headers are added.

Yet, GzipMiddleware is considered a security threat currently by the security experts. So, make sure you use it with caution.

Sessions: order to optimize performance

Cached sessions can be stored in order to optimize performance. The cached Session will help to bring down the number of times a system tries to reach the database. The cached Session will save the information in itself.

Django supports the session cached by using SessionMiddleware. You can add the middleware by adding “django.contrib.sessions.middleware.SessionMiddleware” in your default settings.py.

High Performance by default stores your session in the database, but you can change it to store the data either on the file system or in the cache.

You can set the SESSION_ENGINE field to “django.contrib.sessions.backends.cache” in order to save the sessions in the cache.

You can also save the sessions in cookies. The session saved in the cookies is encrypted using the SECRET_KEY and you should make sure attackers should not get access to SECRET_KEY otherwise they may read the session data and execute code remotely.

Code Optimization

It is important that you optimize your domain logic code so that it does not take a lot of time to execute.

There are three prominent ways to lift the efficiency of the code:

• Upgrading/Replacing the third-party packages
• Simplifying the code
• Restructuring the code

To upgrade or replace the third-party packages you can give a quick glance at the code and find the third-party packages that come with a lot of baggage for any simple function.

Then you can remove or replace them as per requirement. Otherwise, you can also use a Python profiler to do this task.

Another way is to simplify the code: In Django, the rest framework is really multifaceted and has a lot of innovative features. But for simpler purposes, you can write your own serializer and save your time.

Also, you can use python profiler for not only third-party packages but also your own code. You can restructure the part of the code which is eating a lot of CPU and memory in order to optimize the Django Performance.

Below is a very simple example where we are profiling regex call

import cProfile

import re

cProfile.run('re.compile("foo|bar")')

Once you run the above application, you get the output as below. It shows how many functions calls were executed and how many of them are primitive.

197 function calls (192 primitive calls) in 0.002 seconds

Ordered by: standard name

ncalls  tottime  percall  cumtime  percall filename:lineno(function)

     1    0.000    0.000    0.001    0.001 <string>:1(<module>)

     1    0.000    0.000    0.001    0.001 re.py:212(compile)

     1    0.000    0.000    0.001    0.001 re.py:268(_compile)

     1    0.000    0.000    0.000    0.000 sre_compile.py:172(_compile_charset)

     1    0.000    0.000    0.000    0.000 sre_compile.py:201(_optimize_charset)

     4    0.000    0.000    0.000    0.000 sre_compile.py:25(_identityfunction)

   3/1    0.000    0.000    0.000    0.000 sre_compile.py:33(_compile)

Cached Template Loading

The Cache Template loader of Django is another weapon that can be used for optimization. You can enable Cache Template Loader and it will save all of your time and bandwidth to bring the same templates now and then.

For instance, some templates are frequently used and if you enable Cache Template Loader, those templates will be saved.

You can always get these templates directly from the memory, saving you a round trip to the file system. This can help in significantly reducing the response time and making the application performance better. So, Django Performance Testing is important. 

# In settings.py, enable the cache template loader

TEMPLATES = [

    {

        'BACKEND': 'django.template.backends.django.DjangoTemplates',

        'DIRS': [],

        'APP_DIRS': True,

        'OPTIONS': {

            'loaders': [

                ('django.template.loaders.cached.Loader', [

                    'django.template.loaders.filesystem.Loader',

                    'django.template.loaders.app_directories.Loader',

                ]),

            ],

        },

    },

]

Specific Functions

Since QuerySets lazily loads the data, Django added two functions – select_related and prefetch_related() which can be used to execute the query and get better results.

select_related()

This function will return additional related objects based on foreign keys when the query is executed. It is a single more complex query, which if executed, gives foreign objects. If we had used traditional select objects, we might have to query the database again to get those objects.

select_related() is ideal for single-valued relationships, like ForeignKey.

prefetch_related()

This function is also similar to select_related() and prefetches the foreign objects. select_related() is limited to single-value relationships but prefetch_related() performs a separate lookup for each relationship and fetches more data.

prefetch_related() is better for multi-valued relationships, like ManyToMany or reverse ForeignKey relationships.

# Use select_related for ForeignKey relationships

books = Book.objects.select_related('author').all()

 

# Use prefetch_related for ManyToMany relationships

books = Book.objects.prefetch_related('libraries').all()

Upgradation Using Newer Version

Upgradation is obvious in development. It is always thought that the newer the better and efficient it is.

Yet, one should not always run behind the newer versions. When you try to get the latest version of Python or Django make sure you have your system optimized.

Also, you should not keep your hopes high from the latest versions as sometimes it can be the other way around. So, being more cautious while updating the newer version is the key.

Did you know? There are 54,228 live websites globally that are utilizing Django.

Make Use Of line_profiler

If you’re not sure which part of your code is causing performance bottlenecks, line_profiler is a great tool for profiling each line of Python code. It helps you identify the exact lines that take the most time to execute.

You can install line_profiler via pip:

pip install line_profiler

Using IPython Debugger with line_profiler:

To diagnose performance issues in a function, you can use IPython debugger (embed()) to set a breakpoint just before the function you want to profile. This will allow you to inspect how much time each line in the function takes to execute.

However, for profiling, it’s better to use the @profile decorator that comes with line_profiler to mark the functions you want to analyze.

Example:

Let’s say you have a function that retrieves books for each library. The current implementation might be causing performance issues because it performs multiple database queries for every library, potentially causing the N+1 query problem.

# Example of inefficient code with the N+1 query problem

def get_books_by_library_id():

    libraries = Library.objects.all()  # Query all libraries

    result = {}

 

    for library in libraries:

        # Query for books in each library - N+1 problem

        books_in_library = library.books.all()  # Query executed here for each library

        result[library.id] = list(books_in_library)

 

    return result

The above code executes one query to get all libraries and then performs a query for each library to retrieve its books. This can lead to excessive database queries (i.e., an N+1 query problem), which can slow down your application.

How to Profile the Code:

1. Add the @profile decorator: You can add the @profile decorator to the function you want to profile. This marks the function for line-level profiling.
   
2. Use IPython to start debugging:

from django.http import HttpResponse

from test_app.helpers import get_books_by_library_id

 

def books_by_library_id_view(request):

    # Start IPython debugger here

    from IPython import embed; embed()

 

    # Call the function that you want to profile

    books_by_library_id = get_books_by_library_id()

 

    return HttpResponse("Books data processed")

Now, when you run the application, it will hit the embed() breakpoint and drop you into an interactive IPython shell where you can run the profiler.

Turn On SQL Logging

Another way to find out the cause behind the delay of execution is by enabling SQL Logging.

By adding these lines of code in settings.py, you can print all the SQL queries executing in your code. This also helps in Django Performance Testing.

Enabling SQL logging gives you visibility into what’s happening under the hood and helps you identify issues such as:

• Slow queries: Queries that take too long to execute.
• Redundant queries: Multiple database hits that could be optimized.
• N+1 query problems: Multiple queries executed within loops, leading to performance bottlenecks.

By logging SQL queries, you can gain valuable insights into what’s affecting your Django app’s performance, allowing you to take action to optimize it.

# settings.py

 

LOGGING = {

    'version': 1,

    'filters': {

        'require_debug_true': {

            '()': 'django.utils.log.RequireDebugTrue',  # Ensures logging is only active in DEBUG mode

        }

    },

    'handlers': {

        'console': {

            'level': 'DEBUG',  # Set the logging level to DEBUG

            'filters': ['require_debug_true'],

            'class': 'logging.StreamHandler',  # Log output to the console

        }

    },

    'loggers': {

        'django.db.backends': {

            'level': 'DEBUG',  # Log all SQL queries at the DEBUG level

            'handlers': ['console'],  # Send logs to the console

        }

    }

}

Avoid Queries in Loops

There are a few instances where we execute a query to get some data and then again execute the query to get the related data.

Imagine you want to fetch all countries of the world, so you execute a query for it.  Later you realize you also need their capitals, so you would have to query again.

So in order to solve this problem of executing queries in the loop, Django introduced functions like select_related() and prefetch_related() in the QuerySets API.

It can be used to fetch all the related data in one trip to the database. As we can see from the below code snippet, it fetches the related also.

def get_books_by_library_id():

    libraries = Library.objects.all()

    result = {}

    for library in libraries:

        books_in_library = library.books.all()

        result[library.id] = list(books_in_library)

    return result

This creates a performance problem. Instead of this, use the following coding snippet:

def get_books_by_library_id_one_query():

    books = Book.objects.all()

    result = defaultdict(list)    

    for book in books:

        result[book.library_id].append(book)

    return result

This  will generate only 1 SQL query and it will reflect in the Django Performance, without a pinch of a doubt.!

Use Memcached Server

Django provides a very powerful and efficient cache solution called Memcached.  It was initially developed for livejournal.com and later it was open-sourced.

Few prominent websites like Facebook and Wikipedia use Memcached to reduce database operations and fetching data from the cache.

You have to set your backend to use Memcached by setting BACKEND to ‘django.core.cache.backends.memcached.MemcachedCache’.

Memcached runs as a daemon and it is allocated specific memory on the RAM, so all the operations performed by Memcached are directly interacting with memory and they never fetch data from database or filesystem,

Another major advantage of Memcached is it can run on multiple servers and treats all the cache on these servers as a single cache. As a result of this, duplicate values will not be saved.

However, you cannot completely rely on Memcached for data storage. This feature can only be used for short-term data storage.

Read also: Top 20 Python Based CMS That Every Developer Should Know

Make Use of Elasticsearch

Django has a very efficient paginator. It works well but as the length of pages increases, you might have to fetch a lot of data from the database and it slows down the applications.

An efficient way is to use Elasticsearch where you store your data in indexes and once the index is fetched, you can fetch the data in the database for that particular index. This helps us still have high performance when we have many pages.

Reduce the Number of Queries

One of the most overlooked factors when considering performance is the query execution speed and the way queries are written. Typically, when we develop applications, the amount of data present in the database is less and we are not able to see the performance hit.

But when your application is used by multiple users, the database can grow in size and the query execution can cause slowness. Also since we use Django ORM, which is a wrapper, we do not get a picture of how the query is being executed.

Django offers a tool – Debug Toolbar which has a SQL panel in which you can see the execution of queries and the time taken by each of them.