In the previous blog post, we have discussed F() Expression, we will now explore more query expression in Django, to name few that we will discuss in this post are

  • Func() Expression
  • Subquery Expression
  • Aggregation () Expression

Func() Expression Func () Expression is the base of all the expressions and can be used to create your custom expression for the database level function.

# The table that we using for our query is the *Student* which keeps records of the students for the whole school.

from django.db.models import F, Func
student_obj = Student.objects.annotate(full_name=Func(F('first_name') + F('last_name'), function='UPPER')

# This will give a student object with a new field that is *full_name* of the student in upper case.

Subquery Expression Subquery are like nested condition in the query filter which helps you to make a complex query into a clean concise query. But you need to know the order of the sequence the query will be executed to use effectively. While using a Subquery you will also need to know about the OuterRef, which is like an F() Expression but points to the parent query value, let see both Subquery and OuterRef in action

# you are given a task to get the name of the student whose name starts with *S* and whose fees are due.

from django.db.models import OuterRef, Subquery
fee_objects = Fees.objects.filter(payment_due_gt=0)
student_obj = Student.objects.filter(name__startswith='S').filter(id__in=Subquery(fee_objects.values('student_id')))

# Get the lastest remarks for the students
remark = Remark.objects.filter(student_id=OuterRef('pk')).order_by('-created_at')
student_obj = Student.objects.annotate(newest_remark=Subquery(remark.values('remark_strl')[:1]))

Aggregation () Expression

Aggregation Expression is the Func Expression with GroupBy clause in the query filter.

# get the total student enrolled in the *Blind Faith* subject.

student_obj = Student.objects.filter(subject_name='blind_faith').annotate(total_count=Count('id'))

Note: All queries mentioned above in the code are not tested. So if you see any typo, a query that does not make sense, feel free to reach out to me at sandeepchoudhary1507[at]gmail[DOT]com.


#100DaysToOffload #django #python

What is the F() Expression? First let me explain to you what are Query Expressions are, these expressions let you use value or computation to be used in the update, create and filters, order by, annotation, aggregation. F() object represent the value of the model fields or annotated columns. It lets you help to not load the value of the field into the python memory rather directly handles in the Database query.

How to use the F() Expression? To use the F expression you have to import them from the from django.db.models import F and have to pass the name of the field or annotated column as argument, and it will return the value of the field from the database, without letting know the python any value. Let some example.

from django.db.models import F

# Documents is the table which have the details of the document submitted by user from the registrey portal for GYM membership

# We need update the count of the document submitted by the user with pk=10091

# without using F Expression

document = Documents.objects.get(user_id=10091)
document.document_counts += 1

# Using F expression
document = Documents.objects.get(user_id=10091)
document.document_counts = F('document_counts') + 1

Benefits of the F() Expression.

  • With the help of F expression we can make are query clean and concise.
    from django.db.models import F
  document = Documents.objects.get(user_id=10091)
  document.update(document_counts=F('document_counts') + 1)

   #Here we also have achieved some performance advantage
    #1. All the work is done at database level, rather than throwing the value from the database in the python memory to do the computation.
    #2. Save queries hit on the database.
  • F Expression can save you from the race condition. Consider a scenario where multiple user access your database and when bother user access the Document object for the user 10091, the count value is two, when user updates the value and save it and other user does the same the value will be saved as three not Four because when both user fetches the value its two.

  # user A fetch the document object, and value of document_counts is two.
  document = Documents.objects.get(user_id=10091)
  document.document_counts += 1
  # after the operation value of document_counts is three

  # Code running prallerly, User B also fetch the object, and value of document_counts is two.
  document = Documents.objects.get(user_id=10091)
  document.document_counts  += 1
  # after the operation value of document_counts is three

  # But actually value should be Four, but this is not the case using F expression here will save use from this race condition.
  • F Expression are persistent, which means the query persist after the save operation, so you have to use the refreshfromdb to avoid the persistence.
  document = Documents.objects.get(user_id=10091)
  document.document_counts = F('document_counts') + 1

  document.document_validation = 0

  # This will increase the value of *document_counts* by two rather then one as the query presists and calling save will trigger the increment again.

  • More example of F Expression in action with filter, annotate query.
from django.db.models import F

# annotation example
annotate_document = Document.objects.filter(created_by=F('created_by_first_name') + F('created_by_last_name')

# filter example
filter_document = Documents.objects.filter(total_documents_count__gt=F('valid_documents_count'))

That's all about the F Expression, it is pretty handy and helps you improve the performance, by reducing value loading in the memory and query hit optimization, but you have to keep an eye over the persistent issue, which I assume will not be the case if you are writing your code in structured way.


#100DaysToOffload #django #python

In this post, we will talk about the standard python library decorators and one or more things about the decorators . If you haven't read the previous blog post about decorator, go check out that here. I will be waiting...

We will talk about functools.lru_cache Decorator from Python standard Library, where lru means Least Recently Used.

lru_cache as the name suggested, it saves the previous result of the function expression based on argument and uses that result if the same argument passed. To save expensive calculations.

functools.lru_cache(maxsize=128, typed=False)

maxsize means that numbers of cache result which can be cached, once the cache is full the older result is discarded. One should use maxsize value as a power of 2 for optimal performance.

type true means argument will be treated differently as int and float values as 1 and 1.0 are treated the same, but if type value is set to true it will be treated differently.

>> 1 == 1.0
>> True

lru_cache use dict to the save the argument as position and keyword-based so all the argument passed to the decorator should be hash-able.

Some point as notes to remember about the decorators

  • Decorators are executed when the module is loaded by Python and decorated function only executed if explicitly invoked.
  • Decorators have the power to return the entirely a different function.
  • We can also have a parameterized decorator as we have seen in the lru_cache decorator.
  • Stocked Decorators means when more then one decorator is applied to a function, then the order of execution, is from the decorator nearest to the function definition to outside. Let seen an example
def func:

func = d2(d1(func))

so that wrap from my side on the topic Decorators.


Decorators attach additional responsibility to the object dynamically. A decorator takes other function as an argument which it processes and returns that function or any other callable object.

So how the decorator looks like

def get_full_name(first_name, middle_name, last_name):
    return first_name + middle_name + last_name

In above code snippet we have a decorator clean_strings, this can also be written in this way.

def get_full_name(first_name, middle_name, last_name):
    return first_name + middle_name + last_name

get_full_name = clean_strings(get_full_name)

There are one two things we will talk about decorator after understanding.

  • Variable Scope
  • Closure

Variable Scope

In every language, the variable has a scope where they are accessible and where not. So here we will talk about the local scope and global scope lets jump right into the code to see

def show(a):
>>> show(10) 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in show
NameError: name 'b' is not defined 

We got the error as b is not defined in the scope

b = 101
def show(a):
>>> show(10) 

here its work fine as b is defined in the global scope, which can be accessed from within the function. let see another code snippet.

b = 101
def show(a):
    b =190
>>> show(10) 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in show
UnboundLocalError: local variable 'b' referenced before assignment

here we face error as code interpret b as local variable of the function which being accessed before declaring as it is defined in the scope of the function. To treat b as global variable despite the assignment in function we can use global declaration.

b = 101
def show(a):
    global b
    global b =190
>>> show(10) 
>>> b


Closure are the function which have access to the non global variables referenced in the body of function.

Closure image

Figure from Fluent Python Book, chapter 7

In Python3 nonlocal was introduced which allows assigning the variable inside the scope.

Consider an avg function to compute the mean of an ever-increasing series of values; for example, the average closing price of a commodity over its entire history. Every day a new price is added, and the average is computed taking into account all prices so far.

def avg_series():
    count = 0
    total = 0
    def averager():
        nonlocal count, total
        count += 1
        total += new_value
        return total / count

    return averager

The need of using nonlocal here is that if we don't, Python assumes that count and total are the local variable of averager method, which will break our logic.

Code example is taken from Fluent Python Book.

Now lets build a decorator that can logs the runtime for the function.

import time
def log_time(func):
    def clocked(*args):
        start_time = time.time()
        result = func(*args)
        elapsed_time = time.time() - start_time
        print("Elapsed time: {}".format(elapsed_time))
        return result
    return clocked    

There are also built-in decorators in Python Standard Library which we will discuss in the next blog post, so stay tuned till then cheers.


Function are the code block which contain a logic to process on certain set of given input and return an output. Functions in Python are the First Class Object which basically means function as entity can be

  • Create at Run-time.
  • Passed as argument to function.
  • Return as result from the function.
  • Assigned to the variables.

Some functions are also called as Higher Order Function which means that a function which take other function as an argument or return a function as result. Example of higher order function in Python are Map, Filter, Sorted ...

Let see in Python function are classes or not and try to prove above all points to show in Python Function are First Class Object.

Creating function at run-time in console.

def add(x, y):
    return x+y

add(2, 4)
<class 'function'>

Assigned to the variable

sum = add
<function add at 0x7f2199555b70>
// notice above sum variable pointing to the add function.
sum(3, 4)

Passing function as argument.

list(map(add, range(5), range(5))) // here we pass *add* function as argument to the *map* function.
[0, 2, 4, 6, 8]

Returning function as result.

def factorial(x):
    if x < 1:
        return 1
        return x * factorial(x-1) // here we are returning a function

Above code snippets clearly show that function in Python are First Class Object.