Today, we are going to see how we can use | operator in our python code to achieve clean code.

Here is the code where we have used map and filter for a specific operation.

In [1]: arr = [11, 12, 14, 15, 18]
In [2]: list(map(lambda x: x * 2, filter(lambda x: x%2 ==1, arr)))
Out[2]: [22, 30]

The same code with Pipes.

In [1]: from pipe import select, where
In [2]: arr = [11, 12, 14, 15, 18]
In [3]: list(arr | where (lambda x: x%2 ==1) | select(lambda x:x *2))
Out[3]: [22, 30]

Pipes passes the result of one function to another function, have inbuilt pipes method like select, where, tee, traverse.

Install Pipe

>> pip install pipe

traverse

Recursively unfold iterable:

In [12]: arr = [[1,2,3], [3,4,[56]]]
In [13]: list(arr | traverse)
Out[13]: [1, 2, 3, 3, 4, 56]

select()

An alias for map().

In [1]: arr = [11, 12, 14, 15, 18]
In [2]: list(filter(lambda x: x%2 ==1, arr))
Out[2]: [11, 15]

where()

Only yields the matching items of the given iterable:

In [1]: arr = [11, 12, 14, 15, 18]
In [2]: list(arr | where(lambda x: x % 2 == 0))
Out[2]: [12, 14, 18]

sort()

Like Python's built-in “sorted” primitive. Allows cmp (Python 2.x only), key, and reverse arguments. By default, sorts using the identity function as the key.

In [1]:  ''.join("python" | sort)
Out[1]:  'hnopty'

reverse

Like Python's built-in “reversed” primitive.

In [1]:  list([1, 2, 3] | reverse)
Out[1]:   [3, 2, 1]

strip

Like Python's strip-method for str.

In [1]:  '  abc   ' | strip
Out[1]:  'abc'

That's all for today, In this blog you have seen how to install the Pipe and use the Pipe to write clean and short code using inbuilt pipes, you can check more over here

Cheers!

#100DaysToOffload #Python #DGPLUG

My work required me to profile one of our Django applications, to help identify the point in the application which we can improve to reach our North Star. So, I thought it will be great to share my learning and tools, that I have used to get the job done.

What is Profiling?

Profiling is a measure of the time or memory consumption of the running program. This data further can be used for program optimization.

They are many tools/libraries out there which can be used to do the job. I have found these helpful.

Apache JMeter

It is open-source software, which is great to load and performance test web applications. It's easy to set up and can be configured for what we want in the result report.

Pyinstrument

Pyinstrument is a Python profiler that offers a Django middleware to record the profiling. The profiler generates profile data for every request. The PYINSTRUMENTPROFILEDIR contains a directory that stores the profile data, which is in HTML format. You can check how it works over here

Django Query Count

Django Query Count is a middleware that prints the number of database queries made during the request processing. There are two possible settings for this, which can be found here

Django Silk

Django Silk is middleware for intercepting Requests/Responses. We can profile a block of code or functions, either manually or dynamically. It also has a user interface for inspection and visualization

So, here are some of the tools which can be of great help in profiling your code and putting your effort in the right direction of optimization applications.

Cheers!

#100DaysToOffload #Python #DGPLUG

A data class is a class containing data only, from Python3.7 we can define a data class with the help of decorator @dataclass, which build the class with the basic functionality like __init__ , __repr__, __eq__ and more special methods.

Let see how to define your data class with the decorator @dataclass

from dataclasses import dataclass

@dataclass
class Batch:
    sku: int
    name: str
    qty: int

>>> Batch(1, 'desk', 100)
Batch(sku=1, name='desk', qty=100)

We can also add the default value to the data class, which works exactly as if we do in the __init__ method of regular class. As you have noticed in the above we have defined the fields with type hints, which is kind of mandatory thing in the data class, if you do not do it will not take the field in your data class.

@dataclass
class Batch:
    sku: int = 1
    name: str = 'desk'
    qty: int = 100

# if you don't want to explicity type the fields you can use any
from typing import Any
class AnyBatch:
    sku: Any
    name: Any = 'desk'
    qty: Any = 100

If you want to define mutable default value in data class, it can be done with the help of default_factory and field. Field() is used to customize each field in data class, different parameter that can be passed to field are default_factory, compare, hash, init, you can check about them over here

from dataclasses import dataclass, field
from typing import List

@dataclass()
class Batch:
    sku: int
    name: str
    qty: int = 0
    creator: List[str] = field(default_factory=<function/mutable value>)

Immutable Data Class we can also define our data class as immutable by setting frozen=True, which basically means we cannot assign value to the fields after creation

@dataclass(frozen=True)
class Batch:
    sku: int
    name: str
    qty: int = 0

>>> b = Batch(12, 'desk', 100)
>>> b.qty 
100
>>> b.qty = 90
dataclasses.FrozenInstanceError: cannot assign to field 'qty'

Data class saves us from writing boilerplate code, help us to focus on logic, this new feature of Python3.7 is great, so what waiting for go and right some data classes.

Cheers!

#100DaysToOffload #Python #DataClass #TIL #DGPLUG

Inventory(Hosts) file contains the information of the Ansible nodes. We can group based on our use case like web servers, DB servers and uses these groups to execute commands on them. But you have to create a group in the inventory file, let see how to do that.

[webserver]
IP_ADDRESS_1
IP_ADDRESS_2
IP_ADDRESS_3

[webserverduck]
IP_ADDRESS_1

[dbserver]
IP_ADDRESS_1
IP_ADDRESS_2
IP_ADDRESS_3

[clientduck:children]
dbserver
webserverduck

You can even assign variable nodes or group wise in inventory file

[webserverduck:vars]
ansible_connection=ssh
ansible_user=myotheruser
gather_fact=no


[dbserver]
IP_ADDRESS_1 ansible_user=myuser
IP_ADDRESS_2
IP_ADDRESS_3

Cheers!

#100DaysToOffload #Ansible #dgplug

What is Ansible Ansible is a simple IT automation tool that make application and system easier to deploy and maintain. With the help of Ansible, you can automate task like network configuration, code deployment and cloud management. Ansible is an agent less system, which means that you don't need to install any software on the client system.

Inventory Inventory file is the one which contains the lists of the host servers on which Ansible works, it can be the IP Addresses, Domain name ...

Hosts It contains the IP addresses or Fully qualified domain name of the node servers which you want to configure.

Ansible.cfg This file contains the configuration for the Ansible, like the path of the Inventory file and values for the default variable in the Ansible

Ansible Nodes Ansible's nodes are the client system on which we want to execute, deploy or maintain the code(servers on which we want to work).

Ansible Engine Ansible engine/master/controller from which we manage all the server nodes.

Cheers!

#100DaysToOffload #Ansible #dgplug

Today go through the commands to monitor processes and how to handle them

• ps – It reports the snapshot of the current process
• init- It the parent process of all the processes.
• pstree – Same as ps but list the process in form of tree with more details.
• top- List down all the process running, update the snapshot after a while.
• Kill – it signals the process
  • INT – 2 -Interrupt, stop running
  • TERM – 15 – ask a process to exit gracefully
  • KILL – 9 – force the process to stop running
  • TSTP – 18 – request the process to stop temporarily
  • HUP – 1 – Hang up
• nice – Every process run has priority and with nice we can control this priority, it ranges from +19(very nice) to -20(not very nice) decreased niceness higher the priority
• renice- change the priority of the existing process

>> top
PID  User  PR  NI  VIRT     RES     SHR     S  %CPU %MEM Time+ Command
3911 user  20   0 2855988 206872 141304 S  72.2  2.6   7:15.09 Web Content                                                                       
31980 user  20   0 3703988 509176 188188 S  33.3  6.3  49:36.10 firefox                                                                           
 2839 user  20   0 2834092 191744 128268 S  27.8  2.4  16:13.39 Web Content    

>>ps
  PID TTY          TIME CMD
 2418 pts/2    00:00:00 zsh
 4318 pts/2    00:00:00 ps

>> pstree | less
systemd-+-NetworkManager-+-dhclient
        |                |-dnsmasq---dnsmasq
        |                |-{gdbus}
        |                `-{gmain}
        |-accounts-daemon-+-{gdbus}
        |                 `-{gmain}
        |-acpid
        |-agetty
        |-apache2---2*[apache2---26*[{apache2}]]
        |-at-spi-bus-laun-+-{dconf worker}
        |                 |-{gdbus}
        |                 `-{gmain}
...
>> nice -n 10 long-running-command &
>>renice 20 2984
>>renice 15 -u mike # changing  niceness for all process of mike user
>> kill -9 PID

#shell #dgplug #ilugc

Today was the day with the commands grep and sed.

• grep – command used for the patter matching it have many useful options

  • -i: to make case-insensitive search
  • -r: search through the file in dire recursively
  • – l: print the name of the file with matching string
  • -c: print the counts of match
  • -n: numbers the matching lines in the output
  • -v: it's like not condition, print the reverse of the condition

• sed – its read the input lines, run script on them, and writes them to stdout. This command is good for the string replacement and editing of the files.

Both these commands can be used with regex for the pattern matching.

>> grep -nv  ^# \| ^$  /etc/services |less
# will list all the lines from the file with do not start with *#* and ends with an empty line.

>>sed ``s/UNIX/LINUX` file.txt

# sed command will replace the occurrence of the *UNIX* word with the *LINUX*

#shellrun #dgplug #ilugc

Today was the day of basic File Management, Pipes, and Redirects. So let's jump to the command's

• mkdir – create a directory for you, -p will create the parent directory if it does not exist.
• rmdir – remove the empty directory
• rm – remove the files and directory with -r.
• pushd & popd – this one is the new command I came across, it let you save the previous command and you can pop that command with popd when you require. Dirs let you list down the directories you can pop back too.
• file – tells you about the format of the file
• ?, * – these are the wildcards which help in pattern matching, '*' for any number of character and whereas ? for only one character
• | – called as pipe, it takes the output of one program and gives as an input to another program
• Redirection (<, >) – < this indicates to file to read input from, > this indicates the file to write output to.
• >> – Appends the output to the end of the file, If the file does not exist it creates a new one.
• File Descriptors – Standard Input(0), Standard Output (1), Standard Error (2), make sure to check the example below to see how you can use them
• xargs- it read a text and pass them as input to the following command
• tee – is a combination of > and | and let you copy data from the input to the output or a file

Now let see these commands in action

>> mkdir -p chess/pieces/board # create an directory for you.
>> rmdir -p chess/pieces/board # will delete whole path if no other file or dir exist
>> pushd /media/USB # will let you save this path
... # any command you run b/w
>> popd # this will get you back to the pushed path
>> dirs # list all the dir path saved
~/bash-trail ~ / ~/Code/tranzact ~/bash-trail/program

>> file shopping_list 
shopping_list: ASCII text

>> whoami | rev # will reverse the output from the *whoami* output
keepdnas

>> last > last-login.txt # will save the output of login user to the file

>> wc < last-login.txt # will pass the text from the file as input to the *wc* command
>> program 2> file # will write the Standard error from the program to the file. **File Descriptor**

>> find /media/USB | xargs -l 3 rm -f  # this will pass files for USB dir and xargs will pass the 3 filenames at a time to remove them.

>> last | tee everyone.txt | grep bob > bob.txt
#  To save details of everyone’s logins and save Bob’s in files also.

#shellrun #dgplug #ilugc

Today run through the commands to process the Text streams from the shell.

• less – command let you show less content from the file you are viewing.
• sort – helps you sort the output, -f let you do sort case-insensitive and -n numerical sort.
• cut – help to select the fields(-f)/character(-c)
• fmt – format the output of the file, you can specify the width with -w
• tac – similar to the cat command but in reverse
• sed- this command use to process each line of file with a script

let see these command in action

>> less hello.txt
Hello World
THis is a text file
hello.txt (END)

>> cat shopping_list 
cucumber
bread
fish fingers

>> sort shopping_list         
bread
cucumber
fish fingers

>>date
Thu Jul  9 00:21:17 IST 2020

>>date | cut -d " " -f1
Thu

>>cat COPYING |  less  
The GNU General Public License is a free, copyleft license for
software and other kinds of works.

  The licenses for most software and other practical works are designed

>> cat COPYING |  less  | fmt -w 30
The GNU General Public
  License does not permit
  incorporating your program

>>cat copybump.py
#! /usr/bin/env python3

import datetime
import os
import re
import stat
import sys
...
if __name__ == '__main__':
    do_walk()

>> tac copybump.py
    do_walk()
if __name__ == '__main__':
...
import sys
import stat
import re
import os
import datetime

#! /usr/bin/env python3

>> sed -f spelling.sed < report.txt > corrected.txt # correct the spellling mistake in report.txt and output the correct text in corrected.txt

#shellrun #dgplug #ilugc

This post and the continuing post will be post/notes to share my journey of going through the shell to brush the commands which I forget and to learn some new ones.

So here one or more things I learned today. * !! – will show you the previous command. * ! String- show's the last command with the given string. * !$– will give the last argument of the previous command * !^ – will give you the first argument of the previous command * ^String^replacement- will replace the first occurrence of the * String* with the replacement string. * Ctrl + A – will get you to the start of the line. * Ctrl + E – will get you to the end of the line. * Ctrl + D – will delete the current character, even can close your shell session :). * For loop – yup we can write for loop to certain repetitive actions. * Locate – can help you to search for the file/s in the drive. * file – it not only help you with the file search just not based on the name but has many options to perform on the result and you can use regex for the file search also.

Now let's dive into some cool example

>> ls
...
>> clear
>>!! # will refer to the previous command
>> clear
>> !l # refers to the previous command start with a given string in our case `l`
>> ls
>> cd Documents
>> echo !$ # will refer to the *Documents* args from the previous command, same will be the case with *!^*  which refer the first args of the previous command
>> ls
>> echo !$
>> echo ls
>> echo Documents
>>for file in *; echo ${file}; done # try to run this in the shell to see the output

References

• http://www.linuxtraining.co.uk/download/new_linux_course_modules.pdf
• http://tldp.org/LDP/Bash-Beginners-Guide/Bash-Beginners-Guide.pdf

#shellrun #dgplug #ilugc