sandeepk

Today go through the commands to monitor processes and how to handle them

  • ps – It reports the snapshot of the current process
  • init- It the parent process of all the processes.
  • pstree – Same as ps but list the process in form of tree with more details.
  • top- List down all the process running, update the snapshot after a while.
  • Kill – it signals the process
    • INT – 2 -Interrupt, stop running
    • TERM – 15 – ask a process to exit gracefully
    • KILL – 9 – force the process to stop running
    • TSTP – 18 – request the process to stop temporarily
    • HUP – 1 – Hang up
  • nice – Every process run has priority and with nice we can control this priority, it ranges from +19(very nice) to -20(not very nice) decreased niceness higher the priority
  • renice- change the priority of the existing process

>> top
PID  User  PR  NI  VIRT     RES     SHR     S  %CPU %MEM Time+ Command
3911 user  20   0 2855988 206872 141304 S  72.2  2.6   7:15.09 Web Content                                                                       
31980 user  20   0 3703988 509176 188188 S  33.3  6.3  49:36.10 firefox                                                                           
 2839 user  20   0 2834092 191744 128268 S  27.8  2.4  16:13.39 Web Content    

>>ps
  PID TTY          TIME CMD
 2418 pts/2    00:00:00 zsh
 4318 pts/2    00:00:00 ps

>> pstree | less
systemd-+-NetworkManager-+-dhclient
        |                |-dnsmasq---dnsmasq
        |                |-{gdbus}
        |                `-{gmain}
        |-accounts-daemon-+-{gdbus}
        |                 `-{gmain}
        |-acpid
        |-agetty
        |-apache2---2*[apache2---26*[{apache2}]]
        |-at-spi-bus-laun-+-{dconf worker}
        |                 |-{gdbus}
        |                 `-{gmain}
...
>> nice -n 10 long-running-command &
>>renice 20 2984
>>renice 15 -u mike # changing  niceness for all process of mike user
>> kill -9 PID

#shell #dgplug #ilugc

Today was the day with the commands grep and sed.

  • grep – command used for the patter matching it have many useful options

    • -i: to make case-insensitive search
    • -r: search through the file in dire recursively
    • – l: print the name of the file with matching string
    • -c: print the counts of match
    • -n: numbers the matching lines in the output
    • -v: it's like not condition, print the reverse of the condition
  • sed – its read the input lines, run script on them, and writes them to stdout. This command is good for the string replacement and editing of the files.

Both these commands can be used with regex for the pattern matching.


>> grep -nv  ^# \| ^$  /etc/services |less
# will list all the lines from the file with do not start with *#* and ends with an empty line.

>>sed ``s/UNIX/LINUX` file.txt

# sed command will replace the occurrence of the *UNIX* word with the *LINUX*

#shellrun #dgplug #ilugc

Today was the day of basic File Management, Pipes, and Redirects. So let's jump to the command's

  • mkdir – create a directory for you, -p will create the parent directory if it does not exist.
  • rmdir – remove the empty directory
  • rm – remove the files and directory with -r.
  • pushd & popd – this one is the new command I came across, it let you save the previous command and you can pop that command with popd when you require. Dirs let you list down the directories you can pop back too.
  • file – tells you about the format of the file
  • ?, * – these are the wildcards which help in pattern matching, '*' for any number of character and whereas ? for only one character
  • | – called as pipe, it takes the output of one program and gives as an input to another program
  • Redirection (<, >) – < this indicates to file to read input from, > this indicates the file to write output to.
  • >> – Appends the output to the end of the file, If the file does not exist it creates a new one.
  • File Descriptors – Standard Input(0), Standard Output (1), Standard Error (2), make sure to check the example below to see how you can use them
  • xargs- it read a text and pass them as input to the following command
  • tee – is a combination of > and | and let you copy data from the input to the output or a file

Now let see these commands in action


>> mkdir -p chess/pieces/board # create an directory for you.
>> rmdir -p chess/pieces/board # will delete whole path if no other file or dir exist
>> pushd /media/USB # will let you save this path
... # any command you run b/w
>> popd # this will get you back to the pushed path
>> dirs # list all the dir path saved
~/bash-trail ~ / ~/Code/tranzact ~/bash-trail/program

>> file shopping_list 
shopping_list: ASCII text

>> whoami | rev # will reverse the output from the *whoami* output
keepdnas

>> last > last-login.txt # will save the output of login user to the file

>> wc < last-login.txt # will pass the text from the file as input to the *wc* command
>> program 2> file # will write the Standard error from the program to the file. **File Descriptor**

>> find /media/USB | xargs -l 3 rm -f  # this will pass files for USB dir and xargs will pass the 3 filenames at a time to remove them.

>> last | tee everyone.txt | grep bob > bob.txt
#  To save details of everyone’s logins and save Bob’s in files also.

#shellrun #dgplug #ilugc

Today run through the commands to process the Text streams from the shell.

  • less – command let you show less content from the file you are viewing.
  • sort – helps you sort the output, -f let you do sort case-insensitive and -n numerical sort.
  • cut – help to select the fields(-f)/character(-c)
  • fmt – format the output of the file, you can specify the width with -w
  • tac – similar to the cat command but in reverse
  • sed- this command use to process each line of file with a script

let see these command in action


>> less hello.txt
Hello World
THis is a text file
hello.txt (END)

>> cat shopping_list 
cucumber
bread
fish fingers

>> sort shopping_list         
bread
cucumber
fish fingers

>>date
Thu Jul  9 00:21:17 IST 2020

>>date | cut -d " " -f1
Thu

>>cat COPYING |  less  
The GNU General Public License is a free, copyleft license for
software and other kinds of works.

  The licenses for most software and other practical works are designed

>> cat COPYING |  less  | fmt -w 30
The GNU General Public
  License does not permit
  incorporating your program

>>cat copybump.py
#! /usr/bin/env python3

import datetime
import os
import re
import stat
import sys
...
if __name__ == '__main__':
    do_walk()

>> tac copybump.py
    do_walk()
if __name__ == '__main__':
...
import sys
import stat
import re
import os
import datetime

#! /usr/bin/env python3

>> sed -f spelling.sed < report.txt > corrected.txt # correct the spellling mistake in report.txt and output the correct text in corrected.txt

#shellrun #dgplug #ilugc

This post and the continuing post will be post/notes to share my journey of going through the shell to brush the commands which I forget and to learn some new ones.

So here one or more things I learned today. * !! – will show you the previous command. * ! String- show's the last command with the given string. * !$– will give the last argument of the previous command * !^ – will give you the first argument of the previous command * ^String^replacement- will replace the first occurrence of the * String* with the replacement string. * Ctrl + A – will get you to the start of the line. * Ctrl + E – will get you to the end of the line. * Ctrl + D – will delete the current character, even can close your shell session :). * For loop – yup we can write for loop to certain repetitive actions. * Locate – can help you to search for the file/s in the drive. * file – it not only help you with the file search just not based on the name but has many options to perform on the result and you can use regex for the file search also.

Now let's dive into some cool example

>> ls
...
>> clear
>>!! # will refer to the previous command
>> clear
>> !l # refers to the previous command start with a given string in our case `l`
>> ls
>> cd Documents
>> echo !$ # will refer to the *Documents* args from the previous command, same will be the case with *!^*  which refer the first args of the previous command
>> ls
>> echo !$
>> echo ls
>> echo Documents
>>for file in *; echo ${file}; done # try to run this in the shell to see the output

References

#shellrun #dgplug #ilugc

Git rebase is a very handy command to integrate changes from one branch to another, we can run git rebase in two modes manual and interactive. In manual all commit take from the current branch and applied over the head of the passed branch, but in case of the interactive rebase command you have more control over the option to what do with commits.

So to understand how rebase work, we take an example where we have a master branch with the following commits.

>>> git log
commit 8bfb8c19c3d7b795e9698a9818880d89ca3c214a
Author: Sandeep <sandeepchoudhary1507@gmail.com>
Date:   Sun Jun 14 01:06:07 2020 +0530

    New goals added

from this master branch, we create a new branch dev and do some changes/bug fixes.

>>> git checkout -b dev master
...
>>> git log
commit a05b6cd75e604df0f4434a574809a4fc14e4313e
Author: Sandeep <sandeepchoudhary1507@gmail.com>
Date:   Sun Jun 14 01:08:11 2020 +0530

    workaround bugs

commit 8bfb8c19c3d7b795e9698a9818880d89ca3c214a
Author: Sandeep <sandeepchoudhary1507@gmail.com>
Date:   Sun Jun 14 01:06:07 2020 +0530

    New goals added

but in between the other developer push changes in the master branch and to integrate that changes in your current branch you can use merge or rebase command, the rebase helps you maintain the liner history of your workflow.

>>> git checkout master
>>> git log
commit 3c5d6baf13aeac37d9efb1218bbf3240ec5c2a12
Author: Sandeep <sandeepchoudhary1507@gmail.com>
Date:   Sun Jun 14 01:07:29 2020 +0530

    new release added

commit 8bfb8c19c3d7b795e9698a9818880d89ca3c214a
Author: Sandeep <sandeepchoudhary1507@gmail.com>
Date:   Sun Jun 14 01:06:07 2020 +0530

    New goals added

so now to integrate new changes from master to your branch dev, without making the commit history complex, we can use rebase command, let's check out

>>> git checkout dev
>>> git rebase master
>>> git log
commit 184805896dd5684fc076b9bb9aa34eb3994251b1
Author: Sandeep <sandeepchoudhary1507@gmail.com>
Date:   Sun Jun 14 01:08:11 2020 +0530

    workaround bugs

commit 3c5d6baf13aeac37d9efb1218bbf3240ec5c2a12
Author: Sandeep <sandeepchoudhary1507@gmail.com>
Date:   Sun Jun 14 01:07:29 2020 +0530

    new release added

commit 8bfb8c19c3d7b795e9698a9818880d89ca3c214a
Author: Sandeep <sandeepchoudhary1507@gmail.com>
Date:   Sun Jun 14 01:06:07 2020 +0530

    New goals added

we can also run the rebase command in —interactive mode which gives us the option to edit/squash/... the commits

>>> git rebase -i master
pick 1848058 work around bugs

# Rebase 3c5d6ba..1848058 onto 3c5d6ba (1 command(s))
#
# Commands:
# p, pick = use commit
# r, reword = use commit, but edit the commit message
# e, edit = use commit, but stop for amending
# s, squash = use commit, but meld into previous commit
# f, fixup = like "squash", but discard this commit's log message
# x, exec = run command (the rest of the line) using shell
# d, drop = remove commit
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#
# Note that empty commits are commented out

One of the cool use of the git rebase command is that you can change the base of the branch from one branch to other by use of —onto option.

Let assume you create a branch featureA from master and then another branch featureB from featureA, to change the base of the featureB branch.

# git rebase --onto <newbase> <oldbase>
>>> git rebase --onto master featureA featureB

So this is all about the git rebase command, which can help you to keep your commit history clean and your current working branch commits sync with the master branch.

#git

In this post, we will talk about the standard python library decorators and one or more things about the decorators . If you haven't read the previous blog post about decorator, go check out that here. I will be waiting...

We will talk about functools.lru_cache Decorator from Python standard Library, where lru means Least Recently Used.

lru_cache as the name suggested, it saves the previous result of the function expression based on argument and uses that result if the same argument passed. To save expensive calculations.

functools.lru_cache(maxsize=128, typed=False)

maxsize means that numbers of cache result which can be cached, once the cache is full the older result is discarded. One should use maxsize value as a power of 2 for optimal performance.

type true means argument will be treated differently as int and float values as 1 and 1.0 are treated the same, but if type value is set to true it will be treated differently.

>> 1 == 1.0
>> True

lru_cache use dict to the save the argument as position and keyword-based so all the argument passed to the decorator should be hash-able.

Some point as notes to remember about the decorators

  • Decorators are executed when the module is loaded by Python and decorated function only executed if explicitly invoked.
  • Decorators have the power to return the entirely a different function.
  • We can also have a parameterized decorator as we have seen in the lru_cache decorator.
  • Stocked Decorators means when more then one decorator is applied to a function, then the order of execution, is from the decorator nearest to the function definition to outside. Let seen an example
@d2
@d1
def func:
    print('f')

func = d2(d1(func))

so that wrap from my side on the topic Decorators.

#python

Decorators attach additional responsibility to the object dynamically. A decorator takes other function as an argument which it processes and returns that function or any other callable object.

So how the decorator looks like

@clean_strings
def get_full_name(first_name, middle_name, last_name):
    return first_name + middle_name + last_name

In above code snippet we have a decorator clean_strings, this can also be written in this way.

def get_full_name(first_name, middle_name, last_name):
    return first_name + middle_name + last_name

get_full_name = clean_strings(get_full_name)

There are one two things we will talk about decorator after understanding.

  • Variable Scope
  • Closure

Variable Scope

In every language, the variable has a scope where they are accessible and where not. So here we will talk about the local scope and global scope lets jump right into the code to see

def show(a):
    print(a)
    print(b)
>>> show(10) 
10
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in show
NameError: name 'b' is not defined 

We got the error as b is not defined in the scope

b = 101
def show(a):
    print(a)
    print(b)
>>> show(10) 
10
101

here its work fine as b is defined in the global scope, which can be accessed from within the function. let see another code snippet.

b = 101
def show(a):
    print(a)
    print(b)
    b =190
>>> show(10) 
10
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in show
UnboundLocalError: local variable 'b' referenced before assignment

here we face error as code interpret b as local variable of the function which being accessed before declaring as it is defined in the scope of the function. To treat b as global variable despite the assignment in function we can use global declaration.

b = 101
def show(a):
    global b
    print(a)
    print(b)
    global b =190
>>> show(10) 
10
101
>>> b
190

Closure

Closure are the function which have access to the non global variables referenced in the body of function.

Closure image

Figure from Fluent Python Book, chapter 7

In Python3 nonlocal was introduced which allows assigning the variable inside the scope.

Consider an avg function to compute the mean of an ever-increasing series of values; for example, the average closing price of a commodity over its entire history. Every day a new price is added, and the average is computed taking into account all prices so far.

def avg_series():
    count = 0
    total = 0
    def averager():
        nonlocal count, total
        count += 1
        total += new_value
        return total / count

    return averager

The need of using nonlocal here is that if we don't, Python assumes that count and total are the local variable of averager method, which will break our logic.

Code example is taken from Fluent Python Book.

Now lets build a decorator that can logs the runtime for the function.

import time
def log_time(func):
    def clocked(*args):
        start_time = time.time()
        result = func(*args)
        elapsed_time = time.time() - start_time
        print("Elapsed time: {}".format(elapsed_time))
        return result
    return clocked    

There are also built-in decorators in Python Standard Library which we will discuss in the next blog post, so stay tuned till then cheers.

#python

Function are the code block which contain a logic to process on certain set of given input and return an output. Functions in Python are the First Class Object which basically means function as entity can be

  • Create at Run-time.
  • Passed as argument to function.
  • Return as result from the function.
  • Assigned to the variables.

Some functions are also called as Higher Order Function which means that a function which take other function as an argument or return a function as result. Example of higher order function in Python are Map, Filter, Sorted ...

Let see in Python function are classes or not and try to prove above all points to show in Python Function are First Class Object.

Creating function at run-time in console.

def add(x, y):
    return x+y

add(2, 4)
6
type(add)
<class 'function'>

Assigned to the variable

sum = add
sum
<function add at 0x7f2199555b70>
// notice above sum variable pointing to the add function.
sum(3, 4)
7

Passing function as argument.

list(map(add, range(5), range(5))) // here we pass *add* function as argument to the *map* function.
[0, 2, 4, 6, 8]

Returning function as result.

def factorial(x):
    if x < 1:
        return 1
    else:
        return x * factorial(x-1) // here we are returning a function

Above code snippets clearly show that function in Python are First Class Object.

#python