dgplug member blogs

Reader

Read the latest posts from dgplug member blogs.

from pradhvan

Making sense of docstrings with Python's ast and tokenize module and using recursion to rewrite Python's os.walk()

Last week was among the busy week at work with a release. Sadly personal goals suffered.

I did manage to put some effort into issues that were pending. Let's walk through each of them one at a time:

  • Lynn Root, maintainer of interrogate did give a detailed description of the issue. The approach to reaching the solution makes much more sense to me now. Based on the response wrote a simple script to check # nodocqa on docstrings i.e do not get coverage of the particular class or function. Need to check now how to implement that on interrogate.

  • On reading about Recursion realized, recursive algorithms are widely used to explore and manage the file system. This reminded me about Python's os.walk() method. Did spend some time reading the implementation though the code is well documented I am still not 100% sure how it works. I guess will learn more when I finish the script that mimics os.walk() in some manner.

That's all for the last week. Until next week. Stay safe folks.

#weeknotes2021

 
Read more...

from sandeepk

Able to sneak out sometime today to watch this talk by Brain Chesky. Overall the talk is good worth spending your time on.

Brian Chesky on Launching Airbnb and the Challenges of Scale

My Notes

  • Find > What inside you have that no one else have
  • If you are not passionate about the thing you are doing, you will quit when the hard days come or thing go south
  • First best thing is to watch these all talks, but the second-best thing you can do is go out and build things by yourself.
  • Never wait to learn things before starting, just start and you will learn during the process.

Cheers!

#100DaysToOffload

 
Read more...

from pradhvan

Interrogating docstrings, Django's custom user model, Python's ast and tokenize module.

Oh boy! The week was more fun than I had planned.

Let's walk through each of them one at a time:

  • After finishing one of the courses on Django Celery. I wanted to put my knowledge to the test by building a knockoff version of Buttondown. Started a project called Hermes. Currently, Hermes has all the models done, celery setup is done and basic auth in place. Wanted to implement a custom user model so that the username field can be emailed instead of a username. So currently stuck with a bug related to that hopefully, will be resolved by next week.

  • I picked an issue in a library interrogate. interrogate gives a coverage report of missing docstrings and currently does not have a skip function like noqa. So I picked up the issue to add that. On a quick glance saw that ast module was being internally used. Though ast module doesn't include comments. The tokenize module can give you comments but doesn't provide other program structures. So I guess I need to mix and match both to add the feature.

  • Last year I sprinted for scanapi at EuroPython sprints 2020. I moved back to the project this week. Started back again by adding some docs, adding issue templates, and docstring coverage to the project. Also, this is how I stumbled upon interrogate.

I also managed to stay on course with my yearly health goal. Meditated daily and spent a decent 40min doing some cardio/core exercises on the working weekdays.

That's it for this week. Until next week. Stay safe folks.

#weeknotes2021

 
Read more...

from abbisk

Variable

Variable is a way to refer to memory locations being accessed by a computer program. In languages like C, C++, Java, a variable is a name given to a memory location. So when we say:

int x;
float y;

This small piece of code asks the compiler to create memory space for the variables, 4 bytes each (the size depends on the compiler and programming language). The variables are basically names assigned to the memory location, if the variable is defined for a particular data type then only declared type data can be stored in it. But in python, we do not need to declare a variable beforehand like int x, we can straightaway move to define the variable and start using it like x = 10. alt text

Now the point is, how does Python know what is the type of the variable and how to access it? In Python, data types (integer, list, string, functions, etc. ) are objects and every object has a unique id that is assigned to it, at the time of object creation. Variables are references (pointers) to these objects. What happens when we say :

x = 10
  • Python executes the right-hand side of the code
  • Identifies 10 to be an integer
  • Creates an object of Integer class
  • Assigns x to refer to the integer object 10 alt text

Now if we create new variable y which is equal to x, what do you think happens internally?

y=x

y now refers to the same object as x. alt text

How do we know that? It can be verified by checking the id of the object that these two variables x and y are pointing to.

id(y) == id(x)
# output
True

We can also print the id of the object

print(f"id of the object pointed by x:{id(x)}")
print(f"id of the object pointed by y:{id(y)}") 

# output
#your id value might be different from mine but the two ids printed must be the same
id of the object pointed by x:140733799282752 
id of the object pointed by y:140733799282752

Now, what if we assign a different value to y?

y=40

This will create another integer object with the value 40 and y will now refer to 40 instead of 10. alt text How to check Again, we will use id() to see the ids of the objects.

print(x) # outputs 10
print(y) # outputs 40
print(f"id of the object pointed by x:{id(x)}")
print(f"id of the object pointed by y:{id(y)}")  

# output
id of the object pointed by x:140733799282752
id of the object pointed by y:140733799283712

As we can see from above, the two ids are now different which means a new object with value 40 is created and y refers to that object now.

If we wish to assign some other value to variable x

x = "python"
print(id(x))

# output
1783541430128

We will get an object id that is different from the id “140733799282752” printed by id(x) for integer object 10 previously.

So what happened to the integer object 10? The integer object 10 now remains unreferenced and can no longer be accessed. Python has a smart garbage collection system that looks out for objects that are no longer accessible and reclaims the memory so that it can be used for some other purpose. alt text One last question: What if we create two integer objects with the same value?

p = 100
q = 100

Before I answer this question. Let's check their ids:

print(f"id of the object pointed by x:{id(p)}")
print(f"id of the object pointed by y:{id(q)}") 
# output
id of object pointed by x:140733799285632
id of object pointed by y:140733799285632

So what python does is optimize memory allocation by creating a single integer object with value 100 and then points both the variables p and q towards it.

Let's check one more example-

m = 350
n = 350

print(f"id of object pointed by x:{id(m)}")
print(f"id of object pointed by y:{id(n)}") 

# output
id of the object pointed by x:1783572881872
id of the object pointed by y:1783572881936

To our surprise the ids of the integer object 300 referred to by m and n are different. What is going on here?? Python allocates memory for integers in the range [-5, 256] at startup, i.e., integer objects for these values are created and ids are assigned. This process is also known as integer caching. So whenever an integer is referenced in this range, python variables point to the cached value of that object. That is why the object id for integer object 100 referenced by p and q print the same ids. For integers outside the range [-5, 256], their objects are created as and when they are defined during program implementation. This is the reason why the ids for integer object 350 referenced by m and n are different. As mentioned earlier, variables point to the objects stored in memory. Hence variables are free to point to object of any type.

x = 10 # x points to an object of 'int' type
x = ["python", 20, "apple"] # x now points to an object of type 'list'
x = "python" # x now points to a string type object

I hope you enjoyed reading it. Feel free to suggest improvements to the article or any other topic that you would like to know more about.

 
Read more...

from sandeepk

I came to see this warning a few days back when I was seeing the git diff of a file in reaction to this I open the file and hit enter at the last line to my surprise the warning remains still the same.

What is a new line? The new line is usually \n, aka (CR or CRLF) at the end of the file. It's helpful to identify the last byte of the file.

Why it is good to have a new line at the end of the file?

Quoting from here

A source file that is not empty shall end in a new-line character, which shall not be immediately preceded by a backslash character. Since this is a “shall” clause, we must emit a diagnostic message for a violation of this rule.

So, it turns out that, according to POSIX, every text file should end with a \n, or “newline” (not “a new line”) character. This acts as the eol, or the “end of line” character. It is a line “terminator”.

It's helpful to identify the end of file.

How to automatically add one with your favorite editor?

In your favorite editor, you can add newline automatically like this.

  • Emacs : Add (setq require-final-newline t) to your .emacs or .emacs.d/init.el file.
  • VS Code: set “files.insertFinalNewline”: true

#100DaysToOffload #POSIX #Git #NoNewLine

 
Read more...

from sandeepk

It is an abstract machine that can be in exactly one of a finite number of states at any given time. A state machine is just a specification of how to handle events. It consists of a set of states, one of which is the current state. For each state, we list the events that are significant to that state. For each of those events, we define the new current state of the system.

Let see with an example

Devops:
Stable (DB down) --> Critical (DB up) --> Stable (N/w down) --> Warning (N/w up) --> Stable

FSM

Here we have defined the states Stable, Critical, Warning and events DB down, DB up, and N/W down. In this case, we have a stable state so whenever our database goes down or the network goes down the state changes to critical/warning. We can link some logic with these states to perform any action for transition.

This logic can help in designing a good program around a problem statement that will have less coupled code and easy to maintain. Cheers!

#100DaysToOffload

 
Read more...

from sandeepk

Git is a version control tool, that helps to track changes in the project. We will discuss a few useful commands which are handy to know while working with the git

Creating a branch with no commits on it

>>> git checkout --orphan <branch_name>
>>> git log
fatal: your current branch '<branch_name>' does not have any commits yet

Remove stale branches In your local, if you have branches that are removed in the remote then you can use the prune command to delete those branches in local also.

>>> git remote prune origin

To pick a commit from another branch If you want to pick a commit from other branches into your current branch you can use the cherry-pick command and -x for when recording the commit, append a line that says “(cherry picked from commit ...)” to the original commit message to indicate which commit this change was cherry-picked from.

>>> git cherry-pick -x <commit SHA1>

To view commits that are not pushed yet

>>> git log @{u}..

Run previous command In Git, we are also switching from one branch to the other. Typing the name, again and again, is tedious. You can use - with the Git command, to save your self from typing the name again

>>>  randomos git:(useless_dict) git checkout add_numbers
Switched to branch 'add_numbers'
Your branch is up-to-date with 'origin/add_numbers'.
>>>  randomos git:(add_numbers) git checkout -
Switched to branch 'useless_dict'
Your branch is up-to-date with 'origin/useless_dict'.

If you have any Git commands which you feel are worth sharing and can save some time. Please do share.

Cheers!

#100DaysToOffload #Git #VersionControl

 
Read more...

from sandeepk

This Sunday, I attended the workshop on Creating Python modules using Rust by Kushal Das. The workshop started at 0900 UTC 18th April on Twitch.

The GitHub repo we follow along with the workshop can be found over here. Learned many new things in the workshop where we go through

  • Hello World example
  • User passed arguments
  • Exception Handling
  • File I/O
  • System Interaction (Process, System Information)

Enjoyed my time while going through these examples and like the way Kushal has arranged all the examples branch wise.

Cheers!

Useful Links – https://github.com/kushaldas/workshops/issues/2https://kushaldas.in/

#100DaysToOffload #Python #Rust

 
Read more...

from mrinalraj

According to wiki “ Cloud computing is the on-demand availability of computer system resources, especially data storage (cloud storage) and computing power, without direct active management by the user”

Benefits it has brought

With the benefits of Iaas(Infrastructure as a Service) , Paas(Product as a Service) we can now use the service on the go without any headache of maintenance of the facility.

The Difference

Differnces

 
Read more...

from mrinalraj

Today I am very much excited to share with you my first song video on the Dgplug platform via YouTube 'Zara Zara'.

Do you know what gives more satisfaction in life? Is it academics or getting an engineering degree? ;D Mostly it is Nurturing your skills and finding a platform to present yourself.

For me publishing my song gives me more satisfaction. In the end, when the time will come to retire it will not be the CGPAs but the risky path in chasing your passion and the crazy things we did with friends that will keep us smiling :)

 
Read more...

from mrinalraj

Yesterday on 1st October was the Foundation Day of The Scriptink. I always imagined how easy it seems to maintain the project made apart from the idea of any normal college projects. The audience always sees only the result and not the path. Coming to the point, Me being in Scriptink I came to know how hard is it to keep maintaining the consistency. U may be wondering, consistency in what way.

Definitely, You would be following our Scriptink through the app, youtube LinkedIn page. More than 2 years of consistency in posting the monthly short videos is a great achievement in itself. This is what the audience sees. But inside a team is working day and night to make this 2 min short video possible.

Not much to say, but Scriptink is all about us and not about I.

Click on this to experience the 2 years journey with us

 
Read more...

from mrinalraj

It was one of the best decisions to invest some time with DGPLUG summer training 2019 as I partially owe them my job.

During my virtual on-campus interview, I mentioned what I did during my summer vacation and bonus skills like blogging, helped me land an offer in a multinational company with a networking role.

I was asked what are my hobbies. Mostly everyone begins to answer like I love to sing and so on... I began explaining to them about my blogging and to prove them I briefly shared the story about Edward Snowden and the Internet's Own Boy, Aaron Swartz, which gave me an edge over other participants.

Once again, Thanks DGPLUG :)

Would love to connect with you.

 
Read more...

from mrinalraj

Every year JPMorgan Chase & Co. conducts Code for Good Hackathon at a large scale to hire fresh graduates from the engineering college which every student keeps an eye on...

Luckily, I was one of the shortlisted candidates for the current Code for Good Hackathon 2020. This article is for my friends who are waiting for some insights on their upcoming CFG Hackathon.


Welcome

Welcome


Key Points:

  • The first day was mainly for jelling up with teammates. You need to express what you are good at and decide among teammates to come up on the same page.
  • The ** second-day**, I liked it the most. That was a workshop on technical skills like a web page and Git usage. Also, it's better if you have early knowledge of git and you should at least have an idea on how to solve the merge conflict.
  • NOW, the third day, THE D-Day, Basically the main Hackathon where you will Brainstorm + Code will be of only 24 hours.

Try to focus on developing the main requirement posed by the non-profit NGO's. Keep on updating about your work to your teammates and mentors so that they can help you if you get stuck on it for more time.


Schedule

Schedule for 3 days Hackathon


Skills to focus:

  • Communicate effectively with teammates and mentors. As we all the sailors with a common goal to reach the surface.
  • Knowledge of HTML, CSS, Bootstrap + any app-related knowledge is plus.
  • Knowledge of SQL and XAMP server.
  • Basic knowledge of git push, git pull, git merge, and ability to solve the merge conflict. It is advisable to try collaborating on a dummy file with your friend before entering into the hackathon.

Our Presentation on The Nudge Foundation: Click here for Presentation and Demo

Conclusion

Hackathon is not only for hacking into problems but also knowing new faces, sharing stories, and building memories.

The End is the new beginning


 
Read more...

from pradhvan

Last weekend I attended EuroPython sprints that were virtually conducted. The communication platform for the conference was discord and was kept the same for the sprints too. It served a good platform as we were able to pair program with the maintainer by sharing our screens.

Day 1

Sprints opened at 12:30 PM IST and started with its first round of project introduction. A total of 12 projects that took part in this year's sprint. Though the project maintainers were from varied timezone and timezones are difficult to handle. The first opening of sprints only had a few maintainers of the project to talk about their project.

The project that I started off in the day one of the sprints was terminusdb. I primarily contributed to terminudb's python client which had Cheuk Ting Ho and Kevin Chekov Feeney to help us out. Kevin had coded the JS Client of the project and was here to work on the Python Client.

The issue I picked up was increasing the test coverage of the project and while working on that issue I also discovered some other issues. Some depreciated function was still being used in the client and the make file did not have a command to generate coverage HTML of the project.

By the end of day one, I had moved the coverage of terminusdb_client/woqlclient/connectionConfig.py to 70% from 62% with a PR to remove the deprecated function from the client. Doing that I learned about graph databases and how terminusdb has git like features for the database.

Day 2

I started late on the second day and continued to work on the test coverage PR. I fixed some minor flake8 errors in my test coverage PR and pushed to coverage to 75% and created a PR for that make file command. A lot of people in sprints were confused in setup of project. So opened up a documentation issue for writing the wiki for setup instructions and contributions guidelines for new/first time contributors.

Just an hour before the first closing session I moved to scanapi which is maintained by Camila Maia. I picked up some good first issues and got them merged in no time. I saw this project at the closing of the day-1 and found it very interesting.

The other projects that I really found interesting but could not contribute to were Hypothesis, strawberry GraphQL and commitizen.

Overall I had a really fun weekend and I am excited to contribute more to those projects.

 
Read more...

from pradhvan

I recently stumbled across a very peculiar topic called Bit Manipulation. In most of my programming days, I haven't actually relied on the binary operation to get me the result, I know under the hood everything is converted into 0's and 1's but it was all abstraction to me.

The case was different here. While working with Bit Manipulation, I had to actually rely on arithmetic bit operations to get me to the result. So it became real interesting real soon.

Bitwise operators

Basic operation done on bits are done with bitwise operators. Since we primarily work on bits these operations are fast and are optimized to reduce time complexity.

The first three &, | and ~ are fairly straightforward. So I would briefly go over it.

&: if both bits are of equal size than & operator would compare each position and would return True/1 if input bits are True/1. Similarly for False/0.

    6       : 1 1 0
    5       : 1 0 1
            -------- &
              1 0 0

|: if both bits are of equal size than & operator would compare each position and would return True/1 if input bits differ. Similarly for False/0.

     5       : 1 0 0
     3       : 0 1 1
            --------  |
              1 1 1

~: Not operator just compliments the bit it gets. In fancy computer lingo it gives one’s complement of a number.

    5       : 1 0 1
            -------- ~
              0 1 0

Now coming to more interesting operators:

Operator Name
^ XOR
>> Right Shift
<< Left Shift
XOR

If two bits are of two equal-size ^ of both bits in the compared position would be 1 if compared bits are of different binary and would be 0 if bot the compared bits are the same.

    6       : 1 1 0
    5       : 1 0 1
            -------- ^
              0 1 1
  • XOR of a number with itself is 0

    x = "Any int number"
    (x ^ x) == 0
    
  • XOR of a number with 0 is number itself.

    (x ^ 0) == 0
    
  • Ordering in XOR does not matter, both will give the same output.

    output = (7 ^ 3) ^ (5 ^ 4 ^ 5) ^ (3 ^ 4)
    output = 7 ^ (3 ^ (5 ^ 4 ^ 5)) ^ (3 ^ 4)
    

While discussing Left Shift,<< and Right Shift, >> we will be talking about arithmetic shifts.

Left shift <<

  • Left shift shifts the binary digits by n, pads 0’s on the right.
  • Left shift is equivalent to multiplying the bit pattern with 2 power k( if we are shifting k bits )
1 << 1 = 2 = 1 * (2  ** 1) 
1 << 2 = 4 = 1 *(2  ** 2) 
1 << 3 = 8 = 1 * (2  ** 3)
1 << 4 = 16 = 1* (2  ** 4)
…
1 << n = 2n

Right shift >>

  • Shifts the binary digits by n, pads 0's on the left.
  • Right shift is equivalent to dividing the bit pattern with 2k ( if we are shifting k bits ).
4 >> 1 = 2
6 >> 1 = 3
5 >> 1 = 2
16 >> 4 = 1

Both Right shift and Left shift operators come real handy in masking.

Masking allows the user to check/change a particular bit at a particular position.

Some of the common functions associated with masking are:

Set Bit
  • The set bit method is generally used to SET a particular with 1.
  • To achieve this we would need to create a mask at the particular position where we want to SET
  • The mask can be created with the help of the << if the left shift operator.
def set_bit(x, position):
    mask = 1 << position
    return x | mask

set_bit(6,1)
  • In the above code snippet we are SETing the bit at 0th index.
    masking = 1 << 0 = 1 * (2 ** 0) 
    
    6       : 1 1 0
    1 << 0  : 0 0 1
            -------- |
              1 1 1
IS BIT SET
def is_bit_set(x, position):
    shifted = x >> position
    return shifted & 1
Clearing Bit
def clear_bit(x, position):
    mask = 1 << position
    return x & ~mask
Flip Bit
def flip_bit(x, position):
    mask = 1 << position
    return x ^ mask
Modify Bit
def modify_bit(x, position, state):
    """
    state is param that tells us to set a bit 
    or clear a bit
    """
    mask = 1 << position
    return (x & ~mask) | (-state & mask)

Observations

Bit manipulation can be used to solve problems that you are familiar with but necessarily don't know about. Here are some of my observations that I noted while using bit manipulation.

To check if the number is even
  • & ANDing the number with 1 gives 0 or 1 — 0 if it's even — 1 if it's odd
x = "Any int number here"
(x & 1) == 0

Practice Question

To check if the number is a power of two
  • If a number is x binary representation of (x-1) can be obtained by simply flipping all the bits to the right of rightmost 1 in x and also including the rightmost 1.
Let, x = 4 = (100)2
x - 1 = 3 = (011)2
Let, x = 6 = (110)2
x - 1 = 5 = (101)2
  • x & (x-1) will have all the bits equal to the x except for the rightmost 1 in x. In the given example below the values enclosed in || are the same for both the x and x-1 if x is not the power of 2.
  • If the number is neither zero nor a power of two, it will have 1 in more than one place.
Let, x = 6 = 1|1|0
(x- 1) = 5 = 1|0|1

Let,x = 16 = |1|0000
(x-1) = 15 = |0|1111

Let,x = 8 = |1|000
(x-1) = 7 = |0|111

Let,x = 23 = 1011|1|
(x-1) = 22 = 1011|0|
x = "Any int number here"
(x & x-1) == 0

There are a lot more things that can be done with just bits and are definitely not limited to the above observations. Try to find your own observations. Happy coding!

 
Read more...

from pradhvan

I recently finished reading Python Testing with Pytest by Brian Okken and I am glad I picked this up rather than jumping into the docs. It's definitely a good introduction for people who haven't had their share of testing a python codebase, let alone be with Pytest.

The book introduces a python CLI called Tasks and takes this as a base for writing all of its tests throughout the course of the book. Though eventually, the tests become more complex when you get into the latter half of the book.

The pros of the book are that it covers almost every section of the framework from fixtures, plugins, custom pytest configuration and even using pytest with tools like coverage and mock. But if you're someone like me who hasn't had his share of testing a python codebase you might find yourself with a bit of information overload at times.

I did find the book a bit of overwhelming on chapters like writing your own plugin, custom configuration and using pytest with Jenkins because these are the features that I wouldn't be using right out of the box. I would definitely be coming back to these chapters in the future if I need any of the features.

Overall the book is really well-written keeping in mind beginners who are just picking up pytest as their first testing framework and also for folks who are moving towards pytest from any other testing framework. Exercises at the back of every chapter make sure you also get some hands-on experience of writing tests.

Just a personal tip for anyone who is picking this up and has less experience with pytest. Feel free to skip chapters or skim chapters that aren't useful right out of the box. You can always come back to them when you need those features.

 
Read more...