dgplug member blogs

Reader

Read the latest posts from dgplug member blogs.

from sandeepk

Today, we are going to see how we can use | operator in our python code to achieve clean code.

Here is the code where we have used map and filter for a specific operation.

In [1]: arr = [11, 12, 14, 15, 18]
In [2]: list(map(lambda x: x * 2, filter(lambda x: x%2 ==1, arr)))
Out[2]: [22, 30]

The same code with Pipes.

In [1]: from pipe import select, where
In [2]: arr = [11, 12, 14, 15, 18]
In [3]: list(arr | where (lambda x: x%2 ==1) | select(lambda x:x *2))
Out[3]: [22, 30]

Pipes passes the result of one function to another function, have inbuilt pipes method like select, where, tee, traverse.

Install Pipe

>> pip install pipe

traverse

Recursively unfold iterable:

In [12]: arr = [[1,2,3], [3,4,[56]]]
In [13]: list(arr | traverse)
Out[13]: [1, 2, 3, 3, 4, 56]

select()

An alias for map().

In [1]: arr = [11, 12, 14, 15, 18]
In [2]: list(filter(lambda x: x%2 ==1, arr))
Out[2]: [11, 15]

where()

Only yields the matching items of the given iterable:

In [1]: arr = [11, 12, 14, 15, 18]
In [2]: list(arr | where(lambda x: x % 2 == 0))
Out[2]: [12, 14, 18]

sort()

Like Python's built-in “sorted” primitive. Allows cmp (Python 2.x only), key, and reverse arguments. By default, sorts using the identity function as the key.

In [1]:  ''.join("python" | sort)
Out[1]:  'hnopty'

reverse

Like Python's built-in “reversed” primitive.

In [1]:  list([1, 2, 3] | reverse)
Out[1]:   [3, 2, 1]

strip

Like Python's strip-method for str.

In [1]:  '  abc   ' | strip
Out[1]:  'abc'

That's all for today, In this blog you have seen how to install the Pipe and use the Pipe to write clean and short code using inbuilt pipes, you can check more over here

Cheers!

#100DaysToOffload #Python #DGPLUG

 
Read more...

from sandeepk

My work required me to profile one of our Django applications, to help identify the point in the application which we can improve to reach our North Star. So, I thought it will be great to share my learning and tools, that I have used to get the job done.

What is Profiling?

Profiling is a measure of the time or memory consumption of the running program. This data further can be used for program optimization.

They are many tools/libraries out there which can be used to do the job. I have found these helpful.

Apache JMeter

It is open-source software, which is great to load and performance test web applications. It's easy to set up and can be configured for what we want in the result report.

Pyinstrument

Pyinstrument is a Python profiler that offers a Django middleware to record the profiling. The profiler generates profile data for every request. The PYINSTRUMENTPROFILEDIR contains a directory that stores the profile data, which is in HTML format. You can check how it works over here

Django Query Count

Django Query Count is a middleware that prints the number of database queries made during the request processing. There are two possible settings for this, which can be found here

Django Silk

Django Silk is middleware for intercepting Requests/Responses. We can profile a block of code or functions, either manually or dynamically. It also has a user interface for inspection and visualization

So, here are some of the tools which can be of great help in profiling your code and putting your effort in the right direction of optimization applications.

Cheers!

#100DaysToOffload #Python #DGPLUG

 
Read more...

from sandeepk

Nowadays, every project has a docker image, to ease out the local setup and deployment process. We can attach these running container to our editor and can do changes on the go. We will be talking about the VS Code editor and how to use it to attach to the running container.

First thing first, run your container image and open up the VS Code editor, you will require to install following plugin

  1. Remote-Container
  2. Remote Development

Now, after installing these plugins, follow the steps below.

  • Press the F1 key to open Command Palette and choose Remote-Containers: Attach to Running Container... VS Code Palette
  • Choose your running container from the list, and voilà.
  • After this step, a new VS Code editor will open, which will be connected to your code. VS Code bar
  • You need to re-install plugin for this again form marketplace.

After this, you are all set to playground the code.

Cheers!

#100DaysToOffload #VSCode #Containers #Docker

 
Read more...

from sandeepk

Today we are going to talk about the modes of image. The mode of image defines the depth and type of the pixel. These string values help you understand different information about the image. As of this writing, we have 11 standard modes.

  • 1 (1-bit pixels, black and white, stored with one pixel per byte)

  • L (8-bit pixels, black and white)

  • P (8-bit pixels, mapped to any other mode using a color palette)

  • RGB (3x8-bit pixels, true color)

  • RGBA (4x8-bit pixels, true color with transparency mask)

  • CMYK (4x8-bit pixels, color separation)

  • YCbCr (3x8-bit pixels, color video format)

  • LAB (3x8-bit pixels, the Lab color space)

  • HSV (3x8-bit pixels, Hue, Saturation, Value color space)

  • I (32-bit signed integer pixels)

  • F (32-bit floating point pixels)

So, 1-bit pixel range from 0-1 and 8-bit pixel range from 0-255. The common modes are RGB, RGBA, P mode. Image also consist of band, common band like RGB for red, green, blue also have an Alpha(A) transparency, mainly for PNG image.

We can also change these modes with the help of convert or creating new image with the help of pillow library. Let see the code for example

# here we convert RGBA image to RGB image and painting the Alpha 
# trasparency band to white color
image = Image.open("path/to/image")
image.mode # this will tell image mode
new_image = Image.new("RGB", image.size, (255, 255, 255))
new_image.paste(image, mask=image.split()[3])

That's all about the modes. There is also raw modes where you can create even your modes, but will save that for another blog post :)

Cheers!

#100DaysToOffload #Python #Pillow

 
Read more...

from sandeepk

The other day, I was working with images which need me to use image EXIF rotation to show in right orientation. Which leads me to read about EXIF, so here are my notes about the same.

What is EXIF Exchange image file format is a protocol whose initial definition was produced by Japan Electronic Industries Development Association(JEIDA). It stores the various meta information of the images taken by a digital camera, which is stored as tag and value. There are many tags but for my problem Orientation (rotation) tag is of interest. The orientation tag value can be from 1 to 8 which signifies different meanings according to the position of the camera while taking the image.

EXIF Meta information from GIMP Image metadata from GIMP tool

Different Rotation EXIF rotation helps the image viewer application to show the image in the right orientation if it's compatible with the EXIF metadata. Window users might have noticed that before Window 8 image shown is without rotation, but after Window 8 all images are shown in their right orientation because of the compatibility with EXIF.

Different Rotation Above image is taken from this blog

EXIF Orientation Tag Value Row Column
1 Top Left Side
3 Bottom Right Side
6 Right Side Top
8 Left Side Bottom

What Problem I have and how EXIF meta helpful So, the issue I am trying to solve is that we need to show the user's uploaded images that can have a different orientation, to fix this we need to rotate images which can be achieved at the server or the browser side. Working with Python makes it easy to handle the image with the help of Pillow library.

Image with rotation 3 Image with different rotation

from PIL import Image

rotation_dict = {3: 180, 6: 270, 8: 90}
EXIF_ORIENTATION_TAG = 274
image = Image.open(image)
exif_data = image._getexif()
if exif_data:
    rotation_degree = rotation_dict.get(exif_data.get(EXIF_ORIENTATION_TAG))
    if rotation_degree:
        image_file = image_file.rotate(rotation_degree)

This also can be achieved with CSS image-orientation which can be used with mostly all browser

/* keyword values */
image-orientation: none;
image-orientation: from-image; /* Use EXIF data from the image */

/* Global values */
image-orientation: inherit;
image-orientation: initial;
image-orientation: revert;
image-orientation: unset;

At last, I go with the CSS solution, which solves our use case with the least effort/code changes.

Cheers!

#100DaysToOffload #TIL

 
Read more...

from Pradhvan

PyLadies Berlin Logo

Source: https://www.meetup.com/PyLadies-Berlin/

PyLaides's Berlin chapter hosted a meetup on 20 November which seem very interesting to me when I first saw the agenda. It was on writing better talk proposals.

Though I have not attended many online meetups in the last two years I definitely didn't want to miss this one since I had a conference talk coming up early next year and this meetup just seemed like a really good opportunity to get some early feedback on the talk.

The meetup was sponsored by Ecosia.org and was hosted on GatherTown. The meetup agenda had only one talk and the rest was all an interactive session.

Once everyone settled in, Maria started his talk 'The Call for Proposals is open: Tips for writing a conference talk'. The talk resonated with me a lot because by the end Maria pointed out most folks don't submit a talk just because they think about not having mastery. I being a similar person who overthinks a lot of mastery could relate a lot.

Next was the Ideation round. It was the most fun and interactive part of the meetup where everyone pitched their talk idea(s) and after that, folks would give feedback (if any) on how the talk proposal can be improved.

I had initially joined the meetup to get some feedback on the talk that had already been selected for a conference. But later decided that I can utilize this chance to get some feedback on a completely new talk proposal.

So I presented a completely different talk idea 'Up and Flying with the Django Pony in 10 min' that I cooked up during the break.

Django Pony

Pony is Django's unofficial mascot. Source: https://djangopony.com/

I noticed Maria mention Python Pizza in her talk and saw they were accepting 10 min short talk. So that gave me chance to experiment and present something similar to Go in 100 Seconds.

Two other reasons that made me choose a talk on Django were

  • Most folks when starting out web development run behind learning the best or constantly switch without actually learning one well and knowing its limitations. But as I was rightly pointed out during the feedback, this could be a different talk altogether.

  • This talk could have been a really good starting point for someone who is thinking of picking up Django or moving to Django as it gives a good overview of its batteries included nature.

By the end of the Ideation round, there were only a couple of folks left. So it was decided that it really would help if everyone could write a documented proposal draft that everyone shared during the Ideation round and we could brainstorm on how can we improve it more.

The brainstorming session helped frame a better proposal and left me with a lot of valuable feedback out of which a few I could consider writing all the talk proposals in the future.

Jessica one of the PyLaides Berlin organizers did a wonderful job with the meetup keeping it interactive. While creating a safe space for everyone to share their views and feedback in a constructive manner.

I think other meetup groups can definitely take pages from PyLaides Berlin book in hosting a similar meetup where folks help out writing better talk proposals. As that would directly impact on quality of talk proposal which may also impact on the quality of talk conference gets. Who knows some of these proposals can end a being really good blogposts or meetup talks if not talks.

It might not be possible for organizers of an annual PyCon like PyCon India or other PyCon. But usually, a local chapter of Python User Groups can come up with a similar meetup in their region just after the CFP opens. Which can not only help out beginners but also help out folks improve on their talk proposal.

That's all for now. Until next time. Stay safe folks.

 
Read more...

from mrinalraj

Most probably, you have faced/ heard about the recent Facebook outage. Do you know Cisco Thousand Eyes pinpointed the issue – “Facebook Internal DNS unreachability”

How are Cisco Thousand Eyes able to do it? To explain broadly, it all happens with Agents known as Vintage Points.

  1. External Vantage Points: In these agents, are distributed globally like a public agent.
  2. Internal Vantage Points: Agents are monitoring the enterprise contribute to internal Vantage Points.
  3. End-User Experience: These agents understand the employees and customer experience.

So all these agents cover almost all possible sectors hence are responsible for accurately identifying the issue.

Why Cisco ThousandEyes? To explain the need for Cisco ThousandEyes there is a need to understand the current problem. In the past company relied only on its enterprise Syslog and SNMP protocols to identify and troubleshoot the issues. In this way, the reporting of issues became reactive rather than proactively informing the customer. i.e. customer first used to report the outage then the service provider identifies and troubleshoots it. But this gap is now solved using Cisco ThousandEyes by proactively identifying and troubleshooting before the issue is notifiable to the customer.

Isn't it exciting? Yes, it is.

Here are a few articles related to Cisco ThousandEyes I find useful 1. Link to Issue Reported 2. Link to Complete Analysis of the issue

Next Coming: How Cisco ThousandEyes helping in revolutionizing maintaining high uptime.

 
Read more...

from Pradhvan

Last weekend I attended EuroPython's virtual sprints. Though this was my second year of attending the virtual sprints I was still a bit overwhelmed. This year not only I would have been sprinting for other projects but I would be helping other folks contribute to ScanAPI. As I had been recently added to the core team, I thought it would be a good chance for me to dive deep into the codebase while helping others at the sprints.

Unlike last year where discord was the communication channel for the sprints, this year folks at EuroPython had a matrix server. Each sprint project had its channel integrated with Jitsi to help pair program or just have a hallway-like experience.

Similar to last year sprints were planned for the whole weekend with multiple opening/closing sessions to support different timezones. I liked that they kept this from last year because other folks from ScanAPI are from Brazil and to them, the first opening session which was at 9 AM CEST or 12:30 PM IST would be something like 4:30 in the morning. This gap between multiple openings also gave me some time to just visit other projects and contribute. At least that is what I thought initially and moved the sprints timings for ScanAPI to the second day.

On the first day of the sprints, I wasn't expecting that I had to be present at the opening since we had planned the sprints for the other day. But to my surprise got an invite for the opening session to present for ScanAPI as I was anyway online. So I mostly talked about what the project is, how to run the project locally and how you can pick up the issues in the opening.

After the opening since I did not see much activity in the ScanAPI channel right away, I hopped on to EuroPython's Website sprint channel where Raquel and Francesco(web lead for the project) was already present. I had a really fun time talking to them.

By the end of the day, ScanAPI had three new issues, 4 Pull Request that was to be reviewed, 2 got merged and one contribution to the wiki on how to set up the project on windows without WSL in it.

The second day was a bit slow in general where I had initially planned the sprints. Since we had very little activity on the second day, I did wrap up early after addressing the previous day's Pull Requests. Since there was only one person active on the channel post-lunch, the conversation at the end drifted to bread baking which to me seems a really fun thing to try in your free time.

In the end, it was a weekend well-spent.

 
Read more...

from Pradhvan

Making sense of docstrings with Python's ast and tokenize module and using recursion to rewrite Python's os.walk()

Last week was among the busy week at work with a release. Sadly personal goals suffered.

I did manage to put some effort into issues that were pending. Let's walk through each of them one at a time:

  • Lynn Root, maintainer of interrogate did give a detailed description of the issue. The approach to reaching the solution makes much more sense to me now. Based on the response wrote a simple script to check # nodocqa on docstrings i.e do not get coverage of the particular class or function. Need to check now how to implement that on interrogate.

  • On reading about Recursion realized, recursive algorithms are widely used to explore and manage the file system. This reminded me about Python's os.walk() method. Did spend some time reading the implementation though the code is well documented I am still not 100% sure how it works. I guess will learn more when I finish the script that mimics os.walk() in some manner.

That's all for the last week. Until next week. Stay safe folks.

#weeknotes2021

 
Read more...

from Pradhvan

Interrogating docstrings, Django's custom user model, Python's ast and tokenize module.

Oh boy! The week was more fun than I had planned.

Let's walk through each of them one at a time:

  • After finishing one of the courses on Django Celery. I wanted to put my knowledge to the test by building a knockoff version of Buttondown. Started a project called Hermes. Currently, Hermes has all the models done, celery setup is done and basic auth in place. Wanted to implement a custom user model so that the username field can be emailed instead of a username. So currently stuck with a bug related to that hopefully, will be resolved by next week.

  • I picked an issue in a library interrogate. interrogate gives a coverage report of missing docstrings and currently does not have a skip function like noqa. So I picked up the issue to add that. On a quick glance saw that ast module was being internally used. Though ast module doesn't include comments. The tokenize module can give you comments but doesn't provide other program structures. So I guess I need to mix and match both to add the feature.

  • Last year I sprinted for scanapi at EuroPython sprints 2020. I moved back to the project this week. Started back again by adding some docs, adding issue templates, and docstring coverage to the project. Also, this is how I stumbled upon interrogate.

I also managed to stay on course with my yearly health goal. Meditated daily and spent a decent 40min doing some cardio/core exercises on the working weekdays.

That's it for this week. Until next week. Stay safe folks.

#weeknotes2021

 
Read more...

from abbisk

Variable

Variable is a way to refer to memory locations being accessed by a computer program. In languages like C, C++, Java, a variable is a name given to a memory location. So when we say:

int x;
float y;

This small piece of code asks the compiler to create memory space for the variables, 4 bytes each (the size depends on the compiler and programming language). The variables are basically names assigned to the memory location, if the variable is defined for a particular data type then only declared type data can be stored in it. But in python, we do not need to declare a variable beforehand like int x, we can straightaway move to define the variable and start using it like x = 10. alt text

Now the point is, how does Python know what is the type of the variable and how to access it? In Python, data types (integer, list, string, functions, etc. ) are objects and every object has a unique id that is assigned to it, at the time of object creation. Variables are references (pointers) to these objects. What happens when we say :

x = 10
  • Python executes the right-hand side of the code
  • Identifies 10 to be an integer
  • Creates an object of Integer class
  • Assigns x to refer to the integer object 10 alt text

Now if we create new variable y which is equal to x, what do you think happens internally?

y=x

y now refers to the same object as x. alt text

How do we know that? It can be verified by checking the id of the object that these two variables x and y are pointing to.

id(y) == id(x)
# output
True

We can also print the id of the object

print(f"id of the object pointed by x:{id(x)}")
print(f"id of the object pointed by y:{id(y)}") 

# output
#your id value might be different from mine but the two ids printed must be the same
id of the object pointed by x:140733799282752 
id of the object pointed by y:140733799282752

Now, what if we assign a different value to y?

y=40

This will create another integer object with the value 40 and y will now refer to 40 instead of 10. alt text How to check Again, we will use id() to see the ids of the objects.

print(x) # outputs 10
print(y) # outputs 40
print(f"id of the object pointed by x:{id(x)}")
print(f"id of the object pointed by y:{id(y)}")  

# output
id of the object pointed by x:140733799282752
id of the object pointed by y:140733799283712

As we can see from above, the two ids are now different which means a new object with value 40 is created and y refers to that object now.

If we wish to assign some other value to variable x

x = "python"
print(id(x))

# output
1783541430128

We will get an object id that is different from the id “140733799282752” printed by id(x) for integer object 10 previously.

So what happened to the integer object 10? The integer object 10 now remains unreferenced and can no longer be accessed. Python has a smart garbage collection system that looks out for objects that are no longer accessible and reclaims the memory so that it can be used for some other purpose. alt text One last question: What if we create two integer objects with the same value?

p = 100
q = 100

Before I answer this question. Let's check their ids:

print(f"id of the object pointed by x:{id(p)}")
print(f"id of the object pointed by y:{id(q)}") 
# output
id of object pointed by x:140733799285632
id of object pointed by y:140733799285632

So what python does is optimize memory allocation by creating a single integer object with value 100 and then points both the variables p and q towards it.

Let's check one more example-

m = 350
n = 350

print(f"id of object pointed by x:{id(m)}")
print(f"id of object pointed by y:{id(n)}") 

# output
id of the object pointed by x:1783572881872
id of the object pointed by y:1783572881936

To our surprise the ids of the integer object 300 referred to by m and n are different. What is going on here?? Python allocates memory for integers in the range [-5, 256] at startup, i.e., integer objects for these values are created and ids are assigned. This process is also known as integer caching. So whenever an integer is referenced in this range, python variables point to the cached value of that object. That is why the object id for integer object 100 referenced by p and q print the same ids. For integers outside the range [-5, 256], their objects are created as and when they are defined during program implementation. This is the reason why the ids for integer object 350 referenced by m and n are different. As mentioned earlier, variables point to the objects stored in memory. Hence variables are free to point to object of any type.

x = 10 # x points to an object of 'int' type
x = ["python", 20, "apple"] # x now points to an object of type 'list'
x = "python" # x now points to a string type object

I hope you enjoyed reading it. Feel free to suggest improvements to the article or any other topic that you would like to know more about.

 
Read more...

from mrinalraj

According to wiki “ Cloud computing is the on-demand availability of computer system resources, especially data storage (cloud storage) and computing power, without direct active management by the user”

Benefits it has brought

With the benefits of Iaas(Infrastructure as a Service) , Paas(Product as a Service) we can now use the service on the go without any headache of maintenance of the facility.

The Difference

Differnces

 
Read more...

from mrinalraj

Today I am very much excited to share with you my first song video on the Dgplug platform via YouTube 'Zara Zara'.

Do you know what gives more satisfaction in life? Is it academics or getting an engineering degree? ;D Mostly it is Nurturing your skills and finding a platform to present yourself.

For me publishing my song gives me more satisfaction. In the end, when the time will come to retire it will not be the CGPAs but the risky path in chasing your passion and the crazy things we did with friends that will keep us smiling :)

 
Read more...

from mrinalraj

Yesterday on 1st October was the Foundation Day of The Scriptink. I always imagined how easy it seems to maintain the project made apart from the idea of any normal college projects. The audience always sees only the result and not the path. Coming to the point, Me being in Scriptink I came to know how hard is it to keep maintaining the consistency. U may be wondering, consistency in what way.

Definitely, You would be following our Scriptink through the app, youtube LinkedIn page. More than 2 years of consistency in posting the monthly short videos is a great achievement in itself. This is what the audience sees. But inside a team is working day and night to make this 2 min short video possible.

Not much to say, but Scriptink is all about us and not about I.

Click on this to experience the 2 years journey with us

 
Read more...

from mrinalraj

It was one of the best decisions to invest some time with DGPLUG summer training 2019 as I partially owe them my job.

During my virtual on-campus interview, I mentioned what I did during my summer vacation and bonus skills like blogging, helped me land an offer in a multinational company with a networking role.

I was asked what are my hobbies. Mostly everyone begins to answer like I love to sing and so on... I began explaining to them about my blogging and to prove them I briefly shared the story about Edward Snowden and the Internet's Own Boy, Aaron Swartz, which gave me an edge over other participants.

Once again, Thanks DGPLUG :)

Would love to connect with you.

 
Read more...

from Pradhvan

Last weekend I attended EuroPython sprints that were virtually conducted. The communication platform for the conference was discord and was kept the same for the sprints too. It served a good platform as we were able to pair program with the maintainer by sharing our screens.

Day 1

Sprints opened at 12:30 PM IST and started with its first round of project introduction. A total of 12 projects that took part in this year's sprint. Though the project maintainers were from varied timezone and timezones are difficult to handle. The first opening of sprints only had a few maintainers of the project to talk about their project.

The project that I started off in the day one of the sprints was terminusdb. I primarily contributed to terminudb's python client which had Cheuk Ting Ho and Kevin Chekov Feeney to help us out. Kevin had coded the JS Client of the project and was here to work on the Python Client.

The issue I picked up was increasing the test coverage of the project and while working on that issue I also discovered some other issues. Some depreciated function was still being used in the client and the make file did not have a command to generate coverage HTML of the project.

By the end of day one, I had moved the coverage of terminusdb_client/woqlclient/connectionConfig.py to 70% from 62% with a PR to remove the deprecated function from the client. Doing that I learned about graph databases and how terminusdb has git like features for the database.

Day 2

I started late on the second day and continued to work on the test coverage PR. I fixed some minor flake8 errors in my test coverage PR and pushed to coverage to 75% and created a PR for that make file command. A lot of people in sprints were confused in setup of project. So opened up a documentation issue for writing the wiki for setup instructions and contributions guidelines for new/first time contributors.

Just an hour before the first closing session I moved to scanapi which is maintained by Camila Maia. I picked up some good first issues and got them merged in no time. I saw this project at the closing of the day-1 and found it very interesting.

The other projects that I really found interesting but could not contribute to were Hypothesis, strawberry GraphQL and commitizen.

Overall I had a really fun weekend and I am excited to contribute more to those projects.

 
Read more...