Pradhvan

Python geek who loves to play around with web technologies.

For the last couple of weeks I have been actively job hunting and in one of the place I interviewed I was asked if I can do take-home assignment in Go. Since I had already mentioned in the interview I had been learning Go on the side. I agreeded finishing the assignment in Go thinking this might be a good challenge to take on and by doing this I might have a chance to showcase my ability of picking up new things quickly.

Plus having a deadline to complete the take-home assignment in a week could also accelerate my Go learning, as it provides a time based goal to work towards.

Gopher Coffee Gif

So for a week I spend most of my time reading parts of Learning Go, refering to tutorials Learn Go with Tests, watching old episodes from justforfunc: Programming in Go, a lot of stackoverflowing and talking to friends who use Go as primary language at work to get some pointers(no pun intended) on best practices, blogs to follow etc.

After a week, I finished writing gogrep, a command line program written in Go that implements Unix grep like functionality.

gogrep has a fairly small subset of features it can search a pattern from a file or a directory and also has support for flags like

  • -i : Make the seach case sensitive.
  • -c : Count number of matches.
  • -o : Store the search results in a file.
  • -B : Print 'n' lines before the match.

While adding these features I also added a fair bit of tests to the project to see how tests are written in Go.

At the moment the test coverage of the entire project is around 72.0% where all the major packages that server as helper or have buisness logic have a coverage greater than 83% and a very few have 100%.

Since gogrep is my first Go project, I do have some loose opinions about the language and the tools it offer. Primarly from the prespective of Pythonista, who has been using Python as a primarly language for the last four years.

  • Go does not have classes and it's probably good: If you are coming from a background of classes and methods, you can relate some of it with receiver functions. Receiver functions come very close to behaving like methods for a class. Having written a few receiver functions to for a struct the straight forward use case becomes a no brainer but it might get a bit complicated once you start refactoring a code the second or the third time.

  • Poniter! Oh my! : If you have every done pointer airthmetic in C/C++ you know the pain. Thank god in Go you don't have to deal with pointer airthmetic. You still have to be careful since you are dealing with memory addresses and it can get complicated but it's defenitely not that bad. You get use to it.

  • Tooling: I coded the entire project in VSCode and I loved the Go support it has, go-static check suggestion for code snippetes are quite helpful and for tests automatically changing a color of the test file to red at that instant when the associated function is changed, the whole Developer Experience is amazing. I have heard even better things for GoLand by Jetbrains from folks who write Go daily, can't wait to try that out. Besides that gofmt, golint, go vet and go test were some other things that I found to be really handy. I did not play around much with the debugger so can't comment much on that.

  • Makefile to the resuce: Since go gives you a single binary in the end you have build it every time to check you code changes. Having some automated way like a Makefile makes things really easy. So I would suggest during your initial project setup do invest in making a good automated process for building and checking your changes.

  • Error handeling pattern: Go has this pattern for handling errors which is very straight forward but sometimes it might seem a bit too much in terms of repetitve code.

if err != nil {
// Do something here to log/return error
}

This seems to be the only way to handle error at least that I know of. So most of my code was be sprinkled with a lot of these statements if err != nil in the top layer of the abstraction, for gogrep it was the main.go when I was calling func ParseFlags() for processing arguments.

Like

// main.go

conf, output, err := parseflag.ParseFlags(os.Args[0], os.Args[1:])

if err == flag.ErrHelp {
    // Print the usage of the CLI
    fmt.Println(output)
    os.Exit(1)
} else if err != nil {
    fmt.Println("Error: \n", err)
    os.Exit(1)
}
//parseflag.go

func ParseFlags(searchWord string, args []string) (config *Config, output string, err error) {
     
 // Supressed code 	
err = flags.Parse(args)
if err != nil {
    return nil, buf.String(), err
}

 // Supressed code
if len(flags.Args()) == 0 {
    return nil, "", fmt.Errorf("missing argument searchword and filename")
}

 // Supressed code
return &conf, buf.String(), nil
}
  • Types magic: Since Go is statically typed that means when defenining the variable I have an option to either just declare it with a type and have a zero value in it or directly initializing it with some value.

Even with functions I had to write a signature of input types and output.

func ParseFlags(searchWord string, args []string) (config *Config, output string, err error)

Having types in the function signature and output became such a great advantage later on both developer experience when refactoring and re-reding code that I wrote days back.

I became a fan of types in the codebase to such an extent that now I am all in for type annotations in my Python code if they can give me similar advantages.

  • Things I skipped: One week is a very small time to peak into a language features. So obviously I skipped a lot of Go features too like interfaces(I might have used them like io reader but for sure haven't written them), generics and defenitely did not touch goroutines and channels. One thing that most of the people praise Go for. There might be more but from a birds eye view I can not think anything else.

Overall, it was really fun week of learning something new and not using something that is already in my toolkit.

If I do more Go project in the future I would defenitely like to do a talk “Go for Python Developers” where I talk about my experinces of building a series of projects in both Go and Python like:

  • CLI tool

  • A key value store

  • A fully featured backend system for ecommerce website/twitter clone/blogging engine that will include REST APIs, relational DB, Redis for caching, RabbitMQ as a queue, async workers for processing, etc. As close I can get to the real thing.

Thus giving out more stronger opinions that I can back and giving a much deeper overview on what to expect when working with both the languages.

Gopher Gif

References: – Gopher Gifs: Images

2022 in the hindsight was a rough year for me. The start of the year I was super confident on what I wanted this year to be but by the end of it was as clueless as anyone.

Here are some of highlights of the year in particular order that summarise 2022 for me:

  • Gave two talks this year one at PyCascades 2022 and PyLadies Berlin.

  • Mentored a person to give her first ever talk at EuroPython 2022.

  • Started learning Golang this year and wrote a lot of silly programs in it.

  • Attended an in person conference this year after ages. God how I missed those.

  • Did a two week workcation in Himachal Pradesh, turns out I not that into this workcation thing.

  • Got back into coffee brewing. Aeropress for the win!

  • Picked up Japanese as a language this year failed at it miserably. I hope I am more consistent with it next year.

  • Bonded with a lot of childhood friend whom I thought I had grown out of.

  • Started seeing someone.

  • Sent my parents on a 10 day vacation, which I think is the highlight of the year since I always wanted to do something like this for my parents the day I got my first paycheck.

  • Paid the initial down payment for a car for my parents.

  • Quit the job that I always tried too hard to fit in and was affecting my mental health. Took a big risk here quitting without a new job in hand, hope it plays off in the longer run.

I hope 2023 is a bit kinder to me and brings in a stability.

PyLadies Berlin Logo

Source: https://www.meetup.com/PyLadies-Berlin/

PyLaides's Berlin chapter hosted a meetup on 20 November which seem very interesting to me when I first saw the agenda. It was on writing better talk proposals.

Though I have not attended many online meetups in the last two years I definitely didn't want to miss this one since I had a conference talk coming up early next year and this meetup just seemed like a really good opportunity to get some early feedback on the talk.

The meetup was sponsored by Ecosia.org and was hosted on GatherTown. The meetup agenda had only one talk and the rest was all an interactive session.

Once everyone settled in, Maria started his talk 'The Call for Proposals is open: Tips for writing a conference talk'. The talk resonated with me a lot because by the end Maria pointed out most folks don't submit a talk just because they think about not having mastery. I being a similar person who overthinks a lot of mastery could relate a lot.

Next was the Ideation round. It was the most fun and interactive part of the meetup where everyone pitched their talk idea(s) and after that, folks would give feedback (if any) on how the talk proposal can be improved.

I had initially joined the meetup to get some feedback on the talk that had already been selected for a conference. But later decided that I can utilize this chance to get some feedback on a completely new talk proposal.

So I presented a completely different talk idea 'Up and Flying with the Django Pony in 10 min' that I cooked up during the break.

Django Pony

Pony is Django's unofficial mascot. Source: https://djangopony.com/

I noticed Maria mention Python Pizza in her talk and saw they were accepting 10 min short talk. So that gave me chance to experiment and present something similar to Go in 100 Seconds.

Two other reasons that made me choose a talk on Django were

  • Most folks when starting out web development run behind learning the best or constantly switch without actually learning one well and knowing its limitations. But as I was rightly pointed out during the feedback, this could be a different talk altogether.

  • This talk could have been a really good starting point for someone who is thinking of picking up Django or moving to Django as it gives a good overview of its batteries included nature.

By the end of the Ideation round, there were only a couple of folks left. So it was decided that it really would help if everyone could write a documented proposal draft that everyone shared during the Ideation round and we could brainstorm on how can we improve it more.

The brainstorming session helped frame a better proposal and left me with a lot of valuable feedback out of which a few I could consider writing all the talk proposals in the future.

Jessica one of the PyLaides Berlin organizers did a wonderful job with the meetup keeping it interactive. While creating a safe space for everyone to share their views and feedback in a constructive manner.

I think other meetup groups can definitely take pages from PyLaides Berlin book in hosting a similar meetup where folks help out writing better talk proposals. As that would directly impact on quality of talk proposal which may also impact on the quality of talk conference gets. Who knows some of these proposals can end a being really good blogposts or meetup talks if not talks.

It might not be possible for organizers of an annual PyCon like PyCon India or other PyCon. But usually, a local chapter of Python User Groups can come up with a similar meetup in their region just after the CFP opens. Which can not only help out beginners but also help out folks improve on their talk proposal.

That's all for now. Until next time. Stay safe folks.

Last weekend I attended EuroPython's virtual sprints. Though this was my second year of attending the virtual sprints I was still a bit overwhelmed. This year not only I would have been sprinting for other projects but I would be helping other folks contribute to ScanAPI. As I had been recently added to the core team, I thought it would be a good chance for me to dive deep into the codebase while helping others at the sprints.

Unlike last year where discord was the communication channel for the sprints, this year folks at EuroPython had a matrix server. Each sprint project had its channel integrated with Jitsi to help pair program or just have a hallway-like experience.

Similar to last year sprints were planned for the whole weekend with multiple opening/closing sessions to support different timezones. I liked that they kept this from last year because other folks from ScanAPI are from Brazil and to them, the first opening session which was at 9 AM CEST or 12:30 PM IST would be something like 4:30 in the morning. This gap between multiple openings also gave me some time to just visit other projects and contribute. At least that is what I thought initially and moved the sprints timings for ScanAPI to the second day.

On the first day of the sprints, I wasn't expecting that I had to be present at the opening since we had planned the sprints for the other day. But to my surprise got an invite for the opening session to present for ScanAPI as I was anyway online. So I mostly talked about what the project is, how to run the project locally and how you can pick up the issues in the opening.

After the opening since I did not see much activity in the ScanAPI channel right away, I hopped on to EuroPython's Website sprint channel where Raquel and Francesco(web lead for the project) was already present. I had a really fun time talking to them.

By the end of the day, ScanAPI had three new issues, 4 Pull Request that was to be reviewed, 2 got merged and one contribution to the wiki on how to set up the project on windows without WSL in it.

The second day was a bit slow in general where I had initially planned the sprints. Since we had very little activity on the second day, I did wrap up early after addressing the previous day's Pull Requests. Since there was only one person active on the channel post-lunch, the conversation at the end drifted to bread baking which to me seems a really fun thing to try in your free time.

In the end, it was a weekend well-spent.

Making sense of docstrings with Python's ast and tokenize module and using recursion to rewrite Python's os.walk()

Last week was among the busy week at work with a release. Sadly personal goals suffered.

I did manage to put some effort into issues that were pending. Let's walk through each of them one at a time:

  • Lynn Root, maintainer of interrogate did give a detailed description of the issue. The approach to reaching the solution makes much more sense to me now. Based on the response wrote a simple script to check # nodocqa on docstrings i.e do not get coverage of the particular class or function. Need to check now how to implement that on interrogate.

  • On reading about Recursion realized, recursive algorithms are widely used to explore and manage the file system. This reminded me about Python's os.walk() method. Did spend some time reading the implementation though the code is well documented I am still not 100% sure how it works. I guess will learn more when I finish the script that mimics os.walk() in some manner.

That's all for the last week. Until next week. Stay safe folks.

#weeknotes2021

Interrogating docstrings, Django's custom user model, Python's ast and tokenize module.

Oh boy! The week was more fun than I had planned.

Let's walk through each of them one at a time:

  • After finishing one of the courses on Django Celery. I wanted to put my knowledge to the test by building a knockoff version of Buttondown. Started a project called Hermes. Currently, Hermes has all the models done, celery setup is done and basic auth in place. Wanted to implement a custom user model so that the username field can be emailed instead of a username. So currently stuck with a bug related to that hopefully, will be resolved by next week.

  • I picked an issue in a library interrogate. interrogate gives a coverage report of missing docstrings and currently does not have a skip function like noqa. So I picked up the issue to add that. On a quick glance saw that ast module was being internally used. Though ast module doesn't include comments. The tokenize module can give you comments but doesn't provide other program structures. So I guess I need to mix and match both to add the feature.

  • Last year I sprinted for scanapi at EuroPython sprints 2020. I moved back to the project this week. Started back again by adding some docs, adding issue templates, and docstring coverage to the project. Also, this is how I stumbled upon interrogate.

I also managed to stay on course with my yearly health goal. Meditated daily and spent a decent 40min doing some cardio/core exercises on the working weekdays.

That's it for this week. Until next week. Stay safe folks.

#weeknotes2021

Last weekend I attended EuroPython sprints that were virtually conducted. The communication platform for the conference was discord and was kept the same for the sprints too. It served a good platform as we were able to pair program with the maintainer by sharing our screens.

Day 1

Sprints opened at 12:30 PM IST and started with its first round of project introduction. A total of 12 projects that took part in this year's sprint. Though the project maintainers were from varied timezone and timezones are difficult to handle. The first opening of sprints only had a few maintainers of the project to talk about their project.

The project that I started off in the day one of the sprints was terminusdb. I primarily contributed to terminudb's python client which had Cheuk Ting Ho and Kevin Chekov Feeney to help us out. Kevin had coded the JS Client of the project and was here to work on the Python Client.

The issue I picked up was increasing the test coverage of the project and while working on that issue I also discovered some other issues. Some depreciated function was still being used in the client and the make file did not have a command to generate coverage HTML of the project.

By the end of day one, I had moved the coverage of terminusdb_client/woqlclient/connectionConfig.py to 70% from 62% with a PR to remove the deprecated function from the client. Doing that I learned about graph databases and how terminusdb has git like features for the database.

Day 2

I started late on the second day and continued to work on the test coverage PR. I fixed some minor flake8 errors in my test coverage PR and pushed to coverage to 75% and created a PR for that make file command. A lot of people in sprints were confused in setup of project. So opened up a documentation issue for writing the wiki for setup instructions and contributions guidelines for new/first time contributors.

Just an hour before the first closing session I moved to scanapi which is maintained by Camila Maia. I picked up some good first issues and got them merged in no time. I saw this project at the closing of the day-1 and found it very interesting.

The other projects that I really found interesting but could not contribute to were Hypothesis, strawberry GraphQL and commitizen.

Overall I had a really fun weekend and I am excited to contribute more to those projects.

I recently stumbled across a very peculiar topic called Bit Manipulation. In most of my programming days, I haven't actually relied on the binary operation to get me the result, I know under the hood everything is converted into 0's and 1's but it was all abstraction to me.

The case was different here. While working with Bit Manipulation, I had to actually rely on arithmetic bit operations to get me to the result. So it became real interesting real soon.

Bitwise operators

Basic operation done on bits are done with bitwise operators. Since we primarily work on bits these operations are fast and are optimized to reduce time complexity.

The first three &, | and ~ are fairly straightforward. So I would briefly go over it.

&: if both bits are of equal size than & operator would compare each position and would return True/1 if input bits are True/1. Similarly for False/0.

    6       : 1 1 0
    5       : 1 0 1
            -------- &
              1 0 0

|: if both bits are of equal size than & operator would compare each position and would return True/1 if input bits differ. Similarly for False/0.

     5       : 1 0 0
     3       : 0 1 1
            --------  |
              1 1 1

~: Not operator just compliments the bit it gets. In fancy computer lingo it gives one’s complement of a number.

    5       : 1 0 1
            -------- ~
              0 1 0

Now coming to more interesting operators:

Operator Name
^ XOR
>> Right Shift
<< Left Shift
XOR

If two bits are of two equal-size ^ of both bits in the compared position would be 1 if compared bits are of different binary and would be 0 if bot the compared bits are the same.

    6       : 1 1 0
    5       : 1 0 1
            -------- ^
              0 1 1
  • XOR of a number with itself is 0

    x = "Any int number"
    (x ^ x) == 0
    
  • XOR of a number with 0 is number itself.

    (x ^ 0) == 0
    
  • Ordering in XOR does not matter, both will give the same output.

    output = (7 ^ 3) ^ (5 ^ 4 ^ 5) ^ (3 ^ 4)
    output = 7 ^ (3 ^ (5 ^ 4 ^ 5)) ^ (3 ^ 4)
    

While discussing Left Shift,<< and Right Shift, >> we will be talking about arithmetic shifts.

Left shift <<

  • Left shift shifts the binary digits by n, pads 0’s on the right.
  • Left shift is equivalent to multiplying the bit pattern with 2 power k( if we are shifting k bits )
1 << 1 = 2 = 1 * (2  ** 1) 
1 << 2 = 4 = 1 *(2  ** 2) 
1 << 3 = 8 = 1 * (2  ** 3)
1 << 4 = 16 = 1* (2  ** 4)
…
1 << n = 2n

Right shift >>

  • Shifts the binary digits by n, pads 0's on the left.
  • Right shift is equivalent to dividing the bit pattern with 2k ( if we are shifting k bits ).
4 >> 1 = 2
6 >> 1 = 3
5 >> 1 = 2
16 >> 4 = 1

Both Right shift and Left shift operators come real handy in masking.

Masking allows the user to check/change a particular bit at a particular position.

Some of the common functions associated with masking are:

Set Bit
  • The set bit method is generally used to SET a particular with 1.
  • To achieve this we would need to create a mask at the particular position where we want to SET
  • The mask can be created with the help of the << if the left shift operator.
def set_bit(x, position):
    mask = 1 << position
    return x | mask

set_bit(6,1)
  • In the above code snippet we are SETing the bit at 0th index.
    masking = 1 << 0 = 1 * (2 ** 0) 
    
    6       : 1 1 0
    1 << 0  : 0 0 1
            -------- |
              1 1 1
IS BIT SET
def is_bit_set(x, position):
    shifted = x >> position
    return shifted & 1
Clearing Bit
def clear_bit(x, position):
    mask = 1 << position
    return x & ~mask
Flip Bit
def flip_bit(x, position):
    mask = 1 << position
    return x ^ mask
Modify Bit
def modify_bit(x, position, state):
    """
    state is param that tells us to set a bit 
    or clear a bit
    """
    mask = 1 << position
    return (x & ~mask) | (-state & mask)

Observations

Bit manipulation can be used to solve problems that you are familiar with but necessarily don't know about. Here are some of my observations that I noted while using bit manipulation.

To check if the number is even
  • & ANDing the number with 1 gives 0 or 1 — 0 if it's even — 1 if it's odd
x = "Any int number here"
(x & 1) == 0

Practice Question

To check if the number is a power of two
  • If a number is x binary representation of (x-1) can be obtained by simply flipping all the bits to the right of rightmost 1 in x and also including the rightmost 1.
Let, x = 4 = (100)2
x - 1 = 3 = (011)2
Let, x = 6 = (110)2
x - 1 = 5 = (101)2
  • x & (x-1) will have all the bits equal to the x except for the rightmost 1 in x. In the given example below the values enclosed in || are the same for both the x and x-1 if x is not the power of 2.
  • If the number is neither zero nor a power of two, it will have 1 in more than one place.
Let, x = 6 = 1|1|0
(x- 1) = 5 = 1|0|1

Let,x = 16 = |1|0000
(x-1) = 15 = |0|1111

Let,x = 8 = |1|000
(x-1) = 7 = |0|111

Let,x = 23 = 1011|1|
(x-1) = 22 = 1011|0|
x = "Any int number here"
(x & x-1) == 0

There are a lot more things that can be done with just bits and are definitely not limited to the above observations. Try to find your own observations. Happy coding!

I recently finished reading Python Testing with Pytest by Brian Okken and I am glad I picked this up rather than jumping into the docs. It's definitely a good introduction for people who haven't had their share of testing a python codebase, let alone be with Pytest.

The book introduces a python CLI called Tasks and takes this as a base for writing all of its tests throughout the course of the book. Though eventually, the tests become more complex when you get into the latter half of the book.

The pros of the book are that it covers almost every section of the framework from fixtures, plugins, custom pytest configuration and even using pytest with tools like coverage and mock. But if you're someone like me who hasn't had his share of testing a python codebase you might find yourself with a bit of information overload at times.

I did find the book a bit of overwhelming on chapters like writing your own plugin, custom configuration and using pytest with Jenkins because these are the features that I wouldn't be using right out of the box. I would definitely be coming back to these chapters in the future if I need any of the features.

Overall the book is really well-written keeping in mind beginners who are just picking up pytest as their first testing framework and also for folks who are moving towards pytest from any other testing framework. Exercises at the back of every chapter make sure you also get some hands-on experience of writing tests.

Just a personal tip for anyone who is picking this up and has less experience with pytest. Feel free to skip chapters or skim chapters that aren't useful right out of the box. You can always come back to them when you need those features.

2019 has been a year of new beginnings both personally and professionally. This was the year I got my first job, the first salary and on the contrary to that, I did give my first resignation. Yeah, that was fun!

This blog just highlights most of the things I did in the previous year.

Blog Posts

I did post out 8 blogs this year. I know it's not that much. Initially, I had planned one blog a month. But by the end of the year during the time I was giving interviews for the new job things started to fall and I could not commit to one blog a month.

The plan for this year is to blog more or at least be consistent with writing. Stick to at least one blog per month.

Books

The previous year was a good reading year compared to the last few years. The Kindle I bought came real handy during the long metro rides. Plus I got some tech books cheap compared to their paperback prices so I did finish some of them too.

This year I started to take up reading non-tech books a bit more seriously. So I am picking up a book a month and finishing it slowly. Keeping in consideration that the book is less than 800-1000 pages for the initial months just to help in making a momentum.

Recently finished Parliamental and will be moving to The Elephant Vanishes.

Talks

I did give one talk at PyConf Hyderabad 2019 one of my favorite regional conferences in India. I also did submit one for a PyDelhi meetup but sadly by the time, it was scheduled I had already relocated. More on that later.

Open Source Contributions

One of the major things that I want to work towards this year is towards making more upstream contributions.

Last year I did submit two document patches to one org aio-libs . The project was aiopg, async version of Postgres but that happened by sheer luck. As I was going through the documentation I found some of the documentation to be using old-styled decorator based coroutines instead of new async def function. So I submitted a patch to update them.