Feeds

We in the Python community have a deep appreciation for the volunteers who organize, promote, and write the language. A phrase that has become a cornerstone of our community, afterall, ‘Come for the language, stay for the community’ (derived from Python core developer Brett Cannon’s opening remarks at PyCon 2014), reflects the passion of our community and more so of the countless volunteers building our community. One volunteer who has been steadfast in actively building the Python community - from her contributions to CPython to her work as an organizer and co-chair of PyCascades 2018 and more - is Mariatta Wijaya. We at the Python Software Foundation are pleased to name Mariatta Wijaya as a 2018 Q3 Community Service Award recipient: RESOLVED, that the Python Software Foundation award the Q3 2018 Community Service Award to Mariatta Wijaya for her contributions to CPython, diversity efforts for the Python Core Contributor team, and her work on PyCascades. Come talk! The path to becoming a Core Python Developer At Montreal PyCon 2015, Guido Van Rossum delivered the closing keynote during which Guido issued a public ask, “I want at least two female Python core developers in the next year ... and I will try to train them myself if that's what it takes. So come talk to me." Consequently, Mariatta did just that, she reached out to Guido after PyCon 2016 to learn more about starting in Python core development. Mariatta recalls, “I hadn’t contributed to open source [yet] and I wanted to know how to start”. Guido recommended some ways for Mariatta to start including reviewing the dev guide, looking at open issues and joining and introducing herself on the Python dev mailing list . Following Guido’s advice, Mariatta “read the issues [to] see if there is anything I can help with, anything that interests me ... [when I learned that] Brett was starting migration [of Python] to GitHub”. As an engineer at Zapier, Mariatta has a background in web development so the migration provided an initial issue she could begin to explore. Mariatta has since contributed to several bots that improve the workflows for Python contributors and core developers, reviewed and merged 700+ PRs to Python, and is Co-Chair of the Language Summit in 2019 and 2020. Some examples of bots she has written include cherry-picker, a “tool used to backport CPython changes from master into one or more of the maintenance branches”. Additionally, Mariatta is the author of PEP-581: Using GitHub Issues for CPython. Her motivations behind this PEP again come back to improving the core Python development processes, “I think it will be more beneficial to use an out of the box issue tracker like GitHub as it will allow core developers to focus on developing and contributing”. A role model for us all: Increasing the diversity of the Core Python Development team The recent departure of Guido as BDFL and the subsequent discussion about the future governance of core Python led to several suggestions, with the Steering Council ultimately becoming the chosen model. Core Python developer Victor Stinner along with several others nominated Mariatta to the Steering Council. In Victor’s nomination he explains, “Mariatta became the first woman core developer in Python [in 2017]. She is actively sharing her experience to encourage people from underrepresented groups to contribute to Python.” The work required to become a core developer is laborious yet Mariatta has continuously gone the extra mile to lead by example and be active in public outreach. “Mariatta is my role model for mentoring and diversity which is helping a lot to get more people involved in Python,” Victor adds. Python core developer and Steering Council member Carol Willing echoed this sentiment sharing, “Mariatta works to share Python and its possible uses with others. Her blend of hard work, enthusiasm, and caring have welcomed many into the Python Community,” Mariatta’s PyCon 2018 talk titled, “What is a Core Python Developer” is an ideal example of Mariatta’s dedication to building a more diverse core Python team. Beginning with a question, “do you use f’strings” (Mariatta is a known avid fan of f’strings, she even has stickers for them) Mariatta dives into a talk about what the pathway is for core (and contributing) developers ultimately commenting on the very real, stark gender imbalance within the core team, “We have 848 contributors to Python, less than 10 are women. We have 89 core developers, only 2 are women ... This is real but this is also wrong. This is not the right representation of our community”.  While this number is starting to change as more women are promoted to core development (Cheryl Sabella’s promotion this week ups the number of women core developers up to 5 out of 97), Mariatta has continued to be a champion and advocate for diversity and inclusion in the core development team. Even the captionist in Mariatta's PyCon 2018 (seen below in tweets) talk captured their appreciation for Mariatta's dedication. Heard a story today that brought me to tears. A talk I did at #PyCon in 2014 inspired a woman to become involved in the community, speak, and become a core contributor. Diversity is important.@mariatta Thank you for sharing. The python was in you all along ?? — ???? ?????????? ? (@nnja) May 12, 2018 Yes you are amazing @mariatta thank you for your work, your contributions, for sharing your story. Our #python community is better because of people like you. #pycon2018 pic.twitter.com/84FvYr4dSq — Loooorena Mesa @ The Cosmos ??? (@loooorenanicole) May 12, 2018 Awesome talk on becoming a python core developer by the amazing @mariatta. More women need to contribute to CPython and they can look to @mariatta for advice. Please spread the word. @pycon #pycon #pycon2018. — saptarshikar (@saptarshikar) May 12, 2018 The North Star of PyCascades Outside of her contributions to CPython, Mariatta has been an active organizer with PyCascades - a regional Python conference now about to kick off its second conference this week. The inaugural 2018 conference, held in Mariatta’s hometown Vancouver, Canada, introduced a a single track format with 30 minute talks, no question and answer, and includes lightning talks. Inspired by the single track format of Write the Docs and DjangoCon Europe, this format was not only an easier way to get a new conference off the ground but, as Mariatta observed, “is able to give [speakers] the large audience they deserve”. This format also makes it easier for attendees to navigate. Co-Chairing the conference with Mariatta in 2018, Seb Vetter remarked, “Mariatta has been THE driving force behind PyCascades in the inaugural year”. As a co-chair Mariatta helped respond to many last minute issues such as when, the day before the conference at 10:00am local time, Guido informed the organizers he was unable to obtain a visa to travel to speak at PyCascades. Within a few hours, the team setup for Guido to speak remotely, had sent him a badge, when the team learned Guido would be able to attend after all! “When we found out he's coming, we printed one more badge for him. That's why he has multiple badges,” Mariatta explained. Juggling many changing priorities is the life of an organizer. Yet each decision made, “she ensured … considered the potential impact on the diversity of the conference,” Seb remembered adding, “[Mariatta] seems to have an endless stream of enthusiasm and energy and was our North Star for doing everything we could to make it as inclusive for attendees as possible”. The idea of Mariatta acting as a North Star was echoed by PyCascades organizer Don Sheu adding, “she gives voice to folks that aren’t sufficiently represented in tech … [as a part of] PyCascades founding team, Mariatta’s influence is creating a safe environment”. With PyCascades 2019 happening in Seattle this upcoming weekend (February 23 - 24), Mariatta is again contributing as an organizer. What do f-string stickers and food have in common? Mariatta’s love of them! Outside of Python, when asked what else Mariatta likes to do she simply responded, “I love food!”. And her favorite food? Asian cuisine. #IceCreamSelfie at North Bay Python 2018. Source: https://mariatta.ca/img/ics-northbaypython-2018.jpg. If you happen to see Mariatta at an event, say hi. Maybe she’ll have f-string sticker for you! I've just ordered more f-string stickers in preparation for #PyCascades2019 ? — Mariatta ? (@mariatta) January 15, 2019

Sometimes we want to run certain tests only on a specific version of Python. Suppose you are migrating a large project from Python 2 to Python 3 and you know in advance that certain tests won't run under Python 3. Chances are that during the migration you are already using the six library. The six libraries have two boolean properties which are initialised to True depending on the Python version which is being used: PY2 when running under Python 2 and PY3 when running under Python 3. This library, combined with the skipIf method of unittest library can be used to easily skip tests when using Python 3: import six import unittest class MyTestCase(unittest.TestCase): @unittest.skipIf(six.PY3, "not compatible with Python 3") def test_example(self): # This test won't run under Python 3 pass Credits Thanks to my colleague Nicola for giving me the inspiration to write this post.

Yesterday we hosted a webinar with Michael Kennedy from Talk Python To Me podcasts and training presenting Demystifying Python’s async and await Keywords. Turned out to be the highest-rated webinar in 7 years of JetBrains’ webinars. Thanks Michael! The webinar recording is now available, as well as a repository with the Python code he showed and the slides he used. During the webinar, Michael laid the basis for async programming in Python, detailing CPU parallelism versus I/O parallelism. He showed the impacts of each on rendering time and how different forms of parallelism affect rendering times. In the code, he started with a basic, naive function that ran horribly slow, then gradually sped it up with different Python techniques (generators, async/await, etc.) He also covered companion libraries that have emerged in the Python ecosystem. For those who want a deep-dive on this topic, Michael has a 4 hour course Async Techniques and Examples in Python. He also has a thorough Mastering PyCharm. Thanks so much to Michael for this well-prepared, well-presented webinar and staying late to handle the record-number of questions. -PyCharm Team- The Drive to Develop

Introduction In this tutorial, we will explore the conversion of Python scripts to Windows executable files in four simple steps. Although there are many ways to do it, we'll be covering, according to popular opinion, the simplest one so far. This tutorial has been designed after reviewing many common errors that people face while performing this task, and hence contains detailed information to install and set up all the dependencies as well. Feel free to skip any step, if you already have those dependencies installed. Without any further ado, let's start. Step 1: Install cURL cURL provides a library and command line tool for transferring data using various protocols. We need it to download the pip package manager in the next step. Many of you would already have it set up, which you can check by running the following command: $ curl --version If the command above returns a curl version, you can skip the next instructions in this step. As for the rest of you, you can install curl by following these three steps: Go to https://curl.haxx.se/dlwiz/?type=bin&os=Win64&flav=-&ver=*&cpu=x86_64 Download the curl package which matches your system's specifications (32-bit/64-bit) Unzip the file and go to the bin folder, you can find the curl.exe file there However, this means that you can only use the curl command in that particular folder. In order to be able to use the curl command from anywhere on your machine, right-click on curl.exe, click on "Properties" and copy the "Location" value. After that, right-click on "My PC" and click on "Properties". In the option panel on the left, select the option "Advanced System Settings". It has been highlighted in the screenshot below. In the window that appears, click "Environment Variables" near the bottom right. It has been highlighted in the screenshot below. In the next window, find and double click on the user variable named "Path", then click on "New". A new text box will be created in that window; paste the "Location" value of the "curl.exe" file that you copied earlier, and then click on 'OK'. cURL should now be accessible from anywhere in your system. Confirm your installation by running the command below: $ curl --version Let's go to the next step. Step 2: Install pip In this step, we will install pip, which is basically a package manager for Python packages. We need it in the next step to install the pyinstaller library. Most of you would already have it set up, to check run the following command: $ pip --version If the command above returned a pip version, you can skip the next instructions in this step. As for the rest, you can install pip by running the following two commands in the command prompt: $ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py $ python get-pip.py That's it. Pip has now been installed to your local machine! You can run the following command for confirmation: $ pip --version Before moving to the next step, you need to repeat what we did for curl.exe so that you can access the pip command from anywhere in your machine, but this time we'll be doing so for "pip.exe". Hit the Windows key and search for "pip.exe", then right-click on the first search result and click on "Open File Location", it will take you to the folder in which that file is located. Right-click on the "pip.exe" file and then select "Properties". After that, copy the "Location" value and paste it in the Path variable just like we did in Step 1. Step 3: Install PyInstaller In this step, we'll install pyinstaller using pip. We need pyinstaller to convert our Python scripts into executable (.exe) files. You just need to copy paste the command below into your command prompt and run it: $ pip install pyinstaller Again, to confirm your installation, run the following command: $ pyinstaller --version Note: If you've Anaconda installed in your system, then you're probably using conda package manager instead. In that case, run the following commands below, in sequence: $ conda install -c conda-forge pyinstaller $ conda install -c anaconda pywin32 This step marks the end of all installations. In the next step, we'll be converting our Python files to an executable with just a single command. Step 4: Convert Python Files to Executables This is the last step. We'll use pyinstaller to convert our .py files to .exe with a single command. So, let's do it! Open up the command prompt and navigate to the directory that your Python file/script is located in. Alternatively, you can open that directory using File Explorer, right-click + shift and then select "Open Command Prompt in this folder". Before converting your file, you should check that your file works as expected. For that purpose, I have written a basic Python script which prints the number 10 when executed. Let's run the script and see if it works fine before converting it to an executable file. Run the following command on your command prompt: $ python name_of_your_file.py In my case, the filename was 'sum.py'. To create a standalone executable file in the same directory as your Python file, run the following command: $ pyinstaller --onefile <file_name>.py This instruction might take some time to complete. Upon completion, it will generate three folders. You can find the executable file in the 'dist' folder. Please note that the "onefile" argument tells pyinstaller to create a single executable file only. Let's now run our executable file to see if the procedure worked! Ta-da! It worked just as expected. A little tip, if your executable file closes too fast for you to notice the output, you can add an input() line at the end of your Python file, which keeps the prompt open while waiting for using input. That is how I was able to take a screenshot of my output as well. Also note that if your executable depends on any other executable files, like phantomjs, you need to keep them in the same directory as your Python file's directory so that pyinstaller can include it in the executable. Conclusion In this tutorial, we discussed in detail the conversion of Python scripts to executable files using Python's pyinstaller library in four steps. We started by installing cURL, followed by pip and pyinstaller. Lastly, we converted a sample Python file to executable to ensure that the procedure works on Windows.

Analyzing images with code can be difficult. How do you make your code "understand" the context of an image? In general, the first step of analyzing images with AI is finding the dominant colors. In this tutorial, we're going to find dominant colors in images using matplotlib's image class. Finding dominant colors is also something you can do with third-party APIs, but we're going to build our own system for doing this so that we have total control over the process. We will first look at converting an image into its component colors in the form of a matrix, and then perform k-means clustering on it to find the dominant colors. Prerequisites This tutorial assumes that you know basics of Python, but you don't need to have worked with images in Python before. This tutorial is based on the following: Python version 3.6.5 matplotlib version 2.2.3: to decode images and visualize dominant colors scipy version 1.1.0: to perform clustering that determines dominant colors The packages matplotlib and scipy can be installed through a package manager — pip. You may want to install specific versions of packages in a virtual environment to make sure there are no clashes with dependencies of other projects you are working on. pip install matplotlib==2.2.3 pip install scipy==1.1.0 Further, we analyze JPG images in this tutorial, the support for which is available only when you install an additional package, pillow. pip install pillow==5.2.0 Alternatively, you can just use a Jupyter notebook. The code for this tutorial was run on a Jupyter notebook in Anaconda version 1.8.7. These above packages are pre-installed in Anaconda. import matplotlib matplotlib.__version__ '3.0.2' import PIL PIL.__version__ '5.3.0' import scipy scipy.__version__ '1.1.0' Decoding images Images may have various extensions — JPG, PNG, TIFF are common. This post focuses on JPG images only, but the process for other image formats should not be very different. The first step in the process is to read the image. An image with a JPG extension is stored in memory as a list of dots, known as pixels. A pixel, or a picture element, represents a single dot in an image. The color of the dot is determined by a combination of three values — its three component colors (Red, Blue and Green). The color of the pixel is essentially a combination of these three component colors. Image source: Datagenetics Let us use Dataquest's logo for the purpose of finding dominant colors in the image. You can download the image here. To read an image in Python, you need to import the image class of matplotlib (documentation). The imread() method of the image class decodes an image into its RGB values. The output of the imread() method is an array with the dimensions M x N x 3, where M and N are the dimensions of the image. from matplotlib import image as img image = img.imread('./dataquest.jpg') image.shape (200, 200, 3) You can use the imshow() method of matplotlib's pyplot class to display an image, which is in the form of a matrix of RGB values. %matplotlib inline from matplotlib import pyplot as plt plt.imshow(image) plt.show() The matrix we get from the imread() method depends on the type of the image being read. For instance, PNG images would also include an element measuring a pixel's level of transparency. This post will only cover JPG images. Before moving on to clustering the images, we need to perform an additional step. In the process of finding out the dominant colors of an image, we are not concerned about the position of the pixel. Hence, we need to convert the M x N x 3 matrix to three individual lists, which contain the respective red, blue and green values. The following snippet converts the matrix stored in image into three individual lists, each of length 40,000 (200 x 200). r = [] g = [] b = [] for line in image: for pixel in line: temp_r, temp_g, temp_b = pixel r.append(temp_r) g.append(temp_g) b.append(temp_b) The snippet above creates three empty lists, and then loops through each pixel of our image, appending the RGB values to our r, g, and b lists, respectively. If done correctly, each list wiill have a length of 40,000 (200 x 200). Clustering Basics Now that we have stored all of the component colors of our image, it is time to find the dominant colors. Let us now take a moment to understand the basics of clustering and how it is going to help us find the dominant colors in an image. Clustering is a technique that helps in grouping similar items together based on particular attributes. We are going to apply k-means clustering to the list of three colors that we have just created above. The colors at each cluster center will reflect the average of the attributes of all members of a cluster, and that will help us determine the dominant colors in the image. How Many Dominant Colors? Before we perform k-means clustering on the pixel data points, it might be good for us to figure out how many clusters are ideal for a given image, since not all images will have the same number of dominant colors. Since we are dealing with three variables for clustering — the Red, Blue and Green values of pixels — we can visualize these variables on three dimensions to understand how many dominant colors may exist. To make a 3D plot in matplotlib, we will use the Axes3D() class (documentation). After initializing the axes using the Axes3D() class, we use the scatter method and use the three lists of color values as arguments. from mpl_toolkits.mplot3d import Axes3D fig = plt.figure() ax = Axes3D(fig) ax.scatter(r, g, b) plt.show() In the resultant plot, we can see that the distribution of points forms two elongated clusters. This is also supported by the fact that by looking at it, we can see that the image is primarily composed of two colors. Therefore, we will focus on creating two clusters in the next section. First, though, we can probably guess that it's possible that a 3D plot may not yield distinct clusters for some images. Additionally, if we were using a PNG image, we would have a fourth agrument (each pixel's transparency value), which would make plotting in three dimensions impossible. In such cases, you may need to use the elbow method to determine the ideal number of clusters. Perform Clustering in SciPy In the previous step, we've determined that we'd like two clusters, and now we are ready to perform k-means clustering on the data. Let's create a Pandas data frame to easily manage the variables. import pandas as pd df = pd.DataFrame({'red': r, 'blue': b, 'green': g}) There are essentially three steps involved in the process of k-means clustering with SciPy: Standardize the variables by dividing each data point by its standard deviation. We will use the whiten() method of the vq class. Generate cluster centers using the kmeans() method. Generate cluster labels for each data point using the vq() method of the vq class. The first step above ensures that the variations in each variable affects the clusters equally. Imagine two variables with largely different scales. If we ignore the first step above, the variable with the larger scale and variation would have a larger impact on the formation of clusters, thus making the process biased. We therefore standardize the variable using the whiten() function. The whiten() function takes one argument, a list or array of the values of a variable, and returns the standardized values. After standardization, we print a sample of the data frame. Notice that the variations in the columns has reduced considerably in the standardized columns. from scipy.cluster.vq import whiten df['scaled_red'] = whiten(df['red']) df['scaled_blue'] = whiten(df['blue']) df['scaled_green'] = whiten(df['green']) df.sample(n = 10) red blue green scaled_red scaled_blue scaled_green 24888 255 255 255 3.068012 3.590282 3.170015 38583 255 255 255 3.068012 3.590282 3.170015 2659 67 91 72 0.806105 1.281238 0.895063 36954 255 255 255 3.068012 3.590282 3.170015 39967 255 255 255 3.068012 3.590282 3.170015 13851 75 101 82 0.902357 1.422033 1.019377 14354 75 101 82 0.902357 1.422033 1.019377 18613 75 100 82 0.902357 1.407954 1.019377 6719 75 100 82 0.902357 1.407954 1.019377 14299 71 104 82 0.854231 1.464272 1.019377 The next step is to perform k-means clustering with the standardized columns. We will use the kmeans() function of perform clustering. The kmeans() function (documentation) has two required arguments — the observations and the number of clusters. It returns two values — the cluster centers and the distortion. Distortion is the sum of squared distances between each point and its nearest cluster center. We will not be using distortion in this tutorial. from scipy.cluster.vq import kmeans cluster_centers, distortion = kmeans(df[['scaled_red', 'scaled_green', 'scaled_blue']], 2) The final step in k-means clustering is generating cluster labels. However, in this exercise, we won't need to do that. We are only looking for the dominant colors, which are represnted by the cluster centers. Display Dominant Colors We have performed k-means clustering and generated our cluster centers, so let's see what values they contain. print(cluster_centers) [[2.94579782 3.1243935 3.52525635] [0.91860119 1.05099931 1.4465091 ]] As you can see, the results we get are standardized versions of RGB values. To get the original color values we need to multiply them with their standard deviations. We will display the colors in the form of a palette using the imshow() method of matplotlib's pyplot class. However, to display colors, imshow() needs the values of RGB in the range of 0 to 1, where 1 signifies 255 in our original scale of RGB values. We therefore must divide each RGB component of our cluster centers with 255 in order to get to a value between 0 to 1, and display them through the imshow() method. Finally, we have one more consideration before we plot the colors using the imshow() function (documentation). The dimensions of the cluster centers are N x 3, where N is the number of clusters. imshow() is originally intended to display an A X B matrix of colors, so it expects a 3D array of dimentions A x B x 3 (three color elements for each block in the palette). Hence, we need to convert our N x 3 matrix to 1 x N x 3 matrix by passing the colors of the cluster centers as a list with a single element. For instance, if we stored our colors in colors we need to pass [colors] as an argument to imshow(). Let's explore the dominant colors in our image. colors = [] r_std, g_std, b_std = df[['red', 'green', 'blue']].std() for cluster_center in cluster_centers: scaled_r, scaled_g, scaled_b = cluster_center colors.append(( scaled_r * r_std / 255, scaled_g * g_std / 255, scaled_b * b_std / 255 )) plt.imshow([colors]) plt.show() As expected, the colors seen in the image are quite similar to the prominent colors in the image that we started with. That said, you've probably noticed that the light blue color above doesn't actually appear in our source image. Remember, the cluster centers are the means all of of the RGB values of all pixels in each cluster. So, the resultant cluster center may not actually be a color in the original image, it is just the RBG value that's at the center of the cluster all similar looking pixels from our image. Conclusion In this post, we looked at a step by step implementation for finding the dominant colors of an image in Python using matplotlib and scipy. We started with a JPG image and converted it to its RGB values using the imread() method of the image class in matplotlib. We then performed k-means clustering with scipy to find the dominant colors. Finally, we displayed the dominant colors using the imshow() method of the pyplot class in matplotlib. Want to learn more Python skills? Dataquest offers a full sequence of courses that can take you from zero to data scientist in Python. Sign up and start learning for free!

Introduction Resources are never sufficient to meet growing needs in most industries, and now especially in technology as it carves its way deeper into our lives. Technology makes life easier and more convenient and it is able to evolve and become better over time. This increased reliance on technology has come at the expense of the computing resources available. As a result, more powerful computers are being developed and the optimization of code has never been more crucial. Application performance requirements are rising more than our hardware can keep up with. To combat this, people have come up with many strategies to utilize resources more efficiently – Containerizing, Reactive (Asynchronous) Applications, etc. Though, the first step we should take, and by far the easiest one to take into consideration, is code optimization. We need to write code that performs better and utilizes less computing resources. In this article, we will optimize common patterns and procedures in Python programming in an effort to boost the performance and enhance the utilization of the available computing resources. Problem with Performance As software solutions scale, performance becomes more crucial and issues become more grand and visible. When we are writing code on our localhost, it is easy to miss some performance issues since usage is not intense. Once the same software is deployed for thousands and hundreds of thousands of concurrent end-users, the issues become more elaborate. Slowness is one of the main issues to creep up when software is scaled. This is characterized by increased response time. For instance, a web server may take longer to serve web pages or send responses back to clients when the requests become too many. Nobody likes a slow system especially since technology is meant to make certain operations faster, and usability will decline if the system is slow. When software is not optimized to utilize available resources well, it will end up requiring more resources to ensure it runs smoothly. For instance, if memory management is not handled well, the program will end up requiring more memory, hence resulting in upgrading costs or frequent crashes. Inconsistency and erroneous output is another result of poorly optimized programs. These points highlight the need for optimization of programs. Why and When to Optimize When building for large scale use, optimization is a crucial aspect of software to consider. Optimized software is able to handle a large number of concurrent users or requests while maintaining the level of performance in terms of speed easily. This leads to overall customer satisfaction since usage is unaffected. This also leads to fewer headaches when an application crashes in the middle of the night and your angry manager calls you to fix it instantly. Computing resources are expensive and optimization can come in handy in reducing operational costs in terms of storage, memory, or computing power. But when do we optimize? It is important to note that optimization may negatively affect the readability and maintainability of the codebase by making it more complex. Therefore, it is important to consider the result of the optimization against the technical debt it will raise. If we are building large systems which expect a lot of interaction by the end users, then we need our system working at the best state and this calls for optimization. Also, if we have limited resources in terms of computing power or memory, optimization will go a long way in ensuring that we can make do with the resources available to us. Profiling Before we can optimize our code, it has to be working. This way we can be able to tell how it performs and utilizes resources. And this brings us to the first rule of optimization - Don't. As Donald Knuth - a mathematician, computer scientist, and professor at Stanford University put it: "Premature optimization is the root of all evil." The solution has to work for it to be optimized. Profiling entails scrutiny of our code and analyzing its performance in order to identify how our code performs in various situations and areas of improvement if needed. It will enable us to identify the amount of time that our program takes or the amount of memory it uses in its operations. This information is vital in the optimization process since it helps us decide whether to optimize our code or not. Profiling can be a challenging undertaking and take a lot of time and if done manually some issues that affect performance may be missed. To this effect, the various tools that can help profile code faster and more efficiently include: PyCallGraph - which creates call graph visualizations that represent calling relationships between subroutines for Python code. cProfile - which will describe how often and how long various parts of Python code are executed. gProf2dot - which is a library that visualized profilers output into a dot graph. Profiling will help us identify areas to optimize in our code. Let us discuss how choosing the right data structure or control flow can help our Python code perform better. Choosing Data Structures and Control Flow The choice of data structure in our code or algorithm implemented can affect the performance of our Python code. If we make the right choices with our data structures, our code will perform well. Profiling can be of great help to identify the best data structure to use at different points in our Python code. Are we doing a lot of inserts? Are we deleting frequently? Are we constantly searching for items? Such questions can help guide us choose the correct data structure for the need and consequently result in optimized Python code. Time and memory usage will be greatly affected by our choice of data structure. It is also important to note that some data structures are implemented differently in different programming languages. For Loop vs List Comprehensions Loops are common when developing in Python and soon enough you will come across list comprehensions, which are a concise way to create new lists which also support conditions. For instance, if we want to get a list of the squares of all even numbers in a certain range using the for loop: new_list = [] for n in range(0, 10): if n % 2 == 0: new_list.append(n**2) A List Comprehension version of the loop would simply be: new_list = [ n**2 for n in range(0,10) if n%2 == 0] The list comprehension is shorter and more concise, but that is not the only trick up its sleeve. They are also notably faster in execution time than for loops. We will use the Timeit module which provides a way to time small bits of Python code. Let us put the list comprehension against the equivalent for loop and see how long each takes to achieve the same result: import timeit def for_square(n): new_list = [] for i in range(0, n): if i % 2 == 0: new_list.append(n**2) return new_list def list_comp_square(n): return [i**2 for i in range(0, n) if i % 2 == 0] print("Time taken by For Loop: {}".format(timeit.timeit('for_square(10)', 'from __main__ import for_square'))) print("Time taken by List Comprehension: {}".format(timeit.timeit('list_comp_square(10)', 'from __main__ import list_comp_square'))) After running the script 5 times using Python 2: $ python for-vs-lc.py Time taken by For Loop: 2.56907987595 Time taken by List Comprehension: 2.01556396484 $ $ python for-vs-lc.py Time taken by For Loop: 2.37083697319 Time taken by List Comprehension: 1.94110512733 $ $ python for-vs-lc.py Time taken by For Loop: 2.52163410187 Time taken by List Comprehension: 1.96427607536 $ $ python for-vs-lc.py Time taken by For Loop: 2.44279003143 Time taken by List Comprehension: 2.16282701492 $ $ python for-vs-lc.py Time taken by For Loop: 2.63641500473 Time taken by List Comprehension: 1.90950393677 While the difference is not constant, the list comprehension is taking less time than the for loop. In small-scale code, this may not make that much of a difference, but at large-scale execution, it may be all the difference needed to save some time. If we increase the range of squares from 10 to 100, the difference becomes more apparent: $ python for-vs-lc.py Time taken by For Loop: 16.0991549492 Time taken by List Comprehension: 13.9700510502 $ $ python for-vs-lc.py Time taken by For Loop: 16.6425571442 Time taken by List Comprehension: 13.4352738857 $ $ python for-vs-lc.py Time taken by For Loop: 16.2476081848 Time taken by List Comprehension: 13.2488780022 $ $ python for-vs-lc.py Time taken by For Loop: 15.9152050018 Time taken by List Comprehension: 13.3579590321 cProfile is a profiler that comes with Python and if we use it to profile our code: Upon further scrutiny, we can still see that the cProfile tool reports that our List Comprehension takes less execution time than our For Loop implementation, as we had established earlier. cProfile displays all the functions called, the number of times they have been called and the amount of time taken by each. If our intention is to reduce the time taken by our code to execute, then the List Comprehension would be a better choice over using the For Loop. The effect of such a decision to optimize our code will be much clearer at a larger scale and shows just how important, but also easy, optimizing code can be. But what if we are concerned about our memory usage? A list comprehension would require more memory to remove items in a list than a normal loop. A list comprehension always creates a new list in memory upon completion, so for deletion of items off a list, a new list would be created. Whereas, for a normal for loop, we can use the list.remove() or list.pop() to modify the original list instead of creating a new one in memory. Again, in small-scale scripts, it might not make much difference, but optimization comes good at a larger scale, and in that situation, such memory saving will come good and allow us to use the extra memory saved for other operations. Linked Lists Another data structure that can come in handy to achieve memory saving is the Linked List. It differs from a normal array in that each item or node has a link or pointer to the next node in the list and it does not require contiguous memory allocation. An array requires that memory required to store it and its items be allocated upfront and this can be quite expensive or wasteful when the size of the array is not known in advance. A linked list will allow you to allocate memory as needed. This is possible because the nodes in the linked list can be stored in different places in memory but come together in the linked list through pointers. This makes linked lists a lot more flexible compared to arrays. The caveat with a linked list is that the lookup time is slower than an array's due to the placement of the items in memory. Proper profiling will help you identify whether you need better memory or time management in order to decide whether to use a Linked List or an Array as your choice of the data structure when optimizing your code. Range vs XRange When dealing with loops in Python, sometimes we will need to generate a list of integers to assist us in executing for-loops. The functions range and xrange are used to this effect. Their functionality is the same but they are different in that the range returns a list object but the xrange returns an xrange object. What does this mean? An xrange object is a generator in that it's not the final list. It gives us the ability to generate the values in the expected final list as required during runtime through a technique known as "yielding". The fact that the xrange function does not return the final list makes it the more memory efficient choice for generating huge lists of integers for looping purposes. If we need to generate a large number of integers for use, xrange should be our go-to option for this purpose since it uses less memory. If we use the range function instead, the entire list of integers will need to be created and this will get memory intensive. Let us explore this difference in memory consumption between the two functions: $ python Python 2.7.10 (default, Oct 23 2015, 19:19:21) [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> >>> r = range(1000000) >>> x = xrange(1000000) >>> >>> print(sys.getsizeof(r)) 8000072 >>> >>> print(sys.getsizeof(x)) 40 >>> >>> print(type(r)) <type 'list'> >>> print(type(x)) <type 'xrange'> We create a range of 1,000,000 integers using range and xrange. The type of object created by the range function is a List that consumes 8000072 bytes of memory while the xrange object consumes only 40 bytes of memory. The xrange function saves us memory, loads of it, but what about item lookup time? Let's time the lookup time of an integer in the generated list of integers using Timeit: import timeit r = range(1000000) x = xrange(1000000) def lookup_range(): return r[999999] def lookup_xrange(): return x[999999] print("Look up time in Range: {}".format(timeit.timeit('lookup_range()', 'from __main__ import lookup_range'))) print("Look up time in Xrange: {}".format(timeit.timeit('lookup_xrange()', 'from __main__ import lookup_xrange'))) The result: $ python range-vs-xrange.py Look up time in Range: 0.0959858894348 Look up time in Xrange: 0.140854120255 $ $ python range-vs-xrange.py Look up time in Range: 0.111716985703 Look up time in Xrange: 0.130584001541 $ $ python range-vs-xrange.py Look up time in Range: 0.110965013504 Look up time in Xrange: 0.133008003235 $ $ python range-vs-xrange.py Look up time in Range: 0.102388143539 Look up time in Xrange: 0.133061170578 xrange may consume less memory but takes more time to find an item in it. Given the situation and the available resources, we can choose either of range or xrange depending on the aspect we are going for. This reiterates the importance of profiling in the optimization of our Python code. Note: xrange is deprecated in Python 3 and the range function can now serve the same functionality. Generators are still available on Python 3 and can help us save memory in other ways such as Generator Comprehensions or Expressions. Sets When working with Lists in Python we need to keep in mind that they allow duplicate entries. What if it matters whether our data contains duplicates or not? This is where Python Sets come in. They are like Lists but they do not allow any duplicates to be stored in them. Sets are also used to efficiently remove duplicates from Lists and are faster than creating a new list and populating it from the one with duplicates. In this operation, you can think of them as a funnel or filter that holds back duplicates and only lets unique values pass. Let us compare the two operations: import timeit # here we create a new list and add the elements one by one # while checking for duplicates def manual_remove_duplicates(list_of_duplicates): new_list = [] [new_list.append(n) for n in list_of_duplicates if n not in new_list] return new_list # using a set is as simple as def set_remove_duplicates(list_of_duplicates): return list(set(list_of_duplicates)) list_of_duplicates = [10, 54, 76, 10, 54, 100, 1991, 6782, 1991, 1991, 64, 10] print("Manually removing duplicates takes {}s".format(timeit.timeit('manual_remove_duplicates(list_of_duplicates)', 'from __main__ import manual_remove_duplicates, list_of_duplicates'))) print("Using Set to remove duplicates takes {}s".format(timeit.timeit('set_remove_duplicates(list_of_duplicates)', 'from __main__ import set_remove_duplicates, list_of_duplicates'))) After running the script five times: $ python sets-vs-lists.py Manually removing duplicates takes 2.64614701271s Using Set to remove duplicates takes 2.23225092888s $ $ python sets-vs-lists.py Manually removing duplicates takes 2.65356898308s Using Set to remove duplicates takes 1.1165189743s $ $ python sets-vs-lists.py Manually removing duplicates takes 2.53129696846s Using Set to remove duplicates takes 1.15646100044s $ $ python sets-vs-lists.py Manually removing duplicates takes 2.57102680206s Using Set to remove duplicates takes 1.13189387321s $ $ python sets-vs-lists.py Manually removing duplicates takes 2.48338890076s Using Set to remove duplicates takes 1.20611810684s Using a Set to remove duplicates is consistently faster than manually creating a list and adding items while checking for presence. This could be useful when filtering entries for a giveaway contest, where we should filter out duplicate entries. If it takes 2s to filter out 120 entries, imagine filtering out 10 000 entries. On such a scale, the vastly increased performance that comes with Sets is significant. This might not occur commonly, but it can make a huge difference when called upon. Proper profiling can help us identify such situations, and can make all the difference in the performance of our code. String Concatenation Strings are immutable by default in Python and subsequently, String concatenation can be quite slow. There are several ways of concatenating strings that apply to various situations. We can use the + (plus) to join strings. This is ideal for a few String objects and not at scale. If you use the + operator to concatenate multiple strings, each concatenation will create a new object since Strings are immutable. This will result in the creation of many new String objects in memory hence improper utilization of memory. We can also use the concatenate operator += to join strings but this only works for two strings at a time, unlike the + operator that can join more than two strings. If we have an iterator such as a List that has multiple Strings, the ideal way to concatenate them is by using the .join() method. Let us create a list of a thousand words and compare how the .join() and the += operator compare: import timeit # create a list of 1000 words list_of_words = ["foo "] * 1000 def using_join(list_of_words): return "".join(list_of_words) def using_concat_operator(list_of_words): final_string = "" for i in list_of_words: final_string += i return final_string print("Using join() takes {} s".format(timeit.timeit('using_join(list_of_words)', 'from __main__ import using_join, list_of_words'))) print("Using += takes {} s".format(timeit.timeit('using_concat_operator(list_of_words)', 'from __main__ import using_concat_operator, list_of_words'))) After two tries: $ python join-vs-concat.py Using join() takes 14.0949640274 s Using += takes 79.5631570816 s $ $ python join-vs-concat.py Using join() takes 13.3542580605 s Using += takes 76.3233859539 s It is evident that the .join() method is not only neater and more readable, but it is also significantly faster than the concatenation operator when joining Strings in an iterator. If you're performing a lot of String concatenation operations, enjoying the benefits of an approach that's almost 7 times faster is wonderful. Conclusion We have established that the optimization of code is crucial in Python and also saw the difference made as it scales. Through the Timeit module and cProfile profiler, we have been able to tell which implementation takes less time to execute and backed it up with the figures. The data structures and control flow structures we use can greatly affect the performance of our code and we should be more careful. Profiling is also a crucial step in code optimization since it guides the optimization process and makes it more accurate. We need to be sure that our code works and is correct before optimizing it to avoid premature optimization which might end up being more expensive to maintain or will make the code hard to understand.