Feeds

Dependency management in Python has taken a long and winding path, which has led to the current dominance of Pip. One of the remaining shortcomings is the lack of a robust mechanism for resolving the package and version constraints that are necessary to produce a working system. Thankfully, the Python Software Foundation has funded an effort to upgrade the dependency resolution algorithm and user experience of Pip. In this episode the engineers working on these improvements, Pradyun Gedam, Tzu-Ping Chung, and Paul Moore, discuss the history of Pip, the challenges of dependency management in Python, and the benefits that surrounding projects will gain from a more robust resolution algorithm. This is an exciting development for the Python ecosystem, so listen now and then provide feedback on how the new resolver is working for you.Summary Dependency management in Python has taken a long and winding path, which has led to the current dominance of Pip. One of the remaining shortcomings is the lack of a robust mechanism for resolving the package and version constraints that are necessary to produce a working system. Thankfully, the Python Software Foundation has funded an effort to upgrade the dependency resolution algorithm and user experience of Pip. In this episode the engineers working on these improvements, Pradyun Gedam, Tzu-Ping Chung, and Paul Moore, discuss the history of Pip, the challenges of dependency management in Python, and the benefits that surrounding projects will gain from a more robust resolution algorithm. This is an exciting development for the Python ecosystem, so listen now and then provide feedback on how the new resolver is working for you. Announcements Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great. When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With 200 Gbit/s private networking, node balancers, a 40 Gbit/s public network, fast object storage, and a brand new managed Kubernetes platform, all controlled by a convenient API you’ve got everything you need to scale up. And for your tasks that need fast computation, such as training machine learning models, they’ve got dedicated CPU and GPU instances. Go to pythonpodcast.com/linode to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show! You listen to this show because you love Python and want to keep your skills up to date, and machine learning is finding its way into every aspect of software engineering. Springboard has partnered with us to help you take the next step in your career by offering a scholarship to their Machine Learning Engineering career track program. In this online, project-based course every student is paired with a Machine Learning expert who provides unlimited 1:1 mentorship support throughout the program via video conferences. You’ll build up your portfolio of machine learning projects and gain hands-on experience in writing machine learning algorithms, deploying models into production, and managing the lifecycle of a deep learning prototype. Springboard offers a job guarantee, meaning that you don’t have to pay for the program until you get a job in the space. Podcast.__init__ is exclusively offering listeners 20 scholarships of $500 to eligible applicants. It only takes 10 minutes and there’s no obligation. Go to pythonpodcast.com/springboard and apply today! Make sure to use the code AISPRINGBOARD when you enroll. Your host as usual is Tobias Macey and today I’m interviewing Tzu-ping Chung, Pradyun Gedam, and Paul Moore about their work to improve the dependency resolution capabilities of Pip and its user experience Interview Introductions How did you get introduced to Python? Can you start by describing the focus of the work that you are doing? What is the scope of the work, and what is the established criteria for when it is considered complete? What is your history with working on the Pip source code and what interests you most about this project? What are the main sources or manifestations of technical debt that exist in Pip as of today? How does it currently handle dependency resolution? What are some of the workarounds that developers have had to resort to in the absence of a robust dependency resolver in Pip? How is the new dependency resolver implemented? How has your initial design evolved or shifted as you have gotten further along in its implementation? What are the pieces of information that the resolver will rely on for determining which packages and versions to install? (e.g. will it install setuptools > 45.x in a Python 2 virtualenv?) What are the new capabilities in Pip that will be enabled by this upgrade to the dependency resolver? What projects or features in the encompassing ecosystem will be unblocked with the introduction of this upgrade? What are some of the changes that users will need to make to adopt the updated Pip? How do you anticipate the changes in Pip impacting the viability or adoption of Python and its ecosystem within different communities or industries? What are some of the additional changes or improvements that you would like to see in Pip or other core elements of the Python landscape? What are some of the most interesting, unexpected, or challenging lessons that you have learned while working on these updates to Pip? Keep In Touch Pradyun Website pradyunsg on GitHub @pradyunsg on Twitter Paul pfmoore on GitHub Tzu-Ping uranusjr on GitHub Website @uranusjr on Twitter Picks Tzu-ping Python Launcher Joe Abercrombie author The Shattered Sea Trilogy Anime PipX Standalone Paul pipx Black nox tox scoop Neil Gaiman Good Omens Book TV Series Pradyun because my picks can be anything — things that have kept me sane in this lockdown world Music: Chris Daughtry Video Game: Parkitect Tobias Language Server Protocol Emacs lsp-mode Closing Announcements Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@podcastinit.com) with your story. To help other people find the show please leave a review on iTunes and tell your friends and co-workers Join the community in the new Zulip chat workspace at pythonpodcast.com/chat Links Pip Podcast interview with Donald Stufft Macdown Taiwan Pipenv PyPI Podcast Episode TOML Python Package Metadata Standards iBook G4 Acorn Computer distutils easy_install Python Eggs setuptools Python Wheels CPAN Conda Inside The Cheeseshop Google Summer of Code Zazo PEP517 pip-tools Poetry resolvelib SAT Solver Trove Classifiers PyPA pyproject.toml The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

“Keyword-only arguments” is a Python feature that was added in version 3.0. But I’ve seen and used it much less use than I could have. It is described in PEP 3102, which is pretty readable, but I think the feature could benefit from more exposure with examples and rationale. To understand keyword-only arguments, we first have to clear up a common misunderstanding about Python positional and keyword arguments. Consider the following function: def print_greeting(name, use_colors=False): # ... We often think of this as having one positional argument and one keyword argument, and expect that calling code will use it like this: print_greeting("Mary") print_greeting("Mary", use_colors=True) print_greeting("Mary", use_colors=False) # redundant but explicit In fact, you can also call this function like the following (although it usually surprises me if I see these forms): print_greeting("Mary", True) print_greeting(name="Mary") print_greeting(use_colors=True, name="Mary") In other words, both arguments can be passed positionally or by name. Technically you should think about positional or keyword arguments as being defined by the function call and not the function definition. (If you know your computer science terminology, the function definition defines parameters, while it is the function call that defines arguments, which has now confused me about whether the PEP should really have been titled “Keyword-only parameters”…) The actual distinction between name and use_colors is that use_colors has a default argument value, while name does not. This does introduce limitations in how you can define and how you can call them — for example, the following is illegal: print_greeting(use_colors=True, "Mary") Having got that cleared up, we can now separate in our minds two different concerns: When people call our function, should we encourage/expect/force them to use a keyword argument, or a positional one? Do we have a good default for each of the parameters? With Python’s keyword-only arguments we can address these two independently. To make it impossible to pass use_colors positionally, we can define it like this: def print_greeting(name, *, use_colors=False): # ... This brings some advantages: It makes the calling code more explicit (but also more verbose). It will no longer be possible to see code like: print_greeting("Mary", True) …and wonder “what does that boolean argument mean?” It allows us as function authors to add more keyword parameters, defined in any order we want, without the possibility of breaking things for people using our functions because they were were passing arguments positionally instead of by name. Once we have these keyword-only arguments, we can also ask “do we actually have good default values?” Let’s say we realise that the use of color codes is something that is very important and must be passed every time our print_greeting function is used — we don’t have a good default. We can write our function like this: def print_greeting(name, *, use_colors): pass And call it: print_greeting("Mary", use_colors=True) If the caller fails to pass use_colors, they see this error message: TypeError: print_greeting() missing 1 required keyword-only argument: 'use_colors' I haven’t seen keyword-only arguments that much in the wild, which I think is a shame. But now that Python 2 is much less of a concern for me, I’m going to be making much more use of this feature. For further reading — the twin of this is positional only parameters, a more recent feature that makes it impossible to pass an argument by name.

Since my last post on Rich there have been a number of improvements. Here are the major highlights: Indeterminate progress bars Progress bars now have an indeterminate state which displays a pulsing animation. This a fairly common thing from graphical UIs, here's what it looks like in the terminal: To see it in action install Rich and run the following: python -m rich.progress Or just watch this video. You can use this new state to show when something is working, but before you know how many 'steps' are involved. For instance, when making a request over the internet, you won't know how many bytes you have to download until you get a response. Rather than show a progress bar at 0% you can show this pulse effect while waiting on the server. 16.7 million colours in Windows Terminal Rich now detects the new Windows Terminal and enables truecolor support. Previously colours were downgraded to 8-bit, but now you can enjoy the full 16.7 million colours in Windows. More text styles I've added support for the following terminal styles: double underline, framed, encircled, and overlined. Of the new 4 styles, I've only see double underline and overline supported, and only on Ubuntu terminal. The lack of support is why they didn't go in version 1.0, but if your terminal supports these new styles, you're in luck. I don't even know what frame and encircled is supposed to look like. Let me know if you are using a terminal that supports them (send me a screenshot). © 2020 Will McGugan The overline and double underline terminal styles on Ubuntu. Console markup abbreviations Rich supported abbreviate styles in markup, for instance "[b]Bold[/b]", but only a single abbreviated style could be applied in a tag. If you wanted bold and italic you would have to write "[b][i]Bold and italic[/i][/b]". Now you can write "[b i]Bold and italic[/b i]". Fixed A number of minor fixes and one one serious. There was a possibility of a deadlock in progress bars. If you are using progress bars, please upgrade. Optimizations I've made a number of optimizations in rendering styles which have made Rich faster across the board. You're unlikely to notice anything unless you are generating many pages of rich text (perhaps with a large markdown file). Coverage Coverage has reached 97% which is not bad at all. To be honest though it is the use of type annotations throughout which gives me the most confidence.

The PDF, or Portable Document Format, is one of the most common formats for sharing documents over the Internet. PDFs can contain text, images, tables, forms, and rich media like videos and animations, all in a single file. This abundance of content types can make working with PDFs difficult. There are a lot of different kinds of data to decode when opening a PDF file! Fortunately, the Python ecosystem has some great packages for reading, manipulating, and creating PDF files. In this tutorial, you’ll learn how to: Read text from a PDF Split a PDF into multiple files Concatenate and merge PDF files Rotate and crop pages in a PDF file Encrypt and decrypt PDF files with passwords Create a PDF file from scratch Note: This tutorial is adapted from the chapter “Creating and Modifying PDF Files” in Python Basics: A Practical Introduction to Python 3. The book uses Python’s built-in IDLE editor to create and edit Python files and interact with the Python shell, so you will see occasional references to IDLE throughout this article. However, you should have no problems running the example code from the editor and environment of your choice. Along the way, you’ll have several opportunities to deepen your understanding by following along with the examples. You can download the materials used in the examples by clicking on the link below: Download the sample materials: Click here to get the materials you'll use to learn about creating and modifying PDF files in this tutorial. Extracting Text From a PDF In this section, you’ll learn how to read a PDF file and extract the text using the PyPDF2 package. Before you can do that, though, you need to install it with pip: $ python3 -m pip install PyPDF2 Verify the installation by running the following command in your terminal: $ python3 -m pip show PyPDF2 Name: PyPDF2 Version: 1.26.0 Summary: PDF toolkit Home-page: http://mstamy2.github.com/PyPDF2 Author: Mathieu Fenniak Author-email: biziqe@mathieu.fenniak.net License: UNKNOWN Location: c:\\users\\david\\python38-32\\lib\\site-packages Requires: Required-by: Pay particular attention to the version information. At the time of writing, the latest version of PyPDF2 was 1.26.0. If you have IDLE open, then you’ll need to restart it before you can use the PyPDF2 package. Opening a PDF File Let’s get started by opening a PDF and reading some information about it. You’ll use the Pride_and_Prejudice.pdf file located in the practice_files/ folder in the companion repository. Open IDLE’s interactive window and import the PdfFileReader class from the PyPDF2 package: >>>>>> from PyPDF2 import PdfFileReader To create a new instance of the PdfFileReader class, you’ll need the path to the PDF file that you want to open. Let’s get that now using the pathlib module: >>>>>> from pathlib import Path >>> pdf_path = ( ... Path.home() ... / "creating-and-modifying-pdfs" ... / "practice_files" ... / "Pride_and_Prejudice.pdf" ... ) The pdf_path variable now contains the path to a PDF version of Jane Austen’s Pride and Prejudice. Note: You may need to change pdf_path so that it corresponds to the location of the creating-and-modifying-pdfs/ folder on your computer. Now create the PdfFileReader instance: >>>>>> pdf = PdfFileReader(str(pdf_path)) You convert pdf_path to a string because PdfFileReader doesn’t know how to read from a pathlib.Path object. Recall from chapter 12, “File Input and Output,” that all open files should be closed before a program terminates. The PdfFileReader object does all of this for you, so you don’t need to worry about opening or closing the PDF file! Read the full article at https://realpython.com/creating-modifying-pdf/ » [ Improve Your Python With ? Python Tricks ? – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Introduction In this article, we'll be diving into the Factory Method Design Pattern, implemented in Python. Design Patterns define tried and tested solutions to various recurring problems in software development. They do not represent actual code, but rather ways in which we can organize our code for the optimum results. In a world of limited resources, Design Patterns help us achieve the most results with the least amount of used resources. It is also important to note that Design Patterns do not apply to all situations and it is crucial to assess the problem at hand in order to choose the best approach for that particular scenario. Design Patterns are divided into a few broad categories, though mainly into Creational Patterns, Structural Patterns, and Behavioral Patterns. The Factory Method pattern is a Creational Design Pattern. The Factory Method Design Pattern Definition The Factory Method is used in object-oriented programming as a means to provide factory interfaces for creating objects. These interfaces define the generic structure, but don't initialize objects. The initialization is left to more specific subclasses. The parent class/interface houses all the standard and generic behaviour that can be shared across subclasses of different types. The subclass is in turn responsible for the definition and instantiation of the object based on the superclass. Motivation The main motivation behind the Factory Method Design Pattern is to enhance loose coupling in code through the creation of an abstract class that will be used to create different types of objects that share some common attributes and functionality. This results in increased flexibility and reuse of code because the shared functionality will not be rewritten having been inherited from the same class. This design pattern is also known as a Virtual Constructor. The Factory Method design pattern is commonly used in libraries by allowing clients to choose what subclass or type of object to create through an abstract class. A Factory Method will receive information about a required object, instantiate it and return the object of the specified type. This gives our application or library a single point of interaction with other programs or pieces of code, thereby encapsulating our object creation functionality. Factory Method Implementation Our program is going to be a library used for handling shape objects in terms of creation and other operations such as adding color and calculating the area of the shape. Users should be able to use our library to create new objects. We can start by creating single individual shapes and availing them as is but that would mean that a lot of shared logic will have to be rewritten for each and every shape we have available. The first step to solving this repetition would be to create a parent shape class that has methods such as calculate_area() and calculate_perimeter(), and properties such as dimensions. The specific shape objects will then inherit from our base class. To create a shape, we will need to identify what kind of shape is required and create the subclass for it. We will start by creating an abstract class to represent a generic shape: import abc class Shape(metaclass=abc.ABCMeta): @abc.abstractmethod def calculate_area(self): pass @abc.abstractmethod def calculate_perimeter(self): pass This is the base class for all of our shapes. Let's go ahead and create several concrete, more specific shapes: class Rectangle(Shape): def __init__(self, height, width): self.height = height self.width = width def calculate_area(self): return self.height * self.width def calculate_perimeter(self): return 2 * (self.height + self.width) class Square(Shape): def __init__(self, width): self.width = width def calculate_area(self): return self.width ** 2 def calculate_perimeter(self): return 4 * self.width class Circle(Shape): def __init__(self, radius): self.radius = radius def calculate_area(self): return 3.14 * self.radius * self.radius def calculate_perimeter(self): return 2 * 3.14 * self.radius So far, we have created an abstract class and extended it to suit different shapes that will be available in our library. In order to create the different shape objects, clients will have to know the names and details of our shapes and separately perform the creation. This is where the Factory Method comes into play. The Factory Method design pattern will help us abstract the available shapes from the client, i.e. the client does not have to know all the shapes available, but rather only create what they need during runtime. It will also allow us to centralize and encapsulate the object creation. Let us achieve this by creating a ShapeFactory that will be used to create the specific shape classes based on the client's input: class ShapeFactory: def create_shape(self, name): if name == 'circle': radius = input("Enter the radius of the circle: ") return Circle(float(radius)) elif name == 'rectangle': height = input("Enter the height of the rectangle: ") width = input("Enter the width of the rectangle: ") return Rectangle(int(height), int(width)) elif name == 'square': width = input("Enter the width of the square: ") return Square(int(width)) This is our interface for creation. We don't call the constructors of concrete classes, we call the Factory and ask it to create a shape. Our ShapeFactory works by receiving information about a shape such as a name and the required dimensions. Our factory method create_shape() will then be used to create and return ready objects of the desired shapes. The client doesn't have to know anything about the object creation or specifics. Using the factory object, they can create objects with minimal knowledge of how they work: def shapes_client(): shape_factory = ShapeFactory() shape_name = input("Enter the name of the shape: ") shape = shape_factory.create_shape(shape_name) print(f"The type of object created: {type(shape)}") print(f"The area of the {shape_name} is: {shape.calculate_area()}") print(f"The perimeter of the {shape_name} is: {shape.calculate_perimeter()}") Running this code will result in: Enter the name of the shape: circle Enter the radius of the circle: 7 The type of object created: <class '__main__.Circle'> The area of the circle is: 153.86 The perimeter of the circle is: 43.96 Or, we could build another shape: Enter the name of the shape: square Enter the width of the square: 5 The type of object created: <class '__main__.Square'> The area of the square is: 25 The perimeter of the square is: 20 What's worth noting is that besides the client not having to know much about the creation process - when we'd like to instantiate an object, we don't call the constructor of the class. We ask the factory to do this for us based on the info we pass to the create_shape() function. Pros and Cons Pros One of the major advantages of using the Factory Method design pattern is that our code becomes loosely coupled in that the majority of the components of our code are unaware of other components of the same codebase. This results in code that is easy to understand and test and add more functionality to specific components without affecting or breaking the entire program. The Factory Method design pattern also helps uphold the Single Responsibility Principle where classes and objects that handle specific functionality resulting in better code. Cons Creation of more classes eventually leads to less readability. If combined with an Abstract Factory (factory of factories), the code will soon become verbose, though, maintainable. Conclusion In conclusion, the Factory Method Design Pattern allows us to create objects without specifying the exact class required to create the particular object. This allows us to decouple our code and enhances its reusability. It is important to note that, just like any other design pattern, it is only suitable for specific situations and not every development scenario. An assessment of the situation at hand is crucial before deciding to implement the Factory Method Design Pattern to reap the benefits of the pattern.

<meta charset="utf-8">The optimization process of deep learning models is based on the gradient descent method. Deep learning frameworks such as PyTorch and Tensorflow can be divided into three parts: model api, gradient calculation and gpu acceleration. Gradient calculation plays an important role, and the core technology of this part is automatic differentiation. 1 Differential method There are four differential methods: Manual differentiation Numerical differentiation Sign differential Automatic differentiation   Manual differentiation is to use the derivation formula to manually write the derivation formula. This method is accurate and effective, and the only disadvantage is that it takes effort. Numerical differentiation uses the definition of derivative:   This method is simple to implement, but there are two serious problems: truncation error and roundoff error. But this method can be a good way to check whether the gradient is accurate. Another method is symbolic differentiation, which transfers the work we did in manual differentiation to the computer. The problem with this method is that the expression must be closed-form, that is, there cannot be loops and conditional expressions. So that the entire problem can be converted into a pure mathematical symbol problem can be solved using some algebraic software. However, when expressions are complex, the problem of "expression swell" is prone to occur. The last is our protagonist: automatic differentiation. It is also the most widely used derivation method in programe. 2 Automatic differentiation The automatic differentiation discovers the essence of differential calculation: Differential calculation is a combination of a limited series of differentiable operators. We can regarded the formula as a calculation graph (What’s more, it can be regarded as a tree structure, too). In the process of forward calculation, we can obtain the value of each node. Then we can express the derivation process of    as follows: It can be seen that the whole derivation can be split into a series of differential operator combinations. The calculation can be divided into two types: calculating the formula from forward to backward is called Forward Mode, and calculating the formula from backward to forward is called Reverse Mode. The process of the two modes is expressed as follows: The gradient values ​​calculated by the two modes are the same, but for the calculation order is different, the calculation speed is different. Generally, if the Jacobian matrix is relatively high, then the forward mode is more efficient; if the Jacobian matrix is ​​wider, then the reverse mode is more efficient. 3 JVP, VJP and vmap If you have used pytorch, you will find that if y is a tensor instead of a scalar, you will be asked to pass a  grad_variables in y.backward(). And the derivative result x.grad has the same shape as x. Where is the Jacobian matrix? The reason is that deep learning frameworks such as Tensorflow and PyTorch prohibit the derivatives with tensor by tensor, but only retain scalar by tensor. When we call y.backward () and enter a grad_variables . In fact, it actually converts y into a weighted sum l = torch.sum(y * v) , where l is a scalar, and then the gradient of x.grad is naturally of the same shape as x. The reason for this is that the loss of deep learning is definitely a scalar, and gradient descent requires that the gradient must be of the same type as x. But what if we want to obtain the Jacobian matrix? The answer is to derive x for each value of y.In addition, Google ’s new deep learning framework JAX uses a more advanced method, the vectorization operation vmap to speed up the calculation.  4 Reference The Autodiff Cookbook Automatic Differentiation in Machine Learning: a Survey AUTOGRAD: AUTOMATIC DIFFERENTIATION  

Demystifying Python Regular Expression