Feeds

If I mention “decorators,” even if you’ve been programming with Python for a while, what comes to mind? To many, decorators are mysterious and powerful — and beyond the reach of mere mortals. Their @ signs pop up in all sorts of places, from “pytest” to Flask to properties… but it’s not really clear what they are, what they’re doing, how they work, or how they can be useful. Here’s the thing: If you are comfortable writing functions and classes in Python, then you can write decorators. And once you start, you’ll wonder how you lived without them. I’ve taught decorators to engineers at some of the world’s largest companies, and also at conferences like PyCon and Euro Python. And this Sunday, I’ll be teaching a live, online course all about decorators. Once you’re done with the course, you’ll be able to write and understand a wide variety of decorators, from those that filter arguments to those that cache function results and monitor your code’s performance. I’m running these class like I do my corporate training: With lots of time for Q&A, and a ton of hands-on exercises. Without slides, as I live-code into Jupyter. We’ll even have a private forum, so you can ask questions when the class is done. And of course, you’ll have access to the video recording. If you want to finally understand decorators, then you want this class.  Sign up here: https://store.lerner.co.il/live-class-decorators Questions? Comments? Eligible for a discount (for students, retirees/pensioners, or people living outside the world’s 30 richest countries)? E-mail me at reuven@lerner.co.il or get me on Twitter as @reuvenmlerner, and I’ll answer ASAP. The post Stop fearing Python decorators appeared first on Reuven Lerner.

The third PyCharm 2021.3 EAP build is out! Please give it a try, and don’t forget to share your feedback on Twitter (by mentioning @pycharm) or by reporting issues to our tracker! Important: PyCharm EAP builds are not fully tested and might be unstable. Starting from this build, you’ll need to provide your JetBrains credentials to use the trial versions of PyCharm. You can read more about the change in this blog post. DOWNLOAD PYCHARM 2021.3 EAP Poetry support One of the most popular issues in our tracker is getting closer to being marked as “fixed”, as this EAP build bundles Poetry support out of the box. We’ve also bundled the TOML plugin so you will also get code insight for your pyproject.toml file out of the box. Kudos to Koudai Aono, who developed the original plugin, which is now being fine-tuned and will become available for all PyCharm users by default. Endpoints tool window for FastAPI and Flask projects If you develop web applications using FastAPI and Flask, you likely work with endpoints. Contrary to the name, endpoints are where your work as a Python developer often begins. Although working with them is generally simple in minor projects, it can become cumbersome in larger ones. Our team is always looking for ways to make you more productive and remove roadblocks from your workflow. This is why we’ve made the Endpoints tool window default for FastAPI and Flask projects. When you start working on a new or existing project, PyCharm will scan its routes and list them as endpoints in a dedicated tool window. You’ll also get code completion and refactoring capabilities for your URLs. The tool window will present relevant documentation, and you’ll have full navigation capabilities, including the ability to go to an endpoint’s declaration and to find usages of it across your project. You’ll also have quick access to the HTTP Client, an ad hoc tool that allows you to create requests and receive their results quickly for testing purposes. Bug fixes and improvements Fixed an issue that was causing the UI to freeze on macOS. [IDEA-274712] Fixed a problem that was causing tabs to close incorrectly when multiple tabs were open. [IDEA-274154] Fixed a bug that was causing the Pytest runner to miss a part of the test output when using assertEqual. [PY-30212] Fixed a bug that made it impossible for users to re-run parametrized pytests with keywords in a run configuration template. [PY-41848] Ready to join the EAP? Some ground rules EAP builds are free to use and expire 30 days after the build date. You can install an EAP build side by side with your stable PyCharm version. These builds are not fully tested and can be unstable. How to download You can download this EAP build from our website. Alternatively, you can use the JetBrains Toolbox App to stay up to date throughout the entire EAP. If you’re on Ubuntu 16.04 or later, you can use snaps to make sure you always have the latest PyCharm EAP build. The PyCharm team

I'm happy to announce that I just released qutebrowser v2.4.0! This release fixes a high-severity arbitrary command execution on Windows via URL handlers, see the security advisory and commit message for details. Windows users are urged to update as soon as possible. For everyone else, this is a …

The np.any() function tests whether any element in a NumPy array evaluates to true: np.any(np.array([[1, 0], [0, 0]])) # Expected result # True The input can have any shape and the data type does not have to be boolean (as long as it’s truthy). If none of the elements evaluate to true, the function returns false: np.any(np.array([[0, 0], [0, 0]])) # Expected result # False Passing in a value for the axis argument makes np.any() a reducing operation. Say we want to know which rows in a matrix have any truthy elements. We can do that by passing in axis=-1: np.any(np.zeros((2, 3)), axis=-1) # Expected result # array([False, False]) There are two rows and for each of them, none of the elements evaluate to true. The -1 value here is shorthand for “the last axis”. Easy enough! NumPy also has a function called np.all() which has the same API as np.any() but returns true when all of the elements evaluate to true. The post NumPy Any: Understanding np.any() appeared first on Sparrow Computing.

TorchVision, a PyTorch computer vision package, has a simple API for image pre-processing in its torchvision.transforms module. The module contains a set of common, composable image transforms and gives you an easy way to write new custom transforms. As you would expect, these custom transforms can be included in your pre-processing pipeline like any other transform from the module. Let’s start with a common use case, preparing PIL images for one of the pre-trained TorchVision image classifiers: import io import requests import torchvision.transforms as T from PIL import Image resp = requests.get('https://sparrow.dev/assets/img/cat.jpg') img = Image.open(io.BytesIO(resp.content)) preprocess = T.Compose([ T.Resize(256), T.CenterCrop(224), T.ToTensor(), T.Normalize( mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225] ) ]) x = preprocess(img) x.shape # Expected result # torch.Size([3, 224, 224]) Here, we apply the following in order: Resize a PIL image to (<height>, 256), where <height> is the value that maintains the aspect ratio of the input image.Crop the (224, 224) center pixels.Convert the PIL image to a PyTorch tensor (which also moves the channel dimension to the beginning).Normalize the image by subtracting a known ImageNet mean and standard deviation. Let’s go a notch deeper to understand exactly how these transforms work. Transforms TorchVision transforms are extremely flexible – there are just a few rules. In order to be composable, transforms need to be callables. That means you can actually just use lambdas if you want: times_2_plus_1 = T.Compose([ lambda x: x * 2, lambda x: x + 1, ]) x.mean(), times_2_plus_1(x).mean() # Expected result # (tensor(1.2491), tensor(3.4982)) But often, you’ll want to use callable classes because they give you a nice way to parameterize the transform at initialization. For example, if you know you want to resize images to have height of 256 you can instantiate the T.Resize transform with a 256 as input to the constructor: resize_callable = T.Resize(256) Any PIL image passed to resize_callable() will now get resized to (<height>, 256): resize_callable(img).size # Expected result # (385, 256) This behavior is important because you will typically want TorchVision or PyTorch to be responsible for calling the transform on an input. We actually saw this in the first example: the component transforms (Resize, CenterCrop, ToTensor, and Normalize) were chained and called inside the Compose transform. And the calling code would not have knowledge of things like the size of the output image you want or the mean and standard deviation for normalization. Interestingly, there is no Transform base class. Some transforms have no parent class at all and some inherit from torch.nn.Module. This means that if you’re writing a transform class, the constructor can do whatever you want. The only requirement is that there must be a __call__() method to ensure the instantiated object is callable. Note: when transforms override the torch.nn.Module class, they will typically define the forward() method and then the base class takes care of __call__(). Additionally, there are no real constraints on the callable’s inputs or outputs. A few examples: T.Resize: PIL image in, PIL image out.T.ToTensor: PIL image in, PyTorch tensor out.T.Normalize: PyTorch tensor in, PyTorch tensor out. NumPy arrays may also be a good choice sometimes. Ok. Now that we know a little about what transforms are, let’s look at an example that TorchVision gives us out of the box. Example Transform: Compose The T.Compose transform takes a list of other transforms in the constructor and applies them sequentially to the input. We can take a look at the __init__() and __call__() methods from a recent commit hash to see how this works: class Compose: def __init__(self, transforms): self.transforms = transforms def __call__(self, img): for t in self.transforms: img = t(img) return img Very simple! You can pass the T.Compose constructor a list (or any other in-memory sequence) of callables and it will dutifully apply them to any input one at a time. And notice that the input img can be any type you want. In the first example, the input was PIL and the output was a PyTorch tensor. In the second example, the input and output were both tensors. T.Compose doesn’t care! Let’s instantiate a new T.Compose transform that will let us visualize PyTorch tensors. Remember, we took a PIL image and generated a PyTorch tensor that’s ready for inference in a TorchVision classifier. Let’s take a PyTorch tensor from that transformation and convert it into an RGB NumPy array that we can plot with Matplotlib: %matplotlib inline import matplotlib.pyplot as plt import numpy as np reverse_preprocess = T.Compose([ T.ToPILImage(), np.array, ]) plt.imshow(reverse_preprocess(x)); The T.ToPILImage transform converts the PyTorch tensor to a PIL image with the channel dimension at the end and scales the pixel values up to int8. Then, since we can pass any callable into T.Compose, we pass in the np.array() constructor to convert the PIL image to NumPy. Not too bad! Functional Transforms As we’ve now seen, not all TorchVision transforms are callable classes. In fact, TorchVision comes with a bunch of nice functional transforms that you’re free to use. If you look at the torchvision.transforms code, you’ll see that almost all of the real work is being passed off to functional transforms. For example, here’s the functional version of the resize logic we’ve already seen: import torchvision.transforms.functional as F F.resize(img, 256).size # Expected result # (385, 256) It does the same work, but you have to pass additional arguments in when you call it. My advice: use functional transforms for writing custom transform classes, but in your pre-processing logic, use callable classes or single-argument functions that you can compose. At this point, we know enough about TorchVision transforms to write one of our own. Custom Transforms Let’s write a custom transform that erases the top left corner of an image with the color of a randomly selected pixel. We’ll use the F.erase() function and we’ll allow the caller to specify what how many pixels they want to erase in both directions: import torch class TopLeftCornerErase: def __init__(self, n_pixels: int): self.n_pixels = n_pixels def __call__(self, img: torch.Tensor) -> torch.Tensor: all_pixels = img.reshape(3, -1).transpose(1, 0) idx = torch.randint(len(all_pixels), (1,))[0] random_pixel = all_pixels[idx][:, None, None] return F.erase(img, 0, 0, self.n_pixels, self.n_pixels, random_pixel) In the constructor, all we do is take the number of pixels as a parameter from the caller. The magic happens in the __call__() method: Create a reshaped view of the image tensor as a (n_pixels, 3) tensorRandomly select a pixel index using torch.randint()Add two dummy dimensions to the tensor. This is because F.erase() and to the image, which has these two dimensions.Call and return F.erase(), which takes five arguments: the tensor, the i coordinate to start at, the j coordinate to start at, the height of the box to erase, the width of the box to erase and the random pixel. We can apply this custom transform just like any other transform. Let’s use T.Compose to both apply this erase transform and then convert it to NumPy for plotting: torch.manual_seed(1) erase = T.Compose([ TopLeftCornerErase(100), reverse_preprocess, ]) plt.imshow(erase(x)); We’ve seen this type of transform composition multiple times now. One thing that is important to point out is that you need to call torch.manual_seed() if you want a deterministic (and therefore reproducible) result for any TorchVision transform that has random behavior in it. This is new as of version 0.8.0. And that’s about all there is to know about TorchVision transforms! They’re lightweight and flexible, but using them will make your image preprocessing code much easier to reason about. The post TorchVision Transforms: Image Preprocessing in PyTorch appeared first on Sparrow Computing.

When you’re building a production machine learning system, reproducibility is a proxy for the effectiveness of your development process. But without locking all your Python dependencies, your builds are not actually repeatable. If you work in a Python project without locking long enough, you will eventually get a broken build because of a transitive dependency (that is, a dependency of a dependency). But a broken build isn’t the most dangerous problem. A transitive dependency can change in a way that affects an algorithm’s results without you ever knowing it. Imagine being unable to reproduce an algorithm result because an ancestor dependency that never got pinned changes in an unexpected way. Not good. Enter Poetry. Poetry is an environment management tool for Python projects. Admittedly, there are a lot of environment management tools for Python and there’s no guarantee that Poetry will win. But the Python development community does seem to be slowly standardizing around Poetry. What is Poetry? Virtual environment tools like virtualenv and conda have been an important part of modern Python development for a while. But there are a few problems: Different tools and approaches don’t play nicely with each otherThe standards for configuration are inconsistentDependency resolution can fail in mysterious ways Ideally, you want all your builds to be precisely repeatable and you want a straightforward solution that you can use across all your projects. This is where Poetry shines. Think of Poetry like npm for Python. It’s centered around pyproject.toml, a single config file that defines your entire Python environment and it comes with a lock file, poetry.lock, which pins all dependencies (including transitive dependencies). Poetry configuration Poetry extends the pyproject.toml file (introduced in PEP 517) to specify build requirements, project dependencies and development dependencies. It also includes project-level configuration (things like the name, description and license), scripts, extra dependencies and more. Importantly, the Poetry spec for pyproject.toml works for both Python packages and Python applications, which prevents you from using a different approach across Python projects. This is important for machine learning development because reusable utilities like config and basic calculations like bounding box math often fit better in Python packages than in your actual machine learning application. Lock file The poetry.lock file is similar in concept to running pip freeze > requirements.txt every time you add or remove a package. It’s just that Poetry manages this for you when you add or remove a dependency. And Poetry comes with a good dependency resolver. Since some packages on PyPI don’t specify all dependencies in their metadata, it can make adding a new package a little slower in Poetry. But it does ensure that you won’t get broken builds down the road as dependencies evolve, which is important. Quick Start To install the latest Poetry into your Python environment, run: curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python After installation, the script will print instructions to your screen to add poetry to your path. To initialize a new project, run: poetry init This will guide you interactively through the process of setting up a new Python project with Poetry. You will add information about the project, the version of Python to use and (optionally) dependencies and development dependencies. Say you want to add a dependency, like NumPy, after your project has been initialized. You can do that with the poetry add command: poetry add numpy@~1.20 This will install NumPy with version >=1.20.0,<1.21.0. Alternatively, you could run poetry add numpy@^1.20 to install version >=1.20.0,<2.0.0. When you run this command Poetry does a few things: Update pyproject.toml to specify the new dependency.Use the dependency resolver to find the set of package versions that best fit the configuration. In this case, just numpy 1.20.3, but this will also include other packages if they’re specified.Install all the packages found by the resolver. By default, these will be installed into a virtual environment managed by Poetry. But you can also tell Poetry to install dependencies directly to the system Python.Freeze all dependencies and save them to poetry.lock so the exact build can be repeated in the future. If you add the --dev flag to the poetry add command, Poetry treats the dependency as a development dependency. For example: poetry add black --dev This distinction is useful because for production builds, you can install all non-development dependencies by adding the --no-dev flag: poetry install --no-dev This prevents unnecessary bloat in your production environment where you don’t need tools like formatters and linters. Finally, you can enter a shell with the virtual environment managed by Poetry activated: poetry shell python Recommended Setup As mentioned above, Poetry manages virtual environments for you and gives you a dedicated shell. But for production machine learning projects, I think it’s usually better to develop inside a Docker container. In this case, I recommend turning off Poetry’s virtual environment management. You can do that with a config command: poetry config virtualenvs.create false Or you can use an environment variable, for example: POETRY_VIRTUALENVS_CREATE=false poetry install And now, you should know everything you need to get started using Poetry to manage your Python environment machine learning projects! Of course, there’s a ton more to learn about Poetry. Fortunately, their documentation is pretty good. Happy ML engineering! The post Poetry for Package Management in Machine Learning Projects appeared first on Sparrow Computing.

In many situations, you’ll need to find the number of items stored in a data structure. Python’s built-in function len() is the tool that will help you with this task. There are some cases in which the use of len() is straightforward. However, there are other times when you’ll need to understand how this function works in more detail and how to apply it to different data types. In this tutorial, you’ll learn how to: Find the length of built-in data types using len() Use len() with third-party data types Provide support for len() with user-defined classes By the end of this article, you’ll know when to use the len() Python function and how to use it effectively. You’ll know which built-in data types are valid arguments for len() and which ones you can’t use. You’ll also understand how to use len() with third-party types, such as ndarray in NumPy and DataFrame in pandas, and with your own classes. Free Bonus: Click here to get a Python Cheat Sheet and learn the basics of Python 3, like working with data types, dictionaries, lists, and Python functions. Getting Started With Python’s len() The function len() is one of Python’s built-in functions. It returns the length of an object. For example, it can return the number of items in a list. You can use the function with many different data types. However, not all data types are valid arguments for len(). You can start by looking at the help for this function: >>>>>> help(len) Help on built-in function len in module builtins: len(obj, /) Return the number of items in a container. The function takes an object as an argument and returns the length of that object. The documentation for len() goes a bit further: Return the length (the number of items) of an object. The argument may be a sequence (such as a string, bytes, tuple, list, or range) or a collection (such as a dictionary, set, or frozen set). (Source) When you use built-in data types and many third-party types with len(), the function doesn’t need to iterate through the data structure. The length of a container object is stored as an attribute of the object. The value of this attribute is modified each time items are added to or removed from the data structure, and len() returns the value of the length attribute. This ensures that len() works efficiently. In the following sections, you’ll learn about how to use len() with sequences and collections. You’ll also learn about some data types that you cannot use as arguments for the len() Python function. Using len() With Built-in Sequences A sequence is a container with ordered items. Lists, tuples, and strings are three of the basic built-in sequences in Python. You can find the length of a sequence by calling len(): >>>>>> greeting = "Good Day!" >>> len(greeting) 9 >>> office_days = ["Tuesday", "Thursday", "Friday"] >>> len(office_days) 3 >>> london_coordinates = (51.50722, -0.1275) >>> len(london_coordinates) 2 When finding the length of the string greeting, the list office_days, and the tuple london_coordinates, you use len() in the same manner. All three data types are valid arguments for len(). The function len() always returns an integer as it’s counting the number of items in the object that you pass to it. The function returns 0 if the argument is an empty sequence: >>>>>> len("") 0 >>> len([]) 0 >>> len(()) 0 In the examples above, you find the length of an empty string, an empty list, and an empty tuple. The function returns 0 in each case. A range object is also a sequence that you can create using range(). A range object doesn’t store all the values but generates them when they’re needed. However, you can still find the length of a range object using len(): >>>>>> len(range(1, 20, 2)) 10 This range of numbers includes the integers from 1 to 19 with increments of 2. The length of a range object can be determined from the start, stop, and step values. In this section, you’ve used the len() Python function with strings, lists, tuples, and range objects. However, you can also use the function with any other built-in sequence. Read the full article at https://realpython.com/len-python-function/ » [ Improve Your Python With ? Python Tricks ? – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]