Feeds

Django is one of the most popular Python Web Frameworks. However, optimizing django performance can often be difficult. Most performance problems boil down to one thing: the database. According to our guest speaker, Andrew Brookins, the only way to master Django performance is to master database performance. In this Webinar, Andrew is going to take us on a journey through an app that he created to demonstrate all the bottlenecks that arise when trying to optimize your Django app. Monday, August 10 7:00PM CEST – 1:00PM Eastern Daylight Time Register here Outline Querying Tackling the N+1 Problem: Reducing the number of queries to make your pages faster. Annotations: Getting your database to do the heavy lifting for you Iterator(): Get Django to chuck responses from your database Indexing Covering Indexes: Turning queries that take several hundred milliseconds to tens of milliseconds. Materialized Views: One of the most powerful ways to have large queries created for you Caching with Redis Using Redis as a Cache Custom Redis Authentication About the Speaker Andrew Brookins has over a decade of experience with Django and Relational Databases. That experience spans DevOps, application development, and platform engineering. He has worked on ecommerce sites, mobile APIs, comic book readers, particle simulation systems, learning and volunteering apps, and now works on a Databases as a Service at Redis Labs. Andrew has also authored "Temple of Django Database Performance" which dives into the even greater detail than this webinar does. Furthermore, he has recently released a course on "Redis for Python Developers" at Redis University. Andrew lives in Portland, Oregon with his wife and two wonderful children.

We're back with another GeekOut episode. Richard Campbell, a developer and podcaster who also dives deep into science and tech topics, is back for our second GeekOut episode. Last time we geeked out about the real science and progress around a moon base. This time it's why is there life on Earth, where could it be or have been in the solar system, and beyond. <br/> <br/> In case you didn't catch the first GeekOut, episode 253, this one is more of a general science and tech episode. I love digging into the deep internals of all the tools of the Python space, but given all that is going on in the world, I thought it'd be fun to take a step back and just enjoy some fun geekery and give you all something to just sit back and let your mind dream.<br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Richard on Twitter</b>: <a href="https://twitter.com/richcampbell" target="_blank" rel="noopener">@richcampbell</a><br/> <b>All Richard's GeekOut Episodes</b>: <a href="http://geekout.show" target="_blank" rel="noopener">geekout.show</a><br/> <b>Moonbase Geekout Episode</b>: <a href="https://talkpython.fm/253" target="_blank" rel="noopener">talkpython.fm/253</a><br/> <b>High Altitude Venus Operational Concept (HAVOC)</b>: <a href="https://sacd.larc.nasa.gov/smab/havoc/" target="_blank" rel="noopener">sacd.larc.nasa.gov</a><br/> <b>New Horizons</b>: <a href="https://solarsystem.nasa.gov/missions/new-horizons/in-depth/" target="_blank" rel="noopener">solarsystem.nasa.gov</a><br/> <b>The Planets: Saturn - NOVA documentary (Cassini)</b>: <a href="https://www.pbs.org/wgbh/nova/video/the-planets-saturn/" target="_blank" rel="noopener">pbs.org</a><br/> <b>Mission to Jupiter - Galileo</b>: <a href="https://www.jpl.nasa.gov/missions/galileo/" target="_blank" rel="noopener">jpl.nasa.gov</a><br/></div><br/> <strong>Sponsors</strong><br/> <br/> <a href='https://talkpython.fm/brilliant'>Brilliant</a><br> <a href='https://talkpython.fm/training'>Talk Python Training</a>

<p>Sponsored by us! Support our work through:</p> <ul> <li>Our <a href="https://training.talkpython.fm/"><strong>courses at Talk Python Training</strong></a></li> <li><a href="https://t.co/AKfVKcveg6?amp=1"><strong>Brian’s pytest book</strong></a></li> </ul> <p><strong>Brian #1: Start using pip install --use-feature=2020-resolver if you aren’t already</strong></p> <ul> <li>Mathew Feickert</li> <li><p>You may see something interesting when you run pip <img src="https://pbs.twimg.com/media/EeERublXoAILPct?format=png&amp;name=small" alt="Image" /></p></li> <li><p>This is not a problem. Do not adjust your sets. </p></li> <li>But, you should be aware of it.</li> <li>Especially if you install from requirements generated with <code>pip freeze</code>, you’ll want to use <code>--use-feature=2020-resolver</code> everywhere:</li> </ul> <pre><code> $ python -m pip install --use-feature=2020-resolver -r requirements_original.txt $ python -m pip freeze &gt; requirements_lock.txt $ python -m pip install --use-feature=2020-resolver -r requirements_lock.txt </code></pre> <ul> <li>Otherwise, you may run into issues <ul> <li>see <a href="https://gist.github.com/matthewfeickert/78cd03376435c05b6104f376cd43b537">Example of --use-feature=2020-resolver breaking a pip freeze requirements.txt</a></li> <li>and <a href="https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020">Changes to the pip dependency resolver in 20.2 (2020)</a></li> </ul></li> </ul> <p><strong>Michael #2:</strong> <a href="https://twitter.com/mkennedy/status/1276606467156537344"><strong>Profiling Python import statements</strong></a></p> <ul> <li>Conversation with Brandon Braner lead to <a href="https://pypi.org/project/import-profiler/">import-profiler</a></li> <li>A basic python import profiler to find bottlenecks in import times. </li> <li>Not often a problem, imports can be an issue for applications that need to start quickly, such as CLI tools. </li> <li>Goal of import profiler is to help find the bottlenecks when importing a given package.</li> <li>Example</li> </ul> <pre><code> from import_profiler import profile_import with profile_import() as context: # Anything expensive in here import requests # Print cumulative and inline times. The number of + in the 3rd column # indicates the depth of the stack. context.print_info() </code></pre> <p>Output:</p> <pre><code> umtime (ms) intime (ms) name 83 0.5 requests 55 0.5 +packages.urllib3.contrib 54.1 0.3 ++ 53.1 0.7 +++connectionpool 6.3 1.1 ++++logging ... </code></pre> <p><strong>Brian #3:</strong> <a href="https://www.mattlayman.com/blog/2020/django-testing-toolbox/"><strong>Django Testing Toolbox</strong></a></p> <ul> <li>Matt Layman</li> <li>Testing packages commonly used on Django projects</li> <li>Techniques Matt uses</li> <li>packages <ul> <li>pytest-django - duh</li> <li>factory_boy - fake data</li> <li>django-test-plus - beefed up TestCase with tons of helper utilities</li> </ul></li> <li>techniques <ul> <li>Using TestCase and test classes instead of functions</li> <li>Arrange, Act Assert structure</li> <li>In-memory SQLite database</li> <li>Disable migrations while testing</li> <li>faster password hasher </li> <li>Use your editor effectively to run tests</li> </ul></li> </ul> <p><strong>Michael #4:</strong> <a href="https://github.com/pandas-profiling/pandas-profiling"><strong>Pandas-profiling</strong></a></p> <ul> <li>Recommended by Oz</li> <li>Considering the fact that Data Science users are almost the majority now for Python, I thought it would be nice to spread the word of pandas-profiling package. </li> <li>This package enables you to do Exploratory Data Analysis (EDA) with a single command. It is really useful to understand what percent of the data is NULL, or how it is distributed. </li> <li><p>I used to do these steps manually and it was quite time consuming. Now it is a breeze.</p></li> <li><p>Generates profile reports from a pandas <code>DataFrame</code>. The pandas <code>df.describe()</code> function is great but a little basic for serious exploratory data analysis. <code>pandas_profiling</code> extends the pandas DataFrame with <code>df.profile_report()</code> for quick data analysis.</p></li> <li>Features <ul> <li><strong>Type inference</strong>: detect the <a href="https://github.com/pandas-profiling/pandas-profiling#types">types</a> of columns in a dataframe.</li> <li><strong>Essentials</strong>: type, unique values, missing values</li> <li><strong>Quantile statistics</strong> like minimum value, Q1, median, Q3, maximum, range, interquartile range</li> <li><strong>Descriptive statistics</strong> like mean, mode, standard deviation, sum, median absolute deviation, coefficient of variation, kurtosis, skewness</li> <li><strong>Most frequent values</strong></li> <li><strong>Histogram</strong></li> <li><strong>Correlations</strong> highlighting of highly correlated variables, Spearman, Pearson and Kendall matrices</li> <li><strong>Missing values</strong> matrix, count, heatmap and dendrogram of missing values</li> <li><strong>Text analysis</strong> learn about categories (Uppercase, Space), scripts (Latin, Cyrillic) and blocks (ASCII) of text data.</li> <li><strong>File and Image analysis</strong> extract file sizes, creation dates and dimensions and scan for truncated images or those containing EXIF information.</li> </ul></li> <li>Nice examples too <ul> <li><a href="https://pandas-profiling.github.io/pandas-profiling/examples/master/census/census_report.html">Census Income</a> (US Adult Census data relating income)</li> <li><a href="https://pandas-profiling.github.io/pandas-profiling/examples/master/meteorites/meteorites_report.html">NASA Meteorites</a> (comprehensive set of meteorite landings) </li> <li><a href="https://pandas-profiling.github.io/pandas-profiling/examples/master/titanic/titanic_report.html">Titanic</a> (the "Wonderwall" of datasets) </li> <li><a href="https://pandas-profiling.github.io/pandas-profiling/examples/master/nza/nza_report.html">NZA</a> (open data from the Dutch Healthcare Authority)</li> <li><a href="https://pandas-profiling.github.io/pandas-profiling/examples/master/stata_auto/stata_auto_report.html">Stata Auto</a> (1978 Automobile data)</li> <li><a href="https://pandas-profiling.github.io/pandas-profiling/examples/master/vektis/vektis_report.html">Vektis</a> (Vektis Dutch Healthcare data)</li> <li><a href="https://pandas-profiling.github.io/pandas-profiling/examples/master/colors/colors_report.html">Colors</a> (a simple colors dataset)</li> </ul></li> </ul> <p><strong>Brian #5:</strong> <a href="https://rednafi.github.io/digressions/python/2020/07/03/python-mixins.html"><strong>Interfaces, Mixins and Building Powerful Custom Data Structures in Python</strong></a></p> <ul> <li>Redowan Delowar</li> <li>“Supercharging Python's built-in data structures”</li> <li>Discussion of interfaces, abstract base classes, mixins, and Python</li> <li>Class hierarchies and utilizing interfaces, ABCs (informal and formal), mixins, etc. are less common in Python than in some other languages, but you can still use them to do some really cool things.</li> <li>I especially liked the introduction to the concepts and why they are useful, as well as why ABCs are different than interfaces.</li> <li>“Interfaces can be thought of as a special case of Abstract Base Classes</li> </ul> <blockquote> <p>"It’s imperative that all the methods of an interface are abstract methods and the classes don’t store any state (instance variables). However, in case of abstract base classes, the methods are generally abstract but there can also be methods that provide implementation (concrete methods) and also, these classes can have instance variables."</p> </blockquote> <ul> <li>Shows an example, and goes on to discuss mixins, a parent class that provides functionality to subclasses but is not intended to be instantiated itself.</li> </ul> <p><strong>Michael #6:</strong> <a href="https://nedbatchelder.com/blog/202006/pickles_nine_flaws.html"><strong>Pickle’s 9 flaws</strong></a></p> <ul> <li>Python’s pickle module is a very convenient way to serialize and de-serialize objects. It needs no schema, and can handle arbitrary Python objects. But it has problems. This post briefly explains the problems.</li> <li>Via <a href="https://pycoders.com/">pycoders.com</a></li> <li>Article by Ned Batchelder <ul> <li><strong>Insecure</strong> - The insecurity is not because pickles contain code, but because they create objects by calling constructors named in the pickle. Any callable can be used in place of your class name to construct objects.</li> <li><strong>Old pickles look like old code</strong> - if your code changes between the time you made the pickle and the time you used it, your objects may not correspond to your code.</li> <li><strong>Implicit</strong> - they will serialize whatever structure your object has.</li> <li><strong>Over-serializes</strong> - They serialize everything in your objects, even data you didn’t want to serialize. For example, you might have an attribute that is a cache of computation that you don’t want serialized.</li> <li><strong>__init__ isn’t called</strong> - Pickles store the entire structure of your objects. When the pickle module recreates your objects, it does not call your __init__ method, since the object has already been created.</li> <li><strong>Python only</strong> - Pickles are specific to Python, and are only usable by other Python programs.</li> <li><strong>Unreadable</strong> - A pickle is a binary data stream (actually instructions for an abstract execution engine.) If you open a pickle as a plain file, you cannot read its contents.</li> <li><strong>Appears to pickle code</strong> - Functions and classes are first-class objects in Python: you can store them in lists, dicts, attributes, and so on. Pickle will gladly serialize objects that contain callables like functions and classes. But it doesn’t store the code in the pickle, just the name of the function or class.</li> <li><strong>Slow</strong> - Compared to other serialization techniques, pickle can be slow as Ben Frederickson demonstrates in <a href="https://www.benfrederickson.com/dont-pickle-your-data/">Don’t pickle your data</a>.</li> </ul></li> </ul> <p>Extras:</p> <p>Michael: </p> <ul> <li>More on <code>Pathlib</code> by <a href="https://twitter.com/flutefreak7/status/1287200446847823874">Brett Ables:</a> <code>text = Path(file).read_text()</code></li> <li>I was a guest on <a href="https://twitter.com/aQoCode/status/1287826329203810310">A Question of Code podcast</a>. Ed, Tom, and I discussed:</li> </ul> <p>Why is coding in Python such fun? And why is it so good for beginners and experts alike? Why might Python give you tangible results faster than JavaScript? And once you've learnt some Python, what are your career options? Find out all this and more in this week's pythonic installment of A Question of Code.</p> <p><strong>Joke</strong></p> <ul> <li>(<a href="https://twitter.com/ab_was_taken/status/1287884897802006528">via Aaron Brown</a>)</li> <li>By Caitlin Hudon ??‍? <a href="https://twitter.com/beeonaposy/status/1287797396471193601">@beeonaposy</a>:</li> <li>I have a Python joke, but I don't think this is the right environment.</li> </ul>

The post Python: How to Convert a List into a String? (The Right Way) appeared first on Afternerd.

What I did this week? I took a break for the week. So there's nothing on my list. What is coming up next? From my last work there were few minor things that need to be changed in my triage PR. I'll be working on adding a new format that is PDF. I need some time to figure out what is the best case to add this feature that is either using a different output format or providing a new template so the users can directly print from the HTML.   Have I got stuck anywhere? There are no blocking issues for me at this moment.

Hi everyone  a little late on this blog as I have to go back to college to get my stuff back. I passed the second evaluation, yay,. Till now I have been able to get the module metadata except the licenses. The licenses can be collected in 2 ways one is through making bash scripts to find and collect licenses from the module directory. Since licenses files dont have a set structure or a convention that's why its getting difficult and a foolproof  guarantee cant be given. Another way is through making an in house copyright to License parser. For debian licenses we have debut  used in tern I have to come up with similar kind of parser. I also have one idea of training a model from data collected by using github's api but thats just overkill.

The post Python: Sort vs Sorted appeared first on Afternerd.