This year marks our first-ever TC Sessions: Space event, and what better way to kick things off than with the head of NASA: Administrator Jim Bridenstine will join us onstage on June 25 in LA. NASA has been more open than ever before to working with startups and entrepreneurs, and we’ll hear directly from the […]

Apple says the coronavirus outbreak will hurt its manufacturing and sales, Jeff Bezos makes a big commitment to fighting climate change and SpaceX launches more Starlink satellites. Here’s your Daily Crunch for February 18, 2020. 1. Apple will miss revenue forecast as coronavirus impacts its manufacturing and sales In a letter to investors, Apple said […]

Jaguar Land Rover has introduced a new concept vehicle that cuts a very different figure relative to its usual fare: It’s a four-wheeled electric urban mobility concept called ‘Project Vector’ that looks more like a low-floored airport shuttle train car than a traditional car. This is a look that’s increasingly become popular among automakers designing […]

Video editing startup InVideo has new funding and a new product. The San Francisco-headquartered company bills itself as the easiest way for anyone to create professional-quality videos, using a drag-and-drop interface, along with a library of templates and stock photos and videos. The resulting videos can then be optimized for Facebook, Instagram, YouTube and other […]

In early December, Spotify launched its annual personalized Wrapped playlist with its users’ most-streamed sounds of 2019. That has become a bit of a tradition and isn’t necessarily anything new, but for 2019, it also gave users a look back at how they used Spotify over the last decade. Because this was quite a large […]

Spotify’s ongoing investments in the podcast-streaming side of its business helped boost podcast listening on its service by 200% last year. But today, only 16% of Spotify’s monthly listeners are engaging with podcats — a number the company today hopes to nudge higher by redesigning the podcast side of its streaming app. The new layout […]

Robotic process automation — the ability to automate certain repetitive software-based tasks to free up people to focus on work that computers cannot do — has become a major growth area in the world of IT. Today, a startup called Aisera is coming out of stealth that has taken this idea and supercharged it, by […]

There is plenty of exciting stuff in Microsoft Dynamics that you can use for data analysis or data visualization projects in large organizations. So how do you get your hands on it using a Python script? Good news, it is not that complicated!

Tom Christie is probably best known as the creator of Django REST Framework, but his contributions to the state the web in Python extend well beyond that. In this episode he shares his story of getting involved in web development, his work on various projects to power the asynchronous web in Python, and his efforts to make his open source contributions sustainable. This was an excellent conversation about the state of asynchronous frameworks for Python and the challenges of making a career out of open source.Summary Tom Christie is probably best known as the creator of Django REST Framework, but his contributions to the state the web in Python extend well beyond that. In this episode he shares his story of getting involved in web development, his work on various projects to power the asynchronous web in Python, and his efforts to make his open source contributions sustainable. This was an excellent conversation about the state of asynchronous frameworks for Python and the challenges of making a career out of open source. Announcements Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great. When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With 200 Gbit/s private networking, node balancers, a 40 Gbit/s public network, and a brand new managed Kubernetes platform, all controlled by a convenient API you’ve got everything you need to scale up. And for your tasks that need fast computation, such as training machine learning models, they’ve got dedicated CPU and GPU instances. Go to pythonpodcast.com/linode to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show! You listen to this show to learn and stay up to date with the ways that Python is being used, including the latest in machine learning and data analysis. For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Corinium Global Intelligence, ODSC, and Data Council. Upcoming events include the Software Architecture Conference in NYC, Strata Data in San Jose, and PyCon US in Pittsburgh. Go to pythonpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today. Your host as usual is Tobias Macey and today I’m interviewing Tom Christie about the Encode organization and the work he is doing to drive the state of the art in async for Python Interview Introductions How did you get introduced to Python? Can you start by describing what the Encode organization is and how it came to be? What are some of the other approaches to funding and sustainability that you have tried in the past? What are the benefits to the developers provided by an organization which you were unable to achieve through those other means? What benefits are realized by your sponsors as compared to other funding arrangements? What projects are part of the Encode organization? How do you determine fund allocation for projects and participants in the organization? What is the process for becoming a member of the Encode organization and what benefits and responsibilities does that entail? A large number of the projects that are part of the organization are focused on various aspects of asynchronous programming in Python. Is that intentional, or just an accident of your own focus and network? For those who are familiar with Python web programming in the context of WSGI, what are some of the practices that they need to unlearn in an async world, and what are some new capabilities that they should be aware of? Beyond Encode and your recent work on projects such as Starlette you are also well known as the creator of Django Rest Framework. How has your experience building and growing that project influenced your current focus on a technical, community, and professional level? Now that Python 2 is officially unsupported and asynchronous capabilities are part of the core language, what future directions do you foresee for the community and ecosystem? What are some areas of potential focus that you think are worth more attention and energy? What do you have planned for the future of Encode, your own projects, and your overall engagement with the Python ecosystem? Keep In Touch Website tomchristie on Github @_tomchristie on Twitter Picks Tobias Maleficent: Mistress of Evil Abominable Tom The Lobster The Master And His Emissary by Ian McGilchrist Closing Announcements Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@podcastinit.com) with your story. To help other people find the show please leave a review on iTunes and tell your friends and co-workers Join the community in the new Zulip chat workspace at pythonpodcast.com/chat Links Encode Django Rest Framework Starlette Zope Django Django Piston Django Tastypie Andrew Godwin ASGI Django Channels Podcast Episode Flask Pyramid Sentry Podcast Episode Tidelift Uvicorn HTTPX Tidelift Open Collective Stripe Github Sponsors Python Software Foundation Podcast Episode Firebase Databases ORM HTTP3 The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

We’re pleased to announce our official conference logo for EuroPython 2020, July 20-26, in Dublin, Ireland:The logo is inspired by the colors and symbols often associated with Ireland: the shamrock and the celtic harp. It was again created by our designer Jessica Peña Moro from Simétriko, who had already helped us in previous years with the conference design.Some more updates:We’re working on launching the website, the CFP and ticket sales in March.We are also preparing the sponsorship packages and should have them ready early in March as well. Early bird sponsors will again receive a 10% discount on the package price. If you’re interested in becoming a launch sponsor, please contact our sponsor team at sponsoring@europython.eu.Enjoy,– EuroPython 2020 Teamhttps://ep2020.europython.eu/

Find your perfect Python development setup with this review of Python IDEs and code editors. Writing Python using IDLE or the Python REPL is great for simple things, but not ideal for larger programming projects. With this course you’ll get an overview of the most common Python coding environments to help you make an informed decision. By the end of this course, you’ll know how to: Choose the Python editing environment that’s right for you Perform common tasks like creating, running, and debugging code Dig deeper into optimizing your favorite editing setup [ Improve Your Python With ? Python Tricks ? – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Introduction In this post, we will dive into MongoDB as a data store from a Python perspective. To that end, we'll write a simple script to showcase what we can achieve and any benefits we can reap from it. Web applications, like many other software applications, are powered by data. The organization and storage of this data are important as they dictate how we interact with the various applications at our disposal. The kind of data handled can also have an influence on how we undertake this process. Databases allow us to organize and store this data, while also controlling how we store, access, and secure the information. NoSQL Databases There are two main types of databases - relational and non-relational databases. Relational databases allow us to store, access, and manipulate data in relation to another piece of data in the database. Data is stored in organized tables with rows and columns with relationships linking the information among tables. To work with these databases, we use the Structured Query Language (SQL) and examples include MySQL and PostgreSQL. Non-relational databases store data in neither relation or tabular, as in relational databases. They are also referred to as NoSQL databases since we do not use SQL to interact with them. Furthermore, NoSQL databases can be divided into Key-Value stores, Graph stores, Column stores, and Document Stores, which MongoDB falls under. MongoDB and When to Use it MongoDB is a document store and non-relational database. It allows us to store data in collections that are made up of documents. In MongoDB, a document is simply a JSON-like binary serialization format referred to as a BSON, or Binary-JSON, and has a maximum size of 16 megabytes. This size limit is in place to ensure efficient memory and bandwidth usage during transmission. MongoDB also provides the GridFS specification in case there is a need to store files larger than the set limit. Documents are made up of field-value pairs, just like in regular JSON data. However, this BSON format can also contain more data types, such as Date types and Binary Data types. BSON was designed to be lightweight, easily traversable, and efficient when encoding and decoding data to and from BSON. Being a NoSQL datastore, MongoDB allows us to enjoy the advantages that come with using a non-relational database over a relational one. One advantage is that it offers high scalability by efficiently scaling horizontally through sharding or partitioning of the data and placing it on multiple machines. MongoDB also allows us to store large volumes of structured, semi-structured, and unstructured data without having to maintain relationships between it. Being open-source, the cost of implementing MongoDB is kept low to just maintenance and expertise. Like any other solution, there are downsides to using MongoDB. The first one is that it does not maintain relationships between stored data. Due to this, it is hard to perform ACID transactions that ensure consistency. Complexity is increased when trying to support ACID transactions. MongoDB, like other NoSQL data stores, is not as mature as relational databases and this can make it hard to find experts. The non-relational nature of MongoDB makes it ideal for the storage of data in specific situations over its relational counterparts. For instance, a scenario where MongoDB is more suitable than a relational database is when the data format is flexible and has no relations. With flexible/non-relational data, we don't need to maintain ACID properties when storing data as opposed to relational databases. MongoDB also allows us to easily scale data into new nodes. However, with all its advantages, MongoDB is not ideal when our data is relational in nature. For instance, if we are storing customer records and their orders. In this situation, we will need a relational database to maintain the relationships between our data, which are important. It is also not suitable to use MongoDB if we need to comply with ACID properties. Interacting with MongoDB via Mongo Shell To work with MongoDB, we will need to install the MongoDB Server, which we can download from the official homepage. For this demonstration, we will use the free Community Server. The MongoDB server comes with a Mongo Shell that we can use to interact with the server via the terminal. To activate the shell, just type mongo in your terminal. You'll be greeted with information about the MongoDB server set-up, including the MongoDB and Mongo Shell version, alongside the server URL. For instance, our server is running on: mongodb://127.0.0.1:27017 In MongoDB, a database is used to hold collections that contains documents. Through the Mongo shell, we can create a new database or switch to an existing one using the use command: > use SeriesDB Every operation we execute after this will be effected in our SeriesDB database. In the database, we will store collections, which are similar to tables in relational databases. For example, for the purposes of this tutorial, let's add a few series to the database: > db.series.insertMany([ ... { name: "Game of Thrones", year: 2012}, ... { name: "House of Cards", year: 2013 }, ... { name: "Suits", year: 2011} ... ]) We're greeted with: { "acknowledged" : true, "insertedIds" : [ ObjectId("5e300724c013a3b1a742c3b9"), ObjectId("5e300724c013a3b1a742c3ba"), ObjectId("5e300724c013a3b1a742c3bb") ] } To fetch all the documents stored in our series collection, we use db.inventory.find({}), whose SQL equivalent is SELECT * FROM series. Passing an empty query (i.e. {}) will return all the documents: > db.series.find({}) { "_id" : ObjectId("5e3006258c33209a674d1d1e"), "name" : "The Blacklist", "year" : 2013 } { "_id" : ObjectId("5e300724c013a3b1a742c3b9"), "name" : "Game of Thrones", "year" : 2012 } { "_id" : ObjectId("5e300724c013a3b1a742c3ba"), "name" : "House of Cards", "year" : 2013 } { "_id" : ObjectId("5e300724c013a3b1a742c3bb"), "name" : "Suits", "year" : 2011 } We can also query data using the equality condition, for instance, to return all the TV series that premiered in 2013: > db.series.find({ year: 2013 }) { "_id" : ObjectId("5e3006258c33209a674d1d1e"), "name" : "The Blacklist", "year" : 2013 } { "_id" : ObjectId("5e300724c013a3b1a742c3ba"), "name" : "House of Cards", "year" : 2013 } The SQL equivalent would be SELECT * FROM series WHERE year=2013. MongoDB also allows us to update individual documents using db.collection.UpdateOne(), or perform batch updates using db.collection.UpdateMany(). For example, to update the release year for Suits: > db.series.updateOne( { name: "Suits" }, { $set: { year: 2010 } } ) { "acknowledged" : true, "matchedCount" : 1, "modifiedCount" : 1 } Finally, to delete documents, the Mongo Shell offers the db.collection.deleteOne() and db.collection.deleteMany() functions. For instance, to delete all the series that premiered in 2012, we'd run: > db.series.deleteMany({ year: 2012 }) { "acknowledged" : true, "deletedCount" : 2 } More information on the CRUD operations on MongoDB can be found in the online reference including more examples, performing operations with conditions, atomicity, and mapping of SQL concepts to MongoDB concepts and terminology. Integrating Python with MongoDB MongoDB provides drivers and tools for interacting with a MongoDB datastore using various programming languages including Python, JavaScript, Java, Go, and C#, among others. PyMongo is the official MongoDB driver for Python, and we will use it to create a simple script that we will use to manipulate data stored in our SeriesDB database. With Python 3.6+ and Virtualenv installed in our machines, let us create a virtual environment for our application and install PyMongo via pip: $ virtualenv --python=python3 env --no-site-packages $ source env/bin/activate $ pip install pymongo Using PyMongo, we are going to write a simple script that we can execute to perform different operations on our MongoDB database. Connecting to MongoDB First, we import pymongo in our mongo_db_script.py and create a client connected to our locally running instance of MongoDB: import pymongo # Create the client client = MongoClient('localhost', 27017) # Connect to our database db = client['SeriesDB'] # Fetch our series collection series_collection = db['series'] So far, we have created a client that connects to our MongoDB server and used it to fetch our 'SeriesDB' database. We then fetch our 'series' collection and store it in an object. Creating Documents To make our script more convenient, we will write functions that wrap around PyMongo to enable us to easily manipulate data. We will use Python dictionaries to represent documents and we will pass these dictionaries to our functions. First, let us create a function to insert data into our 'series' collection: # Imports truncated for brevity def insert_document(collection, data): """ Function to insert a document into a collection and return the document's id. """ return collection.insert_one(data).inserted_id This function receives a collection and a dictionary of data and inserts the data into the provided collection. The function then returns an identifier that we can use to accurately query the individual object from the database. We should also note that MongoDB adds an additional _id key to our documents, when they are not provided, when creating the data. Now let's try adding a show using our function: new_show = { "name": "FRIENDS", "year": 1994 } print(insert_document(series_collection, new_show)) The output is: 5e4465cfdcbbdc68a6df233f When we run our script, the _id of our new show is printed on the terminal and we can use this identifier to fetch the show later on. We can provide an _id value instead of having it assigned automatically, which we'd provide in the dictionary: new_show = { "_id": "1", "name": "FRIENDS", "year": 1994 } And if we were to try and store a document with an existing _id, we'd be greeted with an error similar to the following: DuplicateKeyError: E11000 duplicate key error index: SeriesDB.series.$id dup key: { : 1} Retrieving Documents To retrieve documents from the database we'll use find_document(), which queries our collection for single or multiple documents. Our function will receive a dictionary that contains the elements we want to filter by, and an optional argument to specify whether we want one document or multiple documents: # Imports and previous code truncated for brevity def find_document(collection, elements, multiple=False): """ Function to retrieve single or multiple documents from a provided Collection using a dictionary containing a document's elements. """ if multiple: results = collection.find(elements) return [r for r in results] else: return collection.find_one(elements) And now, let's use this function to find some documents: result = find_document(series_collection, {'name': 'FRIENDS'}) print(result) When executing our function, we did not provide the multiple parameter and the result is a single document: {'_id': ObjectId('5e3031440597a8b07d2f4111'), 'name': 'FRIENDS', 'year': 1994} When the multiple parameter is provided, the result is a list of all the documents in our collection that have a name attribute set to FRIENDS. Updating Documents Our next function, update_document(), will be used to update a single specific document. We will use the _id of the document and the collection it belongs to when locating it: # Imports and previous code truncated for brevity def update_document(collection, query_elements, new_values): """ Function to update a single document in a collection. """ collection.update_one(query_elements, {'$set': new_values}) Now, let's insert a document: new_show = { "name": "FRIENDS", "year": 1995 } id_ = insert_document(series_collection, new_show) With that done, let's update the document, which we'll specify using the _id returned from adding it: update_document(series_collection, {'_id': id_}, {'name': 'F.R.I.E.N.D.S'}) And finally, let's fetch it to verify that the new value has been put in place and print the result: result = find_document(series_collection, {'_id': id_}) print(result) When we execute our script, we can see that our document has been updated: {'_id': ObjectId('5e30378e96729abc101e3997'), 'name': 'F.R.I.E.N.D.S', 'year': 1995} Deleting Documents And finally, let's write a function for deleting documents: # Imports and previous code truncated for brevity def delete_document(collection, query): """ Function to delete a single document from a collection. """ collection.delete_one(query) Since we're using the delete_one method, only one document can be deleted per call, even if the query matches multiple documents. Now, let's use the function to delete an entry: delete_document(series_collection, {'_id': id_}) If we try retrieving that same document: result = find_document(series_collection, {'_id': id_}) print(result) We're greeted with the expected result: None Next Steps We have highlighted and used a few of PyMongo's methods to interact with our MongoDB server from a Python script. However, we have not utilized all the methods available to us through the module. All the available methods can be found in the official PyMongo documentation and are classified according to the submodules. We've written a simple script that performs rudimentary CRUD functionality on a MongoDB database. While we could import the functions in a more complex codebase, or into a Flask/Django application for example, these frameworks have libraries to achieve the same results already. These libraries make it easier, more conventient, and help us connect more securely to MongoDB. For example, with Django we can use libraries such as Django MongoDB Engine and Djongo, while Flask has Flask-PyMongo that helps bridge the gap between Flask and PyMongo to facilitate seamless connectivity to a MongoDB database. Conclusion MongoDB is a document store and falls under the category of non-relational databases (NoSQL). It has certain advantages compared to relational databases, as well as some disadvantages. While it is not suitable for all situations, we can still use MongoDB to store data and manipulate the data from our Python applications using PyMongo among other libraries - allowing us to harness the power of MongoDB in situations where it is best suited. It is therefore up to us to carefully examine our requirements before making the decision to use MongoDB to store data. The script we have written in this post can be found on GitHub.

Introduction Record linking and fuzzy matching are terms used to describe the process of joining two data sets together that do not have a common unique identifier. Examples include trying to join files based on people’s names or merging data that only have organization’s name and address. This problem is a common business challenge and difficult to solve in a systematic way - especially when the data sets are large. A naive approach using Excel and vlookup statements can work but requires a lot of human intervention. Fortunately, python provides two libraries that are useful for these types of problems and can support complex matching algorithms with a relatively simple API. The first one is called fuzzymatcher and provides a simple interface to link two pandas DataFrames together using probabilistic record linkage. The second option is the appropriately named Python Record Linkage Toolkit which provides a robust set of tools to automate record linkage and perform data deduplication. This article will discuss how to use these two tools to match two different data sets based on name and address information. In addition, the techniques used to do matching can be applied to data deduplication and will be briefly discussed. The problem Anyone that has tried to merge disparate data sets together has likely run across some variation of this challenge. In the simple example below, we have a customer record in our system and need to determine the data matches - without the use of a common identifier. With a small sample set and our intuition, it looks like account 18763 is the same as account number A1278. We know that Brothers and Bro as well as Lane and LN are equivalent so this process is relatively easy for a person. However, trying to program logic to handle this is a challenge. In my experience, most people start using excel to vlookup the various components of the address and try to find the best match based on the state, street number or zip code. In some cases, this can work. However there are more sophisticated ways to perform string comparisons that we might want to use. For example, I wrote briefly about a package called fuzzy wuzzy several years ago. The challenge is that these algorithms (e.g. Levenshtein, Damerau-Levenshtein, Jaro-Winkler, q-gram, cosine) are computationally intensive. Trying to do a lot of matching on large data sets is not scaleable. If you are interested in more mathematical details on these concepts, wikipedia is a good place to start and this article contains much more additional detail. Finally, this blog post discusses some of the string matching approaches in more detail. Fortunately there are python tools that can help us implement these methods and solve some of these challenging problems. The data For this article, we will be using US hospital data. I chose this data set because hospital data has some unique qualities that make it challenging to match: Many hospitals have similar names across different cities (Saint Lukes, Saint Mary, Community Hospital) In urban areas, hospitals can occupy several city blocks so addresses can be ambiguous Hospitals tend to have many clinics and other associated and related facilities nearby Hospitals also get acquired and name changes are common - making this process even more difficult Finally, there are a thousands of medical facilities in the US so the problem is challenging to scale In these examples, I have two data sets. The first is an internal data set that contains basic hospital account number, name and ownership information. The second data set contains hospital information (called provider) as well as the number of discharges and Medicare payment for a specific Heart Failure procedure. The full data sets are available from Medicare.gov and CMS.gov and the simplified and cleaned version are available on github. The business scenario is that we want to match up the hospital reimbursement information with our internal account data so we have more information to analyze our hospital customers. In this instance we have 5339 hospital accounts and 2697 hospitals with reimbursement information. Unfortunately we do not have a common ID to join on so we will see if we can use these python tools to merge the data together based on a combination of name and address information. Approach 1 - fuzzymatcher For the first approach, we will try using fuzzymatcher. This package leverages sqlite’s full text search capability to try to match records in two different DataFrames. To install fuzzy matcher, I found it easier to conda install the dependencies (pandas, metaphone, fuzzywuzzy) then use pip to install fuzzymatcher. Given the computational burden of these algorithms you will want to use the compiled c components as much as possible and conda made that easiest for me. If you wish to follow along, this notebook contains a summary of all the code. After everything is setup, let’s import and get the data into our DataFrames: import pandas as pd from pathlib import Path import fuzzymatcher hospital_accounts = pd.read_csv('hospital_account_info.csv') hospital_reimbursement = pd.read_csv('hospital_reimbursement.csv') Here is the hospital account information: Here is the reimbursement information: Since the columns have different names, we need to define which columns to match for the left and right DataFrames. In this case, our hospital account information will be the left DataFrame and the reimbursement info will be the right. left_on = ["Facility Name", "Address", "City", "State"] right_on = [ "Provider Name", "Provider Street Address", "Provider City", "Provider State" ] Now we let fuzzymatcher try to figure out the matches using fuzzy_left_join : matched_results = fuzzymatcher.fuzzy_left_join(hospital_accounts, hospital_reimbursement, left_on, right_on, left_id_col='Account_Num', right_id_col='Provider_Num') Behind the scenes, fuzzymatcher determines the best match for each combination. For this data set we are analyzing over 14 million combinations. On my laptop, this takes about 2 min and 11 seconds to run. The matched_results DataFrame contains all the data linked together as well as as best_match_score which shows the quality of the link. Here’s a subset of the columns rearranged in a more readable format for the top 5 best matches: cols = [ "best_match_score", "Facility Name", "Provider Name", "Address", "Provider Street Address", "Provider City", "City", "Provider State", "State" ] matched_results[cols].sort_values(by=['best_match_score'], ascending=False).head(5) The first item has a match score of 3.09 and certainly looks like a clean match. You can see that the Facility Name and Provider Name for the Mayo Clinic in Red Wing has a slight difference but we were still able to get a good match. We can check on the opposite end of the spectrum to see where the matches don’t look as good: matched_results[cols].sort_values(by=['best_match_score'], ascending=True).head(5) Which shows some poor scores as well as obvious mismatches: This example highlights that part of the issue is that one set of data includes data from Puerto Rico and the other does not. This discrepancy highlights the need to make sure you really understand your data and what cleaning and filtering you may need to do before trying to match. We’ve looked at the extreme cases, let’s take a look at some of the matches that might be a little more challenging by looking at scores < 80: matched_results[cols].query("best_match_score <= .80").sort_values( by=['best_match_score'], ascending=False).head(5) This example shows how some of the matches get a little more ambiguous. For example, is ADVENTIST HEALTH UKIAH VALLEY the same as UKIAH VALLEY MEDICAL CENTER? Depending on your data set and your needs, you will need to find the right balance of automated and manual match review. Overall, fuzzymatcher is a useful tool to have for medium sized data sets. As you start to get to 10,000’s of rows, it will take a lot of time to compute, so plan accordingly. However the ease of use - especially when working with pandas makes it a great first place to start. Approach 2 - Python Record Linkage Toolkit The Python Record Linkage Toolkit provides another robust set of tools for linking data records and identifying duplicate records in your data. The Python Record Linkage Toolkit has several additional capabilities: Ability to define the types of matches for each column based on the column data types Use “blocks” to limit the pool of potential matches Provides ranking of the matches using a scoring algorithm Multiple algorithms for measuring string similarity Supervised and unsupervised learning approaches Multiple data cleaning methods The trade-off is that it is a little more complicated to wrangle the results in order to do further validation. However, the steps are relatively standard pandas commands so do not let that intimidate you. For this example, make sure you install the library using pip . We will use the same data set but we will read in the data with an explicit index column. This makes subsequent data joins a little easier to interpret. import pandas as pd import recordlinkage hospital_accounts = pd.read_csv('hospital_account_info.csv', index_col='Account_Num') hospital_reimbursement = pd.read_csv('hospital_reimbursement.csv', index_col='Provider_Num') Because the Record Linkage Toolkit has more configuration options, we need to perform a couple of steps to define the linkage rules. The first step is to create a indexer object: indexer = recordlinkage.Index() indexer.full() WARNING:recordlinkage:indexing - performance warning - A full index can result in large number of record pairs. This WARNING points us to a difference between the record linkage library and fuzzymatcher. With record linkage, we have some flexibility to influence how many pairs are evaluated. By using full indexer all potential pairs are evaluated (which we know is over 14M pairs). I will come back to some of the other options in a moment. Let’s continue with the full index and see how it performs. The next step is to build up all the potential candidates to check: candidates = indexer.index(hospital_accounts, hospital_reimbursement) print(len(candidates)) 14399283 This quick check just confirmed the total number of comparisons. Now that we have defined the left and right data sets and all the candidates, we can define how we want to perform the comparison logic using Compare() compare = recordlinkage.Compare() compare.exact('City', 'Provider City', label='City') compare.string('Facility Name', 'Provider Name', threshold=0.85, label='Hosp_Name') compare.string('Address', 'Provider Street Address', method='jarowinkler', threshold=0.85, label='Hosp_Address') features = compare.compute(candidates, hospital_accounts, hospital_reimbursement) We can define several options for how we want to compare the columns of data. In this specific example, we look for an exact match on the city. I have also shown some examples of string comparison along with the threshold and algorithm to use for comparison. In addition to these options, you can define your own or use numeric, dates and geographic coordinates. Refer to the documentation for more examples. The final step is to perform all the feature comparisons using compute . In this example, using the full index, this takes 3 min and 41 s. Let’s go back and look at alternatives to speed this up. One key concept is that we can use blocking to limit the number of comparisons. For instance, we know that it is very likely that we only want to compare hospitals that are in the same state. We can use this knowledge to setup a block on the state columns: indexer = recordlinkage.Index() indexer.block(left_on='State', right_on='Provider State') candidates = indexer.index(hospital_accounts, hospital_reimbursement) print(len(candidates)) 475830 With the block on state, the candidates will be filtered to only include those where the state values are the same. We have filtered down the candidates to only 475,830. If we run the same comparison code, it only takes 7 seconds. A nice speedup! In this data set, the state data is clean but if it were a little messier, we could use another the blocking algorithm like SortedNeighborhood to add some flexibility for minor spelling mistakes. For instance, what if the state names contained “Tenessee” and “Tennessee”? Using blocking would fail but sorted neighborhood would handle this situation more gracefully. indexer = recordlinkage.Index() indexer.sortedneighbourhood(left_on='State', right_on='Provider State') candidates = indexer.index(hospital_accounts, hospital_reimbursement) print(len(candidates)) 998860 In this case, sorted neighbors takes 15.9 seconds on 998,860 candidates which seems like a reasonable trade-off. Regardless of which option you use, the result is a features DataFrame that looks like this: This DataFrame shows the results of all of the comparisons. There is one row for each row in the account and reimbursement DataFrames. The columns correspond to the comparisons we defined. A 1 is a match and 0 is not. Given the large number of records with no matches, it is a little hard to see how many matches we might have. We can sum up the individual scores to see about the quality of the matches. features.sum(axis=1).value_counts().sort_index(ascending=False) 3.0 2285 2.0 451 1.0 7937 0.0 988187 dtype: int6 Now we know that there are 988,187 rows with no matching values whatsoever. 7937 rows have at least one match, 451 have 2 and 2285 have 3 matches. To make the rest of the analysis easier, let’s get all the records with 2 or 3 matches and add a total score: potential_matches = features[features.sum(axis=1) > 1].reset_index() potential_matches['Score'] = potential_matches.loc[:, 'City':'Hosp_Address'].sum(axis=1) Here is how to interpret the table. For the first row, Account_Num 26270 and Provider_Num 868740 match on city, hospital name and hospital address. Let’s look at these two and see how close they are: hospital_accounts.loc[26270,:] Facility Name SCOTTSDALE OSBORN MEDICAL CENTER Address 7400 EAST OSBORN ROAD City SCOTTSDALE State AZ ZIP Code 85251 County Name MARICOPA Phone Number (480) 882-4004 Hospital Type Acute Care Hospitals Hospital Ownership Proprietary Name: 26270, dtype: object hospital_reimbursement.loc[868740,:] Provider Name SCOTTSDALE OSBORN MEDICAL CENTER Provider Street Address 7400 EAST OSBORN ROAD Provider City SCOTTSDALE Provider State AZ Provider Zip Code 85251 Total Discharges 62 Average Covered Charges 39572.2 Average Total Payments 6551.47 Average Medicare Payments 5451.89 Name: 868740, dtype: object Yep. Those look like good matches. Now that we know the matches, we need to wrangle the data to make it easier to review all the data together. I am going to make a concatenated name and address lookup for each of these source DataFrames. hospital_accounts['Acct_Name_Lookup'] = hospital_accounts[[ 'Facility Name', 'Address', 'City', 'State' ]].apply(lambda x: '_'.join(x), axis=1) hospital_reimbursement['Reimbursement_Name_Lookup'] = hospital_reimbursement[[ 'Provider Name', 'Provider Street Address', 'Provider City', 'Provider State' ]].apply(lambda x: '_'.join(x), axis=1) account_lookup = hospital_accounts[['Acct_Name_Lookup']].reset_index() reimbursement_lookup = hospital_reimbursement[['Reimbursement_Name_Lookup']].reset_index() Now merge in with the account data: account_merge = potential_matches.merge(account_lookup, how='left') Finally, merge in the reimbursement data: final_merge = account_merge.merge(reimbursement_lookup, how='left') Let’s see what the final data looks like: cols = ['Account_Num', 'Provider_Num', 'Score', 'Acct_Name_Lookup', 'Reimbursement_Name_Lookup'] final_merge[cols].sort_values(by=['Account_Num', 'Score'], ascending=False) One of the differences between the toolkit approach and fuzzymatcher is that we are including multiple matches. For instance, account number 32725 could match two providers: final_merge[final_merge['Account_Num']==32725][cols] In this case, someone will need to investigate and figure out which match is the best. Fortunately it is easy to save all the data to Excel and do more analysis: final_merge.sort_values(by=['Account_Num', 'Score'], ascending=False).to_excel('merge_list.xlsx', index=False) As you can see from this example, the Record Linkage Toolkit allows a lot more flexibility and customization than fuzzymatcher. The downside is that there is a little more manipulation to get the data stitched back together in order to hand the data over to a person to complete the comparison. Deduplicating data with Record Linkage Toolkit One of the additional uses of the Record Linkage Toolkit is for finding duplicate records in a data set. The process is very similar to matching except you pass match a single DataFrame against itself. Let’s walk through an example using a similar data set: hospital_dupes = pd.read_csv('hospital_account_dupes.csv', index_col='Account_Num') Then create our indexer with a sorted neighbor block on State . dupe_indexer = recordlinkage.Index() dupe_indexer.sortedneighbourhood(left_on='State') dupe_candidate_links = dupe_indexer.index(hospital_dupes) We should check for duplicates based on city, name and address: compare_dupes = recordlinkage.Compare() compare_dupes.string('City', 'City', threshold=0.85, label='City') compare_dupes.string('Phone Number', 'Phone Number', threshold=0.85, label='Phone_Num') compare_dupes.string('Facility Name', 'Facility Name', threshold=0.80, label='Hosp_Name') compare_dupes.string('Address', 'Address', threshold=0.85, label='Hosp_Address') dupe_features = compare_dupes.compute(dupe_candidate_links, hospital_dupes) Because we are only comparing with a single DataFrame, the resulting DataFrame has an Account_Num_1 and Account_Num_2 : Here is how we score: dupe_features.sum(axis=1).value_counts().sort_index(ascending=False) 3.0 7 2.0 206 1.0 7859 0.0 973205 dtype: int64 Add the score column: potential_dupes = dupe_features[dupe_features.sum(axis=1) > 1].reset_index() potential_dupes['Score'] = potential_dupes.loc[:, 'City':'Hosp_Address'].sum(axis=1) Here’s a sample: These 9 records have a high likelihood of being duplicated. Let’s look at an example to see if they might be dupes: hospital_dupes.loc[51567, :] Facility Name SAINT VINCENT HOSPITAL Address 835 SOUTH VAN BUREN ST City GREEN BAY State WI ZIP Code 54301 County Name BROWN Phone Number (920) 433-0112 Hospital Type Acute Care Hospitals Hospital Ownership Voluntary non-profit - Church Name: 51567, dtype: object hospital_dupes.loc[41166, :] Facility Name ST VINCENT HOSPITAL Address 835 S VAN BUREN ST City GREEN BAY State WI ZIP Code 54301 County Name BROWN Phone Number (920) 433-0111 Hospital Type Acute Care Hospitals Hospital Ownership Voluntary non-profit - Church Name: 41166, dtype: object Yes. That looks like a potential duplicate. The name and address are similar and the phone number is off by one digit. How many hospitals do they really need to treat all those Packer fans? :) As you can see, this method can be a powerful and relatively easy tool to inspect your data and check for duplicate records. Advanced Usage In addition to the matching approaches shown here, the Record Linkage Toolkit contains several machine learning approaches to matching records. I encourage interested readers to review the documentation for examples. One of the pretty handy capabilities is that there is a browser based tool that you can use to generate record pairs for the machine learning algorithms. Both tools include some capability for pre-processing the data to make the matching more reliable. Here is the preprocessing content in the Record Linkage Toolkit. This example data was pretty clean so you will likely need to explore some of these capabilities for your own data. Summary Linking different record sets on text fields like names and addresses is a common but challenging data problem. The python ecosystem contains two useful libraries that can take data sets and use multiple algorithms to try to match them together. Fuzzymatcher uses sqlite’s full text search to simply match two pandas DataFrames together using probabilistic record linkage. If you have a larger data set or need to use more complex matching logic, then the Python Record Linkage Toolkit is a very powerful set of tools for joining data and removing duplicates. Part of my motivation for writing this long article is that there are lots of commercial options out there for these problems and I wanted to raise awareness about these python options. Before you engage with an expensive consultant or try to pay for solution, you should spend an afternoon with these two options and see if it helps you out. All of the relevant code examples to get you started are in this notebook. I always like to hear if you find these topics useful and applicable to your own needs. Feel free to comment below and let me know if you use these or any other similar tools. credits: Title image - Un compositeur à sa casse

Python 3.8.2rc2 is the second release candidate of the second maintenance release of Python 3.8. Go get it here: https://www.python.org/downloads/release/python-382rc2/ Why a second release candidate? The major reason for RC2 is that GH-16839 has been reverted. The original change was supposed to fix for some edge cases in urlparse (numeric paths, recognizing netlocs without //; details in BPO-27657). Unfortunately it broke third parties relying on the pre-existing undefined behavior. Sadly, the reverted fix has already been released as part of 3.8.1 (and 3.7.6 where it’s also reverted now). As such, even though the revert is itself a bug fix, it is incompatible with the behavior of 3.8.1. Please test. Timeline Assuming no critical problems are found prior to 2020-02-24, the currently scheduled release date for  3.8.2 (as well as 3.9.0 alpha 4!), no code changes are planned between this release candidate and the final release. That being said, please keep in mind that this is a pre-release of 3.8.2 and as such its main purpose is testing. Maintenance releases for the 3.8 series will continue at regular bi-monthly intervals, with 3.8.3 planned for April 2020 (during sprints at PyCon US). What’s new? The Python 3.8 series is the newest feature release of the Python language, and it contains many new features and optimizations. See the “What’s New in Python 3.8” document for more information about features included in the 3.8 series. Detailed information about all changes made in version 3.8.2 specifically can be found in its change log. We hope you enjoy Python 3.8! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. https://www.python.org/psf/

Alleged Specifications For Surface Book 3 and Surface Go 2 Revealed  Tom's Hardware

NRHA All-Industry Conference to Focus on Disrupters  Hardware Retailing

AMD Epyc Expands Inside Google Cloud  Tom's Hardware

insideBIGDATA Latest News – 2/18/2020  insideBIGDATA

OTN Hardware Market Global Recent Trends, In-depth Analysis, Market Size Research Report Forecast up to 2025  Bandera County Courier

Performance Drives Innovation and Security Vendors Need to Keep Up  Network World

HP Spectre x360 13-aw0053na (2020) review: A diamond in the rough | IT PRO  IT PRO

There's a temptation for in-car displays to add ever more detail, but the XE strips things right back, and is better for it.

Premier leaders face stern Champions League opposition in Madrid. Don't miss a kick with our guide to watching an Atletico Madrid vs Liverpool live stream.

China's government has placed new restrictions on VPNs to prevent citizens from accessing uncensored information about the coronavirus.

Catch all the performances and awards at the Brits 2020- live stream wherever you are in the world.

Microsoft has unveiled the Xbox Series X, finally giving a face and official name to the Xbox Project Scarlett.

Best Buy's Apple Watch sale includes a $100 price cut on the Apple Watch Series 4.

The best deals on cheap laptops in the US.