Recently, I've been working a lot with PySpark in AWS EMR. I have a huge data dump (~300 million users) that I needed to process and transform it into the right format for further processing. The data is in a S3 bucket in AWS and I use AWS EMR ...
Amazon's item-item Collaborative filtering recommendation algorithm [paper summary]
This paper published in 2003 by Amazon explains how their basic recommendation algorithm works and is one of the papers that anyone interested in recommendation systems should give a read. I have been working in RS (recommender systems) domain at my present company for the last two years. Since the ...
Types of data in recommender systems
There are two ways in which we can collect data for building recommender systems — explicit and implicit. In this post, we will talk about both types of data, their characteristics and the challenges with them.
Explicit feedback datasets
The dictionary meaning of explicit is to state clearly and in detail ...
Handling errors with try-catch in Python
In the previous post I discuss about how to convert a string to date format in Python. I was working on similar idea today. I had a column of object type which was string of dates. The column name is 'signed_up_at' and I wanted to convert it to date format ...
Working with dates in Python
I cringe every time I see a date type column in the data. And you may ask why so? Date columns need some methods applied to them
The reason is I don't normally see date columns in the data I work with so I don't remember the functions ...
git and github for data scientists
It has been close to a year since I shifted to a start-up which incidentally got acquired after a month of my joining. Before this I used to work at WalmartLabs where we always wanted to use a version control system like git but it never took off properly. Now ...
Creating a virtual environment in Python
I was trying to get a virtual environment set up on Python 3 using mkvirtualenv but somehow the virtual environment was getting created on Python 2.7 (my system python).
If you already know about virtual environments and why they are useful, you may skip the next two paragraphs. I ...
An easy starter to music theory
I started playing guitar almost three years back and at that time I had wished it would have been better if I had picked this lovely ...
Common docker commands
I recently got to know about dockers. And I love it. For those who don't know what dockers are. Here it is. Dockers help in software ...
How to choose the probability cut-off in classification problem
Yesterday, I was taking a session on Data Science for few of my colleagues. The aim was to give a brief overview of machine learning. There were two of us taking the session. We had a rough idea what all we wanted to cover in the two hours session.
I ...