theCodingInterface

theCodingInterface

Providing quality software engineering content in the form of tutorials, applications, services, and commentary suited for developers.

Intro to Machine Learning with Spammy Emails, Python and, SciKit Learn

Machine Learning

Machine Learning is capturing significant attention among technologists and innovators due to a desire to shift from descriptive analytics focused on understanding what happened in the past towards predicting what is likely to occur in the future as well as prescribe actions to take in response to that prediction. In this article I focus on the use case of classifying email messages as either spam or ham with supervised machine learning using Python and SciKit Learn.

Read more
By Adam McQuistan on 03/19/2021
SciKit Learn MachineLearning Python

Example Driven High Level Overview of Spark with Python

Data Engineering

In this article I give a high level, example driven, overview of writing data processing programs using the Python programming language bindings for Spark which is commonly known as PySpark. I specifically cover the Spark SQL DataFrame API which I've found to be the most useful way to write data analytics code with PySpark. The target audience for this article are Python developers, ideally who have a cursory understanding of other popular PyData Stack libraries such as Pandas and Numpy.

Read more
By Adam McQuistan on 03/12/2021
PySpark Data Engineering Python

Introduction to Redshift using Pagila Sample Dataset Including ETL from Postgres using AWS Glue

Data Engineering

In this article I give a practical introductory tutorial to using Amazon Redshift as an OLAP Data Warehouse solution for the popular Pagila Movie Rental dataset. I start with a basic overview of the unique architecture Redshift uses to accomplish its scalable and robust use case as an enterprise cloud data warehouse. Then armed with this basic knowledge of Redshift architecture I move on to give a practical example of designing a schema optimal for Redshift based off the Pagila sample dataset.

Read more
By Adam McQuistan on 03/05/2021
AWS Glue Redshift Data Engineering AWS AWS S3 psql PostgreSQL Databases DevOps Python

Building Data Lakes in AWS with S3, Lambda, Glue, and Athena from Weather Data

Data Engineering

In this aricle I cover creating rudimentary Data Lake on AWS S3 filled with historical Weather Data consumed from a REST API. The S3 Data Lake is populated using traditional serverless technologies like AWS Lambda, DynamoDB, and EventBridge rules along with several modern AWS Glue features such as Crawlers, ETL PySpark Jobs, and Triggers.

Read more
By Adam McQuistan on 02/25/2021
PySpark AWS Glue Data Engineering AWS Serverless Application Model AWS AWS-Lambda DevOps Python

Exploring Online Analytical Processing Databases plus Extract, Transform and, Load in PostgreSQL

Data Engineering

In this article I give an introduction to Online Analytical Processing databases comparing them against traditional Online Transaction Processing Systems. Emphasis is put on designing and building Star Schemas and Reporting tables using Data Engineering processes like Extract, Transform and Load all within a Aurora PostgreSQL database.

Read more
By Adam McQuistan on 02/17/2021
Data Engineering AWS PostgreSQL Databases DevOps

Streaming Logs to S3 with Kinesis Firehose in a Serverless Project

Data Engineering

In this article I demonstrate how to setup a AWS Serverless Application Model (SAM) project for near realtime streaming of CloudWatch logs to S3 using Kinesis Data Firehose. To keep things interesting I'll be using a Python based demo application that exposes two REST APIs, one for scraping and saving quotes from the web to a DyanmoDB table and another for listing the saved quotes.

Read more
By Adam McQuistan on 02/09/2021
AWS Kinesis AWS Serverless Application Model AWS AWS-Lambda AWS S3 DevOps

Navigation

theCodingInterface