AWS Glue custom classifier enables you to catalog the data in the way you want when AWS Glue built-in classifiers cannot. It is important to catalog the data correctly and the classifier plays an important role in identifying the structure of underlying data. If the built-in classifiers do not catalog…

All of you would have come across this meme. It’s funny right but it is not so funny when it actually happens. While working with EC2 instances, it happens that, we or anyone might forget to stop the EC2 instances which are not required to run over the weekends, and…

In a scenario where it is critical to get alerts or keep track of the login events in the AWS console for the root or any specific IAM users, there is a requirement for a mechanism that can send notifications based on the login event. One of the use-case for…

S3 service is used to store a variety of data for various use cases. Ideally, when there is a requirement to access the data stored as an object, the object will be accessed as a whole entity. But what if you want to retrieve the subset of data from the…

Blob storage is an object store to store vast amounts of unstructured data, blob storage is an option. It uses a flat namespace to store the data. The blob storage can have multiple containers and each container contains the blob objects/data. Suppose, there is a requirement to mount to the…

Containers is one of the preferred ways to deploy applications. It contains everything that the application needs to run. As we know that data does not persist when the container is killed or if it does not exist anymore. For some reason, if there is a need to persist data…

AWS Glue is a fully managed serverless ETL service. It makes it easy to discover, transform and load data that would be consumed by various processes and applications. If you want to learn more about AWS Glue then please refer to the video on AWS Glue Overview

Today, we generally need to do remote development every now and then for different reasons. Let’s try to understand the need for remote development with an example and is not limited to this. For instance, if you are working on a big data engineering project or data science project where…

Have a static web application hosted on S3 for a specific group and wanted to restrict the traffic to certain IP addresses? Let’s see how this can be achieved. To restrict access to whitelisted IP addresses we will use a bucket policy. …