Big data refers to data that would typically be too expensive to store, manage, and analyze using traditional database systems and that is why organizations are moving towards cloud computing. Cloud computing offers access to data storage, processing, and analytics on a more scalable, flexible, cost-effective, and even secure basis than can be achieved with an on-premises deployment. Instead of investing heavily in data centers or servers before you know how you’re going to use them, you can pay only when you consume the resources, and pay only for how much you consume basically a pay-as-you-go model. One of the…

In this article, you can find a step by step quick start guide on how to send messages from an Apache Kafka topic to Google Cloud Platform using the apache beam data pipeline running on dataflow and create Data lake and Data Warehouse hosted on the cloud for big data analytics.

I already have messages on my Kafka server and if you want to learn how to move database records to Kafka please go through my other article here.

So first we will be moving the data to google cloud storage(GCS) which is a RESTful online file storage web service…

Apache Kafka is a highly flexible streaming platform that supports multiple use cases in modern data architecture. One critical use of Kafka could be for database transaction records. Since Kafka is a distributed system, horizontally-scalable (partitioning), fault-tolerant (replication), low latency and commit log make it a good choice to integrate with your data lake, data warehouse, and BI services. Ideally, in a production environment, you would be required to run a Kafka cluster that is made of more than one Kafka server but for this quick setup guide, I would be running it on Ubuntu in a virtual box. …