Streaming the mobile phone activity data in real time using Big Data Framework : AWS-Kinesis, Lambda and Power BI
In this blog article, I will explain an end-to-end architecture by sending the sensor data from an Android phone to the Cloud and visualize the data in real-time. Here, an android app sends the data to a local machine which acts as a Gateway between my smartphone and Cloud. The data has been constantly received from the android phone and sent it to the cloud using a python function written in the gateway.
Here I have used AWS Kinesis service to stream the data from the mobile to the AWS cloud through the gateway and used AWS lambda to connect to the Power BI using REST API for visualizing the real-time mobile sensor data.
The below explain the project in detail:
2.Pushing Real-time Data.
3.Pulling Real-time Data.
4.Processing Real-time Data.
5.Visualizing Real-time Data.
This architecture shows, the services and components that is been used in this project.
- Android smartphone to get the built-in sensor data.
- Local Machine to collect the mobile phone sensor data by acting as a gateway.
- Kinesis, Kinesis Stream and Lambda are the AWS(Amazon Web Services) services.the below diagram shows what we can do with Kinesis and Lambda.
- AWS Kinesis is the service by the Amazon where you can ingest, buffer, and process streaming data in real-time to get great insights instantly.
- AWS Kinesis stream gives you the ability to build custom applications that process or analyze the streaming data for specified needs.Kinesis data stream can continuously capture lots of data per hour from hundreds of thousands of sources.
- AWS lambda is an Event-driven, serverless application service. AWS Lambda is similar to how a platform as a service (PaaS) works. In a platform as a service, you typically create an application and deploy it to a PaaS. Once deployed, the PaaS manages the operational aspects of the application, including scaling and high availability.
- PowerBI is a powerful business analytics cloud-based service provided by the Microsoft Corporation, Which is used to visualize the insights of your data.
2.Pushing Real-time Data
Here the android phone sends data to the local machine which acts as a Gateway, this computer sends that data to the cloud.
An Andriod app (Sensor node) sent built-in mobile sensors data to the defined socket address and python socket code to push the data from computer to AWS Cloud.
This Android app collects the data from the sensors and sends that stream data to the Local Machine.Python socket code, which has been written on the local machine will collect the stream data coming from mobile and push it to the Cloud.
This below figure is a screenshot taken from the Android app used in this project.
3.Pulling Real-time Data:
AWS Kinesis pull the data from the Gateway that has created on the local machine using Python socket program.
AWS Kinesis data stream application will pull the stream data coming from the Android mobile through Gateway.
A Kinesis data stream with a name “MobileStream”, is created in AWS.The following steps will show you how to create a Kinesis data stream in AWS.
- Open the Kinesis console at https://console.aws.amazon.com/kinesis.
- Choose Go to the Streams console.
- In the navigation bar, expand the Region selector and choose a Region.
- Choose Create Kinesis stream.
- Type a name for your stream (for example, MobileStream).
- Type 1for the number of shards, but leave Estimate the number of shards you’ll need collapsed.Here shard is a unit of throughput capacity.Each shard supports writes up to 1mbps and reads up to 2mbps.Each shard supports up to 5 reads per second.
- Choose Create Kinesis stream.
By using python code I have connected to the AWS Kinesis using its RESTful API.This created MobileStream named Kinesis data stream will pull the data from the Gateway.In kinesis, all data is stored and replayable for 24 hours by default.
4.Processing Real-time Data
Functions created in the AWS Lambda triggered from the event sources(Kinesis).
AWS Lambda will process the real-time data coming from the Kinesis data stream.
In this project, I have used AWS Lambda service to create a function that sends data to the stream dataset created in the PowerBI using REST API by triggering with kinesis as an event source.
By using AWS lambda we can create applications without servers.we no need to bother about the maintenance and burdens of the server.This lambda is an event-driven service.
In this service, we can create any type of applications with popular programming languages like node js, python, java, Go etc.
Here in this blog, I have created a lambda function called mobilestreamsend to invoke to the kinesis data stream and send stream data to the PowerBI.Below following steps will show you how to create lambda function and add kinesis data stream as event source to it.
- Creating a Deployment Package: Here I have used python as my programming Below is the sample code which can get data from the Kinesis.
- Creating an Execution role: Here you can create your own role to access different resources.or else there are predefined roles to choose and access your required resources in the below following are the roles I used to create Execution role for my process.
In Role Name, use a name that is unique within your AWS account (for example, lambda-kinesis- execution-role).
In Select Role Type, choose AWS Service Roles, and then choose AWS Lambda. This grants the AWS Lambda service permissions to assume the role.
In Attach Policy, choose AWSLambdaKinesisExecutionRole. The permissions in this policy are sufficient for the Lambda function in this blog.
- Creating a Lambda function: Here you can upload deployed package or there is an editor to write code instantly here. Here you can test your function manually by selecting test topics on the top of the page.
- Adding an event source: Here you add an event source in AWS Lambda to associate the Kinesis stream with your Lambda function. After you create an event source, AWS Lambda starts polling the stream and send it to the PowerBI to visualization.
5.Visualizing the Real-time data: In this blog, I used Microsoft’s PowerBI tool to visualize the real-time data coming from the mobile.
The below Screenshots will show you the real-time mobile sensor data visualization.
If there is any continuous movement while the proximity sensor is on, then a warning advisory will be shown on the dashboard with a red color.This may indicate that the user is moving while he was on the call.
The GPS sensor will be monitored along with the noise level and shown in the dashboard, that helps to understand whether the user is in Office, home or in traffic.
We at GlitterPoint are very excited to play with the technologies like IoT and Big data frameworks and witness the potential of these technologies in action.
We are connecting and monitoring many more physical assets to the cloud in real time.
Do follow us for some more interesting updates on our latest projects.