Exposing Big Edge Data: What Big Cloud Providers Don't Want You to Know

In this session, I will demystify managing 'Big Data in Motion' from edge devices to the cloud. We'll delve into the innovative engineering design I've recently implemented, leveraging the capabilities of AWS Greengrass and StreamManager. This strategy offers an efficient and flexible IoT data ingestion and analysis pathway. The discussion will highlight how this approach distinguishes itself from and enhances the traditional out-of-the-box solutions that large cloud providers typically endorse. Key Topics: • Deciphering the Challenge: An in-depth investigation into the complexities of managing high-frequency data streams from edge devices. • Alternative Engineering Design: A comprehensive exploration of my pioneering approach that utilizes AWS Greengrass and StreamManager to ship high-frequency data whilst still maintaining flexibility. • Unmasking the Data: A practical demonstration of how query services like Athena can be utilized directly on raw data from the edge. Further, I will showcase how Grafana can be effectively employed for rapid historical data visualization. By the end of this session, you will witness a practical demonstration, complete with code, illustrating how high-frequency raw data can be readily exposed to teams without investing significant engineering time in constructing complex pipelines. Gain insights that big cloud providers might prefer to keep under wraps.