Yousry Mohamed

Software Engineering Skills for Data Engineers

As data engineering continues to evolve, the convergence of software engineering principles with data-centric tasks is more critical than ever. In this talk, we will explore how integrating software engineering best practices can elevate your data projects and enhance your productivity. Drawing on my experience in Apache Spark, Databricks, and software engineering, I will cover essential topics that will empower data engineers to write more efficient, maintainable, and robust code. Some of the topics covered in this session are: - Data types and their role in building efficient maintainable pipelines - Writing clean testable code - Multithreading and concurrency and how that even aligns with distributed computing frameworks like Spark - Functional programming concepts to produce cleaner code and solve complex tasks - Debugging and troubleshooting hard problems - Dependency management and how it affects cod stability over time - Curiosity to explore things from different perspectives Join this session to bridge the gap between data engineering and software engineering, enhancing your skills and preparing you to tackle the challenges of modern data projects with confidence and finesse.