Data Engineer Interview Questions-Capgemini

1. Describe your work experience.
2. Provide a detailed explanation of a project, including the data sources, file formats, and methods for file reading.
3. Discuss the transformation techniques you have utilized, offering an example and explanation.
4. Explain the process of reading web API data in Spark, including detailed code explanation.
5. How do you convert lists into data frames?
6. What is the method for reading JSON files in Spark?
7. How do you handle complex data? When is it appropriate to use the “explode” function?
8. How do you determine the continuation of a process and identify necessary transformations for complex data?
9. What actions do you take if a Spark job fails? How do you troubleshoot and find a solution?
10. How do you address performance issues? Explain a scenario where a job is slow and how you would diagnose and resolve it.
11. Given a dataframe with a “department” column, explain how you would add a new employee to a department, specifying their salary and increment.
12. Explain the scenario for finding the highest salary using SQL.
13. If you have three data frames, write SQL queries to join them based on a common column.
14. When is it appropriate to use partitioning or bucketing in Spark? How do you determine when to use each technique? How do you assess cardinality?
15. How do you check for improper memory allocation?