Data Engineer - Hadoop/Spark/Kafka (3-6 Yrs) Hyderabad (Systems/Product Software)

Responsibilities : - Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
-Perform data updates, indexing, and maintenance in application database - Work closely with our backend and frontend engineering teams to log ingest data into our platform and produce a robust high-quality data feed - Assemble large, complex data sets that meet functional / non-functional business requirements - Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS - big data- technologies - Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs - Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader - Keep the data separated and secure across national boundaries through multiple data centers and cloud providers regions. Requirements : - Graduate degree in Computer Science or any related degree - 3 - 5 years of experience in a Data Engineer role - Strong analytic skills related to working with unstructured datasets - Strong project management and organizational skills - Experience with big data tools: Hadoop, Spark, Kafka, Bigquery, etc.
-Experience building and optimizing - big data- data pipelines, architectures and data sets - Experience working on, and deploying, large-scale systems in Python or Go, Scala/Java, or other similar languages - Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement - Build processes supporting data transformation, data structures, metadata, dependency and workload management - Working knowledge of message queuing, stream processing, and highly scalable, bigdata- data stores - Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases - Experience with data pipeline and workflow management tools like Apache Beam/Airflow - Experience with stream-processing systems: Storm, Spark-Streaming, etc.

Required Skills : English