Cloud Pipeline / Data Engineer

Cloud Pipeline / Data Engineer

Melbourne, Sydney, Canberra, Brisbane CBD, Adelaide
Baseline required
12 months with 2 x 12 months extension

About the team:

You will join a large multi-year transformational program which will establish a new data platform in the cloud for the department, using a variety of PaaS services. The program operates using a scaled agile release train made up of agile teams utilising Scrum.

About the Role:

The Cloud Pipeline/Data Engineer will work within an agile team to deliver new ingestion, storage and consumption patterns using AWS cloud native technologies and third-party products, that adhere to the security and policy frameworks.

As the Cloud Data Engineer you will:

• Develop AWS Lambda and Glue
• Write SQL scripts and Stored Procedures for RDS.
• Deploy AWS services using CloudFormation, SAM, and CI/CD DevOps pipelines.
• Drive out issue resolution arising from build reviews (including security, integration, technical challenges).
• Assist in planning, prioritisation, and assessing readiness of features prior to delivery commitment.
• Participate in unit testing activities – management of data, test resolution and oversee the completion of TFS tasks, incorporating test sign offs.

Demonstrated skills and experience:

• Highly proficient in:
o Python language and Apache Spark framework.
o AWS cloud native services such as Lambda, Glue, Step Functions, CloudFormation, S3 and RDS.
o Writing SQL, Spark SQL, and Stored Procedures.
• Writing unit tests with frameworks like Pytest and Moto.
• Working with other AWS cloud native services such as IAM, CloudWatch, CloudTrail, Kinesis, EventBridge, SQS and SNS.
• Big data/analytics platforms such as Teradata Data Warehouse, Cloudera Hadoop, Databricks.
• Agile delivery methodologies such as Scrum and SAFe is required.

Alternatively, contact Ebony Henderson on (02) 6113 7534 for more information and a confidential chat.

Please quote (ID: 367237)
Sydney (Australia)
I.T. & T
Ebony Henderson