Professional Data Engineer

There are no pre-requisites for taking this course

• Design, build and operationalise data solutions
• Process data streams in real-time
• Efficiently store and access data in the cloud
• Use GCP’s pre-trained AI APIs (vision, speech and text)
• Train and operationalise ML models

• Cloud and network administrators
• Data engineers
• Data and systems analysts

Selecting the appropriate storage technologies. Considerations include:

• Mapping storage systems to business requirements
• Data modelling
• Trade-offs involving latency, throughput, transactions
• Distributed systems
• Schema design
• Designing data pipelines. Considerations include:
• Data publishing and visualisation (e.g., BigQuery)
• Batch and streaming data (e.g., Dataflow, Dataproc, Apache Beam, Apache Spark and Hadoop ecosystem,    Pub/Sub, Apache Kafka)
• Online (interactive) vs. batch predictions
• Job automation and orchestration (e.g., Cloud Composer)
• Designing a data processing solution. Considerations include:
• Choice of infrastructure
• System availability and fault tolerance
• Use of distributed systems
• Capacity planning
• Hybrid cloud and edge computing
• Architecture options (e.g., message brokers, message queues, middleware, service-oriented architecture, serverless functions)
• At least once, in-order, and exactly once, etc., event processing
• Migrating data warehousing and data processing. Considerations include:
• Awareness of current state and how to migrate a design to a future state
• Migrating from on-premises to cloud (Data Transfer Service, Transfer Appliance, Cloud Networking)
• Validating a migration
• Building and operationalising data processing systems
• Building and operationalising storage systems. Considerations include:
• Effective use of managed services (Cloud Bigtable, Cloud Spanner, Cloud SQL, BigQuery, Cloud Storage, Datastore, Memorystore)
• Storage costs and performance
• Life cycle management of data
• Building and operationalising pipelines. Considerations include:
• Data cleansing
• Batch and streaming
• Transformation
• Data acquisition and import
• Integrating with new data sources
• Building and operationalising processing infrastructure. Considerations include:
• Provisioning resources
• Monitoring pipelines
• Adjusting pipelines
• Testing and quality control
• Operationalising machine learning models
• Leveraging pre-built ML models as a service. Considerations include:
• ML APIs (e.g., Vision API, Speech API)
• Customising ML APIs (e.g., AutoML Vision, Auto ML text)
• Conversational experiences (e.g., Dialogflow)
• Deploying an ML pipeline. Considerations include:
• Ingesting appropriate data
• Retraining of machine learning models (AI Platform Prediction and Training, BigQuery ML, Kubeflow, Spark ML)
• Continuous evaluation
• Choosing the appropriate training and serving infrastructure. Considerations include:
• Distributed vs. single machine
• Use of edge compute

• Hardware accelerators (e.g., GPU, TPU)

AWS Technical Essentials
Systems Operations on AWS
Architecting on AWS