Staff Data Engineer - IOT
Posted on: June 12, 2021
INNOVATION & ENGINEERING
Join us in creating, growing, and delivering innovation at pace,
enabling us to thrive while transitioning to a net zero world. All
without compromising our operational risk management.
Working with us, you can do this by:
- deploying our integrated capability and standards in service of
our net zero and safety ambitions
- driving our digital transformation and pioneering new business
- collaborating to deliver competitive customer-focused energy
- originating, scaling and commercialising innovative ideas, and
creating ground-breaking new businesses from them
- protecting us by assuring management of our greatest physical
and digital risks
Because together we are:
- Originators, builders, guardians and disruptors
- Engineers, technologists, scientists and entrepreneurs
- Empathetic, curious, creative and inclusive
Job Family Group
Job Profile Summary
Critical to achieving bp's digital ambitions is the delivery of
our high value data and analytics initiatives, and the enablement
of the technologies and platforms that will support those
As a Data Engineer you will be developing and maintaining data
infrastructure and writing, deploying and maintaining software to
build, integrate, manage, maintain, and quality-assure data at bp.
You are passionate about planning and building compelling data
products and services, in collaboration with business stakeholders,
Data Managers, Data Scientists, Software Engineers and Architects
You will be part of bp's Data & Analytics Platform organisation,
the group responsible for the platforms and services that operate
bp's big data supply chain. The portfolio covers technologies that
support the life cycle of critical data products in bp, bringing
together data producers and consumers through enablement and
industrial scale operations of data ingestion, processing, storage
and publishing, including data visualisation, advanced analytics,
data science and data discovery platforms. You will be part of the
Data Hub team, which is the data clearing house for all of bp's big
data and analytics requirements.
For this role specifically, you will be expected to develop the
necessary platform capability on our data hub to enable handling of
IoT workflows. You will also need to define and curate data
specific to digital twin input and outputs to support the creation
and management of digital twin models. This will also involve
designing and developing the mechanism to be able to integrate IOT
data across different data sources and developing the process to
allow us to integrate different digital twin information models
across a wide technology landscape.
- Design, implement and maintain reliable and scalable data
infrastructure, including design and development of industrial
scale data pipelines on Azure and AWS data platforms and services,
building data ingestion and publishing pipelines, and development
and provisioning of data sets for wide scale access for data
professionals specifically for IOT and digital twin
- Design and develop software for distributed systems, data
warehouses, execute on GDPR and other privacy requirements from
digital security and need to have business context and knowledge
about the data domains they are working in.
- Own the end-to-end technical data lifecycle and corresponding
data technology stack for their data domain and to have a deep
understanding of the bp technology stack.
- Write, deploy and maintain software to build, integrate,
manage, maintain, and quality-assure data, and responsible for
deploying secure and well-tested software that meets privacy and
compliance requirements; develops, maintains and improves CI / CD
- Adhere to and advocate for software engineering best practices
(e.g. technical design, technical design review, unit testing,
monitoring & alerting, checking in code, code review,
- Responsible for service reliability and following
site-reliability engineering best practices: on-call rotations for
services they maintain, responsible for defining and maintaining
SLAs. Design, build, deploy and maintain infrastructure as code.
Containerizes server deployments.
- Actively contribute to improve developer velocity.
- Participate in industry working group for standard
- Actively mentor others, contributes to or leads data
engineering learning and development paths
FORMAL EDUCATION & TECHNICAL SKILLS
- Deep and hands-on experience designing, planning, implementing,
maintaining and documenting reliable and scalable data
infrastructure and data products in complex environments.
- Development experience in one or more object-oriented
programming languages (e.g. Python, Go, Java, Scala, C++)
- Experience with Elasticsearch, Kafka, Redis, HDFS.
- Experience in DB Systems like MySQL or Postgres.
- Services - Event Hub, IoT Hub, Kafka, Azure Data Lake Gen1 &
Gen2, Azure SQL, Azure Synapse (formerly Azure Data Warehouse),
Stream Analytics, Azure Message Queue, HDInsight Spark Clusters,
Azure Synapse Analytics, HDInsight Hive Clusters, AWS IoT Core,
Device Management, Greengrass
- Experience designing and implementing large-scale distributed
- Deep knowledge and hands-on experience in technologies across
all data lifecycle stages
- Strong stakeholder management and ability to lead teams through
- Continuous learning and improvement mindset
- BS degree in computer science or related field
- No prior experience in the energy industry required
- Data Engineering: Ability to build cloud data solutions and
provide domain perspective on storage, big data platform services,
serverless architectures, Data bricks, vendor products, RDBMS,
DW/DM, NoSQL databases and security. Experience in micro-service
architecture added bonus.
- Asset Data Sets: Knowledge of sensor type data sets as well as
cad reference data, including experience in classification,
ontology and designing workflows of such data in data
- Data Manipulation: debug and maintain the end-to-end data
engineering lifecycle of the data products; design and
implementation of the end-to-end data stack, including designing
complex data systems, e.g. interoperability across cloud platforms;
experience on various types of data (streaming, structured and
un-structured) is a plus.
- Software Engineering: hands-on experience with SQL and NoSQL
database fundamentals, query structures and design best practices,
including scalability, readability, and reliability; you are
proficient in at least one object-oriented programming language,
e.g. Python [specifically data manipulation packages - Pandas,
seaborn, matplotlib], Apache Spark or Scala;
- Scalability, Reliability, Maintenance: proven experience in
building scalable and re-usable systems that are used by others;
knowledge and experience in automating operations as much as
possible and identifying and building for long-term productivity
over short-term speed/gains, and execute on those opportunities to
improve products or services.
- Data Domain Knowledge: proven understanding of data sources and
data and analytics requirements and typical SLAs associated to data
provisioning and consumption at enterprise scale.
- Standards/Best Practices: understanding of leading insight of
industry and technology trends and best practices for data product
life cycle; demonstrable knowledge of data engineering best
- Right approach / tool choice: proven experience of a wide range
of several commonly available data engineering and data
infrastructure approaches and tools, latest developments in the
field, and ability to mentor others in selecting the right
approaches to solve problems.
- Agile Methodology: good knowledge and understanding of modern
development methodologies (Agile using Scrum and/or Kanban).
- Stakeholder Management: Strong stakeholder management and
ability to lead and work with multiple product teams with competing
Keywords: BP, Houston , Staff Data Engineer - IOT, Other , Houston, Texas
Didn't find what you're looking for? Search again!