HoustonRecruiter Since 2001
the smart solution for Houston jobs

Staff Data Engineer - IOT

Company: BP
Location: Houston
Posted on: June 12, 2021

Job Description:

About BP


Join us in creating, growing, and delivering innovation at pace, enabling us to thrive while transitioning to a net zero world. All without compromising our operational risk management.

Working with us, you can do this by:

  • deploying our integrated capability and standards in service of our net zero and safety ambitions
  • driving our digital transformation and pioneering new business models
  • collaborating to deliver competitive customer-focused energy solutions
  • originating, scaling and commercialising innovative ideas, and creating ground-breaking new businesses from them
  • protecting us by assuring management of our greatest physical and digital risks

Because together we are:

  • Originators, builders, guardians and disruptors
  • Engineers, technologists, scientists and entrepreneurs
  • Empathetic, curious, creative and inclusive

Job Family Group

IT&S Group

Job Profile Summary


Critical to achieving bp's digital ambitions is the delivery of our high value data and analytics initiatives, and the enablement of the technologies and platforms that will support those objectives.

As a Data Engineer you will be developing and maintaining data infrastructure and writing, deploying and maintaining software to build, integrate, manage, maintain, and quality-assure data at bp. You are passionate about planning and building compelling data products and services, in collaboration with business stakeholders, Data Managers, Data Scientists, Software Engineers and Architects in bp.

You will be part of bp's Data & Analytics Platform organisation, the group responsible for the platforms and services that operate bp's big data supply chain. The portfolio covers technologies that support the life cycle of critical data products in bp, bringing together data producers and consumers through enablement and industrial scale operations of data ingestion, processing, storage and publishing, including data visualisation, advanced analytics, data science and data discovery platforms. You will be part of the Data Hub team, which is the data clearing house for all of bp's big data and analytics requirements.

For this role specifically, you will be expected to develop the necessary platform capability on our data hub to enable handling of IoT workflows. You will also need to define and curate data specific to digital twin input and outputs to support the creation and management of digital twin models. This will also involve designing and developing the mechanism to be able to integrate IOT data across different data sources and developing the process to allow us to integrate different digital twin information models across a wide technology landscape.


  • Design, implement and maintain reliable and scalable data infrastructure, including design and development of industrial scale data pipelines on Azure and AWS data platforms and services, building data ingestion and publishing pipelines, and development and provisioning of data sets for wide scale access for data professionals specifically for IOT and digital twin requirements.
  • Design and develop software for distributed systems, data warehouses, execute on GDPR and other privacy requirements from digital security and need to have business context and knowledge about the data domains they are working in.
  • Own the end-to-end technical data lifecycle and corresponding data technology stack for their data domain and to have a deep understanding of the bp technology stack.
  • Write, deploy and maintain software to build, integrate, manage, maintain, and quality-assure data, and responsible for deploying secure and well-tested software that meets privacy and compliance requirements; develops, maintains and improves CI / CD pipeline.
  • Adhere to and advocate for software engineering best practices (e.g. technical design, technical design review, unit testing, monitoring & alerting, checking in code, code review, documentation).
  • Responsible for service reliability and following site-reliability engineering best practices: on-call rotations for services they maintain, responsible for defining and maintaining SLAs. Design, build, deploy and maintain infrastructure as code. Containerizes server deployments.
  • Actively contribute to improve developer velocity.
  • Participate in industry working group for standard evolution.
  • Actively mentor others, contributes to or leads data engineering learning and development paths

Job Advert


  • Deep and hands-on experience designing, planning, implementing, maintaining and documenting reliable and scalable data infrastructure and data products in complex environments.
  • Development experience in one or more object-oriented programming languages (e.g. Python, Go, Java, Scala, C++)
  • Experience with Elasticsearch, Kafka, Redis, HDFS.
  • Experience in DB Systems like MySQL or Postgres.
  • Services - Event Hub, IoT Hub, Kafka, Azure Data Lake Gen1 & Gen2, Azure SQL, Azure Synapse (formerly Azure Data Warehouse), Stream Analytics, Azure Message Queue, HDInsight Spark Clusters, Azure Synapse Analytics, HDInsight Hive Clusters, AWS IoT Core, Device Management, Greengrass
  • Experience designing and implementing large-scale distributed systems
  • Deep knowledge and hands-on experience in technologies across all data lifecycle stages
  • Strong stakeholder management and ability to lead teams through influence
  • Continuous learning and improvement mindset
  • BS degree in computer science or related field
  • No prior experience in the energy industry required


  • Data Engineering: Ability to build cloud data solutions and provide domain perspective on storage, big data platform services, serverless architectures, Data bricks, vendor products, RDBMS, DW/DM, NoSQL databases and security. Experience in micro-service architecture added bonus.
  • Asset Data Sets: Knowledge of sensor type data sets as well as cad reference data, including experience in classification, ontology and designing workflows of such data in data platforms
  • Data Manipulation: debug and maintain the end-to-end data engineering lifecycle of the data products; design and implementation of the end-to-end data stack, including designing complex data systems, e.g. interoperability across cloud platforms; experience on various types of data (streaming, structured and un-structured) is a plus.
  • Software Engineering: hands-on experience with SQL and NoSQL database fundamentals, query structures and design best practices, including scalability, readability, and reliability; you are proficient in at least one object-oriented programming language, e.g. Python [specifically data manipulation packages - Pandas, seaborn, matplotlib], Apache Spark or Scala;
  • Scalability, Reliability, Maintenance: proven experience in building scalable and re-usable systems that are used by others; knowledge and experience in automating operations as much as possible and identifying and building for long-term productivity over short-term speed/gains, and execute on those opportunities to improve products or services.
  • Data Domain Knowledge: proven understanding of data sources and data and analytics requirements and typical SLAs associated to data provisioning and consumption at enterprise scale.
  • Standards/Best Practices: understanding of leading insight of industry and technology trends and best practices for data product life cycle; demonstrable knowledge of data engineering best practices.
  • Right approach / tool choice: proven experience of a wide range of several commonly available data engineering and data infrastructure approaches and tools, latest developments in the field, and ability to mentor others in selecting the right approaches to solve problems.
  • Agile Methodology: good knowledge and understanding of modern development methodologies (Agile using Scrum and/or Kanban).
  • Stakeholder Management: Strong stakeholder management and ability to lead and work with multiple product teams with competing priorities

Keywords: BP, Houston , Staff Data Engineer - IOT, Other , Houston, Texas

Click here to apply!

Didn't find what you're looking for? Search again!

I'm looking for
in category

Log In or Create An Account

Get the latest Texas jobs by following @recnetTX on Twitter!

Houston RSS job feeds