Overview

Regulatory programmes are often seen as compliance exercises. In large technology programmes, the reality is broader.

They are data and technology transformation programmes where organisations need to ensure that data is accurate, complete, traceable, secure and available when required.

Regulatory frameworks covering risk data, financial reporting, resolution planning and operational resilience create requirements that reach across source systems, data integration, quality controls, lineage, reconciliation and reporting.

The technology landscape varies by organisation. It may include core banking, finance, lending, trading, customer and operational systems, connected through data integration services and enterprise data platforms. Many modern data platforms use layered approaches such as Bronze, Silver and Gold to organise data processing.

In practice, regulatory reporting is only as reliable as the data and controls behind it.

For a Data Engineering Manager or Technology Delivery Leader, the job is not simply to deliver a reporting solution. It is to connect regulatory requirements to architecture, engineering, governance and delivery, and ensure that reported figures can be explained, reconciled and audited.

Leading large-scale data programmes

Across large financial services data and technology programmes, delivery brings together data engineering, architecture, risk, finance, compliance and technology teams.

The challenge is not only understanding the regulation. It is establishing where data comes from, which source is authoritative, how it is transformed, whether it is accurate and complete, and whether the final reported result can be traced, reconciled and audited.

At scale, delivery also becomes a capacity, dependency and stakeholder management challenge. Priority changes, specialist skill constraints, upstream and downstream dependencies, shared technology platforms and competing programme demands can all affect delivery outcomes. Managing these dependencies and influencing teams outside the immediate delivery structure becomes as important as managing engineering teams themselves.

The practical lessons in this article cover data lineage, data quality, architecture, regulatory reporting, security, governance and delivery across multiple teams.

It starts with the data

When people hear about financial services regulations, compliance is usually the first thing that comes to mind. In large technology programmes, the bigger challenge is often the data behind the compliance requirement.

A regulation may define what an organisation needs to achieve, but the technology organisation must ensure that the underlying data is available, accurate, complete, traceable, secure and delivered on time.

A typical financial services environment does not have one system containing everything required for regulatory reporting. Data can come from core banking, customer relationship management, lending, trading, finance, digital channels and operational systems.

Each system can have different data models, identifiers, formats, interfaces and ownership arrangements.

The challenge is not simply moving data from one system to another. The real question is whether the data retains its meaning and context when it reaches regulatory reporting.

That is where data engineering becomes critical.

Data lineage makes reporting explainable

A key requirement in risk data aggregation and regulatory reporting is understanding where a reported figure comes from.

Imagine a senior risk report showing total credit exposure. The obvious question is:

Where did that number come from?

A well-controlled data platform should allow the organisation to trace the figure backwards:

Regulatory report → Curated reporting dataset → Transformation → Standardised dataset → Raw ingested data → Source system → Original business record

That is data lineage.

If a figure cannot be traced to its source, investigated and reproduced, the organisation has a control gap.

Lineage is more than a technical diagram. It is part of the control framework. When someone asks why a number changed, the organisation needs to follow the data through the platform and understand what happened.

In a regulated environment, “the system produced this number” is not a sufficient explanation.

Data quality is more than checking for nulls

Data quality is sometimes treated as a simple validation exercise. In a regulatory environment, it is much broader than checking whether a field is populated.

The organisation needs to know:

  • Is the data complete?
  • Is it accurate and consistent?
  • Is it available when required?
  • Is it coming from the appropriate source?
  • Has it been transformed correctly?
  • Can the result be reconciled?
  • Is the data owner clearly identified?
  • Can an exception be investigated and resolved?

For example, a customer account may exist in one system but have a different identifier in another. A transaction may be present in a source system but be missing or incorrectly represented in a downstream dataset. A finance balance may not reconcile with a reporting dataset.

These may first appear to be technical defects, but the impact can extend to regulatory reporting and business decisions.

Data quality therefore needs clear ownership, controls and remediation processes, not just validation rules.

Connecting source systems to reporting

Trading, lending, finance and operational platforms can introduce different types of data and processing complexity. The downstream data platform needs to understand the context of each feed and preserve it through ingestion, standardisation and curation.

A typical flow might look like this:

Source systems → Raw data ingestion → Standardised data → Curated datasets → Risk and regulatory reporting

The important part is not simply moving data through these layers. The organisation needs to be able to answer questions such as:

Which business record, reporting date, input data and transformation produced this reported figure?

That is where engineering, data governance and regulatory requirements come together.

Financial reporting and expected credit loss

Financial reporting requirements provide another example of the importance of data controls.

Expected Credit Loss calculations depend on data and models involving concepts such as:

  • Probability of Default (PD)
  • Loss Given Default (LGD)
  • Exposure at Default (EAD)
  • Staging
  • Expected Credit Loss (ECL)

From a technology perspective, the challenge is ensuring that the required customer, product, exposure and historical data is available, controlled and traceable. The result also needs to reconcile with financial reporting.

The engineering question becomes:

Can the reported ECL figure be traced through the model inputs and calculations back to the source data?

There is a significant difference between producing a number and producing a number that can be explained and audited.

Capital, liquidity and risk reporting

Capital and liquidity reporting introduce additional requirements for reliable datasets and controlled calculations.

Depending on the reporting obligation, data may support exposure measures, risk-weighted assets, capital, liquidity coverage and stable funding metrics.

The technology challenge is not simply calculating a metric. The organisation needs confidence that the underlying data is complete, correctly transformed and reconciled.

If a calculation is correct but the input data is wrong, the result is still wrong. Data controls around the calculation are therefore as important as the calculation itself.

Resolution reporting and authoritative data

Resolution reporting requires controlled information about an organisation’s financial position, liabilities, legal entities and other information needed for resolution planning.

This creates a need for reconciled data and a clear understanding of which information is authoritative for a particular reporting purpose.

Different systems can contain different versions of the same information. The organisation needs to establish which version should be used and ensure that downstream reporting uses the controlled data.

This does not necessarily mean there is one system that owns every data element. It means there is a clear understanding of ownership, authority and permitted use for each business or regulatory purpose.

The Medallion architecture approach

Modern data platforms often use a layered architecture to manage data at different stages of processing.

Bronze

This layer contains raw ingested data. The priority is preservation and traceability. Source information, ingestion timestamps and technical metadata help the organisation understand what was received and when.

Silver

This layer contains standardised and quality-controlled data. Transformation, validation, standardisation, deduplication and data quality controls are applied as appropriate.

Gold

This layer contains curated datasets designed for defined business and reporting needs. These datasets may support risk reporting, finance, regulatory reporting and analytics.

The architecture itself does not make a platform compliant. The controls around the architecture matter.

A modern data platform does not solve the regulatory problem if the organisation cannot explain where data came from, who owns it, how it was transformed or whether it reconciles.

The downstream side matters too

Large data programmes can focus heavily on upstream ingestion while giving less attention to downstream consumers.

Regulatory data may feed reporting platforms, risk engines, finance systems, management reporting and other business processes.

A typical flow is:

Source systems → Data platform → Curated regulatory data → Reporting platform → Regulatory submission

Every transition introduces an opportunity for a data quality, reconciliation, security or lineage issue.

It is therefore important to assess the complete flow rather than treat each platform as a separate technology implementation.

Security is part of the data architecture

Regulatory data programmes can involve sensitive customer, financial and operational information. Security and privacy need to be considered as part of the data architecture.

Depending on the use case, controls can include:

  • Personal data protection
  • Encryption
  • Role-based access control
  • Data masking
  • Audit logging
  • Data retention
  • Access reviews
  • Third-party risk controls

Operational resilience adds another dimension. Organisations need to consider technology risk, incident management, resilience testing, recovery and third-party dependencies.

A well-designed data platform must support not only data correctness, but also security, availability and recoverability.

The technology landscape

The technology landscape varies between organisations, but the architecture commonly spans several layers.

Source systems

  • Core banking and transactional systems
  • Customer relationship management platforms
  • Lending and credit systems
  • Trading and risk platforms
  • Finance and accounting systems
  • Digital and operational systems

Data integration

  • Enterprise data integration tools
  • ETL and ELT pipelines
  • APIs and event-based interfaces
  • Batch processing
  • Data ingestion and validation services

Data platform

  • Enterprise data warehouses
  • Data lakes and lakehouse platforms
  • Distributed data processing
  • Layered data architecture

Data processing

  • SQL and transformation pipelines
  • Data quality rules
  • Reconciliation processes
  • Aggregation and regulatory calculations

Reporting

  • Regulatory reporting platforms
  • Risk and finance reporting
  • Business intelligence and management reporting
  • Controlled reporting datasets

Data governance

  • Data lineage and metadata
  • Data ownership
  • Data quality
  • Critical data elements
  • Audit trails

Security and compliance

  • Personal data protection
  • Encryption
  • Role-based access control
  • Security monitoring
  • Control evidence
  • Resilience and third-party risk management

These capabilities do not operate independently. They form one end-to-end data ecosystem:

Source systems
↓
Data integration
↓
Enterprise data platform
↓
Bronze → Silver → Gold
↓
Data quality + lineage + reconciliation
↓
Reporting and analytical services
↓
Risk, finance and regulatory reporting

The responsibility of a Data Engineering Manager is not simply to manage individual technology components. It is to ensure that the end-to-end data flow operates as one controlled system.

The delivery challenge

Technology is only one part of the problem.

Large regulatory programmes normally involve data engineers, solution architects, business analysts, risk specialists, finance teams, compliance, security, infrastructure, testing, operations, third-party providers and business stakeholders.

Getting these teams to deliver against the same outcome can be harder than the technology itself.

In practice, several areas make a significant difference.

Clear ownership

Who owns the data? Who owns the control? Who approves the transformation?

If these responsibilities are unclear, issues can remain open longer than necessary.

Integrated planning

How do multiple teams deliver against the same regulatory milestone?

A change in one source system can affect the data platform, testing, reporting and ultimately the regulatory submission. The plan needs to connect these activities rather than treat them as separate workstreams.

Dependency management

Which source system, provider, interface or downstream platform could affect delivery?

Dependencies need to be visible early, with clear owners and agreed actions.

Data quality governance

Who investigates an issue? Who owns remediation? Who approves the resolution?

Without clear accountability, data quality issues can move between teams without being resolved.

Testing

Can the organisation demonstrate that the data and calculations are correct?

Testing needs to cover more than application functionality. Data reconciliation, transformation rules, lineage and reporting outputs also need to be considered.

Auditability

Can the organisation reproduce what happened at a particular reporting date?

This becomes important when questions are raised after a reporting cycle has completed.

The role of a data engineering manager

The role is not limited to managing engineering teams. It connects regulatory requirements, data architecture, engineering delivery and programme governance into a coordinated delivery model.

The data delivery lifecycle can be represented as:

Regulatory requirement → Business requirement → Data requirement → Architecture → Engineering → Data quality → Lineage → Testing → Reporting → Evidence

The work involves cross-functional teams, including data engineers, architects, business analysts, risk, finance, compliance, testing and operations.

Key responsibilities include:

  • Translating data requirements into deliverable engineering work.
  • Identifying source systems, downstream dependencies and data flows.
  • Delivering data pipelines and transformations with appropriate controls.
  • Assigning clear ownership for data quality issues and remediation.
  • Maintaining data lineage and traceability across the reporting chain.
  • Building reconciliation into the delivery process.
  • Supporting reliable and auditable reporting.
  • Managing dependencies across multiple delivery teams.
  • Making risks, issues and progress visible through programme governance.
  • Aligning delivery with regulatory milestones and business outcomes.

The engineering challenge is only one part of the role. The broader challenge is connecting people, processes, data and technology across a complex environment.

For me, the role of a Data Engineering Manager in a regulatory programme is about more than delivering data pipelines. It is about creating confidence that the right data reaches the right place, at the right time, with the right controls and a clear audit trail.

That is where data engineering moves beyond technology delivery and becomes part of regulatory confidence and business decision-making.

One version of the truth

The phrase “single source of truth” is often used too casually.

I prefer to think about it as:

One controlled version of the truth for a defined business purpose.

Different systems will continue to exist, each serving its own purpose. The objective is not necessarily to replace every system.

The objective is to establish controlled data flows, clear ownership, consistent definitions, reliable transformations and traceability across the landscape.

That is a more realistic approach to complex financial services data environments.

What I have learned

Regulatory compliance cannot simply be added at the end of a technology project. It needs to be considered from the beginning.

The architecture, data model, engineering pipelines, security, testing, operations and delivery governance all need to support it. Governance must also provide evidence that the controls work as intended.

The bigger lesson

Regulatory programmes are more than compliance projects. They are enterprise data transformation programmes with regulatory outcomes.

When data is accurate, traceable, secure and available when required, the technology organisation is doing more than supporting compliance. It is helping the organisation understand its data, support better risk decisions and operate with greater confidence.