When investing in modern data architecture, it’s critical to explore and understand how design decisions can affect your business in the future. As a key step in designing a data platform, your organization should think through both the technical implementation of your data architecture and the business processes it will impact.
Our first two blog posts in this series discuss use cases for temporal tables and correcting historical data in temporal tables. In our third blog post, we discussed topics to consider when integrating temporal tables into existing data architecture. In our final blog post, we’ll discuss compliance requirements that your organization should understand before using temporal tables in your data architecture. Specifically, we’ll explore SOX compliance and privacy regulations.
The Sarbanes-Oxley (SOX) Act is legislation that holds companies accountable for the accuracy of their financial reporting. This means that an organization’s financial data must meet certain requirements to stay in compliance with regulations.
While SOX doesn’t have specific requirements about how data is stored, it does require that companies have controls in place to ensure the accuracy and security of financial data. The implication is that security and change tracking should be designed into an organization’s data architecture, especially in the case of databases containing financials. This can be achieved through integrating with access management like active directory, keeping user activity logs, and implementing data change tracking.
Temporal tables are a great implementation option for data change tracking. In the case of financial audits, temporal tables allow you to track the changed value and exactly when the change was made.
An important caveat in this discussion is the impact of manual data changes. In our previous blog post, we described how to disable system versioning and make corrections to historical data in temporal tables. In this case, the auditing mechanism of temporal tables is disrupted. Organizations should use extreme caution when this impacts financial data, and any manual changes should be done with a controlled execution approach. We describe best practices for making manual changes in detail in our previous post.
If you need to make manual changes to temporal tables, we recommend adding metadata columns to store the last update user and timestamp. During any changes to historical records, updating these fields will make the change easy to trace and help an organization stay in compliance with SOX regulations.
In addition to SOX regulations, a conversation about long-term compliance considerations would not be complete without a discussion of privacy regulations, which we’ll address in the next section.
In the evolving landscape of data privacy regulations, every organization needs an in-depth strategy on how to store and track personally identifiable information (PII). Both Europe’s General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA) open organizations to the possibility of having to erase a customer’s PII data upon request. As outlined in part two of the Temporal Tables Series, changing historical data within temporal tables comes with additional difficulties when compared to a typical database table. A normal process for editing data in historical tables will include altering the table to run update statements, but this results in downtime for the table and its downstream users and increased level of effort for developers.
With this in mind, each organization will need to evaluate if this process is wise to implement for tables that may include PII data. Below are a few options to weigh when deciding how to approach temporal tables if PII data is present:
Use temporal tables and create a standard set of actions when needing to erase PII data.
Use temporal tables but mask PII data or use keys to obfuscate the explicit customer information.
Don’t use temporal tables when PII data might be included.
Need Help With Compliance?
Developing a modern data solution can help your organization turn data into actionable insights. As you develop a data platform, it’s important to consider how your architecture design will impact your business in the future. Your organization should consider compliance concerns and have a plan for addressing regulations like SOX, GDPR, and CCPA.
If your company is wondering how to design a sustainable, flexible modern data platform, feel free to reach out to us at firstname.lastname@example.org. We’d love to help your business develop a data architecture solution that sets you up for long-term success.