How to create large data sets for performance test

From: Lok P <loknath.73_at_gmail.com>
Date: Tue, 30 May 2023 02:25:33 +0530
Message-ID: <CAKna9VaG5v0juhtQ56bE9jkXz7GCC3nxVMSdu66Pffp+yiLsuA_at_mail.gmail.com>



Hello Listers ,
We have one of the existing production system(Oracle database 19C Exadata) which is live and running on premise(Its a financial system).We have this data replicated to cloud(AWS S3/data lake) and then multiple transformation happens and finally moved to multiple downstream system/databases like Redshift, Snowflake etc on which reporting and analytics APIs/application runs.

For doing performance tests for these reporting/analytics applications and also the data pipeline , we need to have a similar volume of data generated with the same data pattern/skewness and also with the same level of integrity constraints maintained as it exists in the current oracle production database. Performance of the databases like snowflake solely depends , the way incoming data is clustered and for that it's important that we have similar data pattern/skewness/order as that of the production environment or else it wont give accurate results. We are getting ~500millions of rows loaded into our key transaction Table(atleast 5-6 tables are ~10+TB in size in production Oracle database) on a daily basis in the current production system. Current production system holds ~6months of data. And we want to do a performance test at least on the ~3 months worth of data.

We thought of copying the current oracle production data to the performance environment , however the production data has many sensitive customer data/columns which can't be moved to other environments because of compliance restrictions. And also joining the masked column data can be challenging if they are not the same across tables. So I wanted to understand from experts, if there any easy way(or any tool etc) to generate similar performance data in such a high volume in quick time for the performance testing need?

Regards
Lok

--
http://www.freelists.org/webpage/oracle-l
Received on Mon May 29 2023 - 22:55:33 CEST

Original text of this message