SCHEDULING & MONITORING PROGRAMS (TO PROCESS 100 TO 5000 FILES A DAY)

From: Ranjeesh K Ramakrishnan <ranjeesh007_at_yahoo.com>
Date: 22 Apr 2004 00:18:38 -0700
Message-ID: <76ee6a.0404212318.42752d98@posting.google.com>

Hi all,

We have the end to end set of programs for scheduling & monitoring any ETL programs. In big enterprises , the system to system data access always happens through ascii files. if there are systems which needs to recieve/load files & produce handoff files periodically and if the number of files handled in day are quite large (eg:atleast 100 to 5000 files ) you can make use of our programs.The features of the scheduler are

[b]The ETL Program Scheduling[/b]

Incoming files are grouped together ( eg: system_name + country + branch combination , Not necessary ), Similarly outgoing files as well.
Files will get loaded only when the whole set is recieved
More priority can be given to certain sets depending upon the need
Loading set to Loading set dependency (very rare cases)
Output set to loading set dependency (The output set will be generated when all(if any) dependent loading sets are loaded.
Order of execution of files in a set (for both loading / output )
Statistical information about everyday's loading files (Recived time , Start time & End time of a particular file on anyday)
5 Parrellel queues for loading and 5 for output generation ( Multi threading )
Output files are automatically ftp'd if required
Esay query mechanism to know which set failed , Which file/program failed in
Status of execution of any file by ETL program from multiple sources ( Database , File Location in diff. directories , From log files ) .. and more
Dynamic renaming of files & execution

[b]Monitoring Program[/b]

For the Data centre / Back office processing people
For a incoming set, when crossed the deadline time, this monitor log shows the contact details
When anything gets errord out the monitor log gives the set & contact details
Holiday maintenance for any group can be maintained.

[b]End of day programs[/b]

To take care of purging & archiving of files, database & logfiles etc.

Tested with 2 ETL tools ( Constellar hub from DataMirror & Oraclewarehouse builder) and this scheduler is live in an International bank (confidential) in 3 regions (Asia Pacific , Europe & North America) "PROCESSING 1000+ FILES A DAY" at each of the 3 regional data centres for the last 10 years ( constellar ) .

[b]Technology Specifications[/b]

Operating System > HP Unix
Database > Oracle
Software > Couple of Unix shell scripts & Pro*C code

[b]IF INTERESTED PLS MAIL ME BACK AT ranjeesh007_at_yahoo.com (i.e; ranjeesh007 at yahoo dot com)
[u]WITH YOUR REQUIREMENT DETAILS[/u] [/b]

Cheers

Ranjeesh K Ramakrishnan
Mumbai
0091-98216 41777 Received on Thu Apr 22 2004 - 02:18:38 CDT