Dylan Wan

I think that the process of building a data mining scoring engine is similar to develop an application.

We have the requirement analysis, functional design, technical design, coding, testing, deployment, etc. phases.

I think that the process of building a data mining scoring engine is similar to develop an application.

We have the requirement analysis, functional design, technical design, coding, testing, deployment, etc. phases.

Found a very good paper: http://research.microsoft.com/pubs/204499/a20-appuswamy.pdf

This paper discuss if it is a right approach of using Hadoop as the analytics infrastructure.

It is hard to argue with the industry trend.  However, Hadoop is not new any more.  It is time for people to calm down and rethink about the real benefits.

Oracle Transactional Business Intelligence (OTBI) is one of the business intelligence solutions provided as part of Fusion Applications. 

To build a real-time BI, the major challenge is to make sure that it can perform and has no or minimum interfere to the core objective of the transactional application, the online processing.

This is the reason why we need Oracle Business Intelligence Applications (OBIA) for Fusion Applications.  The idea is to keep the minimal processing of detecting changes and capturing changes in the transactional system and leave everything else, such as, preparing and consolidating the data for reporting, to BI Applications.

Here are some of the technologies available to make OTBI possible:

1. SQL Trimming from ADF

ADF stands for Application Development Framework.  It is the application development framework used in developing Fusion Applications.  In general, it is a declarative metadata driven framework to let the application developers to define the data model, define the data access layer, define the UI rendering, put the validation logic and processing in the middle tier.

The underlying data model, in most of cases, is still the relational model defined in the Fusion Apps transactional database under the 3rd NF design.

The key enabling technologies provided from ADF to OTBI is the “Composite VO” or “Composite View Object”.  For me, it can generate the database SQL for us based on the metadata.  Unlike the database technology using the database view, ADF engine can look further down to the entity objects included in the view object and selectively choose which entities are needed in a given SQL.  If the view object includes two tables (EOs), one primary EO for data at the line level, and the other EO for getting the parent data, When the query (Composite VO) does not include any column from the parent EO, the SQL generated by ADF will not include the table in the join. 

This is a superior technologies, comparing to the old technologies of building the business views.

If you are a Java programmer and would like to get the feeling about what Composite View Object looks like and how it works, here is a good blog post:

Do you know what is a Composite View Object?

2. BI Platform – ADFQuery to Composite VO

This enabling technology is provided by BI platform and available as a Java library. It adds a layer on top of the ADF composite VO.  Without writing the Java code, it generates the codes of creating the composite VO on the fly.  It allows us to query the data from the ADF engine by sending them a XML block called ADFQuery.

This doc shows some of the ADFQuery XML blocks.

http://docs.oracle.com/cd/E15586_01/fusionapps.1111/e20836/adf.htm#BIEMG3435

To see better examples, you can find them in NQQuery.log files.

It is a query language like SQL.  You have the section  for the column projection, the join criteria using view links, and the filter using view criteria.

Here are other enabling technologies behind OTBI.

3. ADFQuery generation from BI Server

4. SQL By Pass Database

5. Relational to Dimensional Mapping (Physical Layer to Business Model Layer)

6. SELECT Physical in initialization block

7. ADFQuery Initialization block

8. Physical Lookup function from BI platform

9. Logical Lookup function from BI platform

10. Data Security enabled at the VO layer via Fusion AppCore

11. Applcore Tree Flattening

12. Applcore Business Intelligence Column Flatten VO (BICVO)

13. BI Flexfield VO generator

14. BI Extender via Import Wizard

15. BI View Object created based on the BI EE Logical SQL (BIJDBC)

16. Effective Date VO with as of date filter

17. ADF Application Module to BI variable interface

and more…

Regardless, the goal of these technologies is to enable the users to get the real time data access to the Fusion Apps.  There is really little or no much we can do for providing the feature like data snapshot, pre-built aggregate, multiple currencies, data consolidation and conformance, cross subject area analysis, and the most important, the query performance with complexity logic to be available in a reasonable time without the interfere to the transactional system.

One of the major change introduced in BI Apps 11.1.1.7.1 is the way how we manage the ETL task sequence and trim the unnecessary tasks.

This functionality was accomplished earlier using DAC.  The problem we frequently faced was that the DAC repository and the INFA repository are maintained as two separate repositories.  We have to sync up the name of tasks exactly in order to use DAC to manage the task execution of the Informatica workflow tasks.

Load Plan and Load Plan Generator was designed for addressing this requirement.

Here is a good article that describes the story.

Load Plan Generator – An Inside Look

First Go to OTNwww.oracle.com/technetworkGo to DOWNLOADS -> MiddlewareYou will see Business Intelligence Applications under MiddlewareYou see both 10g and 11g. Click on Oracle Business Intelligence 11g downloadshttp://www.oracle.com/technetwork/middleware/bi-enterprise-edition/downloads/bus-intelligence-11g-165436.htmlYou will see BI Apps 11.1.1.7.1 here:Oracle Business Intelligence Applications (11.1.1.7.1)

Several people are curious about what are OTBI and OBIA, and what are the differences between OTBI and OBIA. I will discuss these in this article.OTBI stands for Oracle Transactional Buisness Intelligence. OBIA stands for Oracle Business Intelligence Applications.Let’s start with OBIA. OBIA is the pre-packaged BI Apps that Oracle has provided for several years. It is the data warehouse based solution. It is based on the universal data warehouse design with different prebuilt adapters that can connect to various source application to bring the data into the data warehouse. It allows you to conslidate the data from various sources and bring them together. It provides a library of metrics that help you measure your business. It also provides a set of predefined reports and dashboards. OBIA works for multiple sources, including E-Business Suite, PeopleSoft, JDE, SAP, and Fusion Applications.OTBI is different. First of all, it is a real time BI. There is no data warehouse or ETL process for OTBI. Second, it is for Fusion Apps only. OTBI is leveraging the advanced technologies from both BI platform and ADF to enable the online BI queries agains the Fusion Applications database directly. In addition, in some area, such as Financial, you can also connect to the Essbase cubes. Unlike OBIA, OTBI does not have a lot of prebuilt dashboards and reports. The reason is that for some advanced analysis, the data need to be prepared. You cannot get eveything you can get from the OBIA data warehouse in OTBI. Both OTBI and OBIA are available from the same metadata repository. Some of the repository objects are shared between OTBI and OBIA. It was designed to allow you have the following configurations: OTBI Only OBIA only OTBI and OBIA coexistIf you implement Fusion Apps, you can enable OTBI. You can use the BI EE Answer to access the prebuild metadata and metrics those are built against the Fusion Apps. You may not get the full powerful prebuild dashboard and repost and prebuilt navigation workflow. However, you can start experiencing what the BI EE based reports look like. You can start bring the data out from your OLTP system. You can provide training to the users to get familar with the subject areas, some of which are shared with OBIA. If you enjoy OTBI and want to further get OBIA with a data warehouse based solution. You can implement OBIA later. Some of the OTBI reports maybe switched to run against OBIA. Some of OTBI reports can continue connecting to Fusion Apps directly. They can coexist in a single BI server and a single BI answer client.Both OTBI and OBIA are accessing Fusion Apps via the ADF. This is a more advanced topic.

1. It does not allow you to use parameters to the PeopleSoft connect. It may be changed later. However, it was a big issue when we try to address customer issues.

2. It requires EFFDT as an option.It expect that people change the EFFDT using Mapping Editor. How can a business user does that every month?

3. It asks for a Tree Name. Many PeopleSoft tree structure supports multiple trees. Tree is just a header of the hierarchy. Whenever you add a new Tree, you need to create a new mapping!!

It does not make sense to use PowerConnect due to the customer demands. All requirements are from customers.

We have no choice but stop using it.

1. The number of days is different in each calendar month.
2. The week and month cannot be aligned. The number of weekends is different in each calendar month.
3. The number of working days is different in each calendar month. It ends up that the number of days in each quarter is also different.
4. The period closing day will fall into different days in each period. The accounting department prefers always close the period by a given day in a week, such as Wednesday or Friday.

In this posting, I will discuss the following topics:

* What is the 13 period calendar? * Who uses the 13 period calendar? * How is it different from the 4-4-5 calendar?

Read the rest of this entry »

My Data Warehouse Presentation.

I read an interesting article, IBM DB2 Minus OLAP from the SQL Server magazine. Essbase used to be OEM-ed and re-branded by IBM as IBM DB2 OLAP server for ten years. The relationship stopped two yeas ago.

Many DB2 customers actually built their custom analytics applications on the top of Essbase.

I found a very old book, called Decision Support Systems: An Organizational Perspective, in a library last weekend. It was written by Peter Keen,an author of several popular books, which help many business managersand users understand the value of information technology. His DSS bookdraw my attention because he is also the author of my textbook Network in Actions.

The DSS book uses a very typical and conventional categorization system which puts the IT systems into three types:

Transactional System, Structure Decision system, and Decision Support System.

These categories are created based on the classification ofdecisions into structured, unstructured, and partially structureddecision. His focus is the 3rd category, DSS. Peter believes that a DSSshould assist in solving the semi-structured problems. A DSS shouldsupport, not replace, the managers.

I feel that the above is a very good framework to view the role ofan analytics apps. A BI analytics application should be a DSS solution.However, BI analytics apps can do much more then just a decisionsupport system. BI may help the structured decision making.

BI is not just a collection of reports. The design of a BI analyticsapps needs to consider what are the business decision need to make andwhat kind of information is helpful for making the decision.