Category Archives: Oracle

OWB SOURCES AND TARGETS SQL

There was a posting on the OWB OTN forum about how to build a report documenting mappings, their sources, and their targets. Patrick Goessens provided an OMBPlus script that fits the bill and should work (perhaps slightly customized) brilliantly. I love OMBPlus and find it to be a very useful addition to the Oracle Warehouse Builder product. I’ve noticed Patricks postings before, and he is in the minority of OWB developers who have command of a very powerful feature.

There might be circumstances when OMBPlus might not be a prefereable option. Not everyone will have OWB installed, or want to pay the additional license costs to just “report” on the metadata. The consumer of this information might not be a “person,” but rather another system or repository. In that case, coordinating the execution of an OMBPlus script, parsing and importing into an alternative application might be troublesome.

I’ve built a small SQL script that runs against the OWB design repository public views (an Oracle provided view into the design metadata). I ran this against the solution for the sample company for my OWB workshop and it reports correctly for it. Use it as a reference, but ensure it works for your actual metadata repository as I don’t claim that this is a complete solution (or even nicely written SQL).
sqplus design_rep/design_rep_password@DB

select
  distinct 'TARGET',
  comp.map_name,
  comp.data_entity_name,
  comp.operator_type
from
  all_iv_xform_map_components comp,
  all_iv_xform_map_parameters param
where
  lower(operator_type)
     in ('table', 'view', 'dimension', 'cube')
     and param.map_component_id = comp.map_component_id
     and param.source_parameter_id is not null
UNION
select
  distinct 'SOURCE',
  t1.c1,
  t1.c2,
  t1.c3
from
  (select
  comp.map_name c1,
  comp.data_entity_name c2,
  comp.operator_type c3,
  max(param.source_parameter_id) c4
from
  all_iv_xform_map_components comp,
  all_iv_xform_map_parameters param
where
  lower(operator_type) in
                           ('table', 'view', 'dimension', 'cube')
     and param.map_component_id = comp.map_component_id
 group by
comp.map_name, comp.data_entity_name, comp.operator_type) t1
where t1.c4 is null
order by 2,1

TARGETS are determined if any of their columns have a source parameter (ie, there’s been a line drawn into them on a mapping). SOURCES are determined if they do not have any source parameters (they don’t have any mapping lines coming “in”). I couldn’t see any special flag or marker to find the “one” target of a mapping, so it had to be inferred. I wonder if anyone from the OWB product team would like to comment on the validity of this logic?

Let me know how people get on with this script; especially if there’s any edge cases or revisions that augment it.

DISCOVERER MEETS DUKE NUKEM

Is there any better way to spend a Monday than to build a 3D Dashboard (using some sample reports from my upcoming OWB course)?

You can obviously see I’m not describing 3D report effects in the context of a web based reporting application, but rather a 3D environment that has the actual dashboard content displayed in 3D space. I had been giving a lot of thought recently to the constant craving of executives for BI dashboards. In fact I think some organizations might consider implementing such a solution because of the following two realities:

  1. Executives love web based dashboards, but what they really want is to have the NASDAQ MarketSite at their office. Screen upon screen of everything they need to know to make great decisions. Building out hundreds of LCD panels might not fly with shareholders though, and I don’t blame them as they are very expensive. With this sort of interface, one can still get the “control room” metaphor without the expense.
  2. The generation of “video game playing and we’ve been using computers since 2nd grade” are now being tapped into the highest C-level positions. It’s not uncommon for an executive to know how to play a little DOOM (even if it was more than a few years back). In other words, there are a significant number of executives that would not feel “uncomfortable” in front of a multiplayer game interface.

However, there are SIGNIFICANT limitations to a 3D space, not the least of which is the ability to handle 2D information quickly and effectively (a la web pages/actual applications). When I began the exercise I was certain that there would be no use in the project unless it could integrate back with EXISTING systems. There is no way one would be able to work effectively entirely in a 3D work, so it would have to play nicely with the current applications. I’ve done a very small integration with Oracle Discoverer where by clicking on one of the dashboard 3D objects you see Discoverer Viewer drilled to the sheet in question alongside the 3D world. Without something like this, where people could get to the “real work”, I think a 3D dashboard is pretty much a nonstarter.

One could take the metaphor further, and have virtual meetings in the control room. Here I’ve pictured the dashboard from the “3rd” person view instead of the “1st” person view. Consider a situation where the District Sales Manager calls the Senior VP of Sales and asks to “show him” some area of concern. He could literally take the SVP/Sales to the dashboard area and they could sift through the data together.

I sort of haphazardly found this 3D software, as I have a customer that used it (ActiveWorlds) as a prototype for a 3D community so their web site visitors could explore articles on their site, chat with each other, play 3D games, etc. While I was at that customer site I helped them integrate some “external media” into the 3D product, and found it rather straightforward and easy. I don’t mean to offend but I found the communities who play 3D games a bit strange and having little to no interest in the external application of their technologies. I found a myriad of resources on the legend and lore of ficticious 3D worlds (and rivalries between them) but few resources on how to “drill through with URL parameters.”

I’ve attached some screenshots in this article, and there is also a 12MB AVI file that demos the solution. I built it on a corporate system so I can’t offer up access to the outside world at this time. However, I greatly value the community traffic and opinions so if there’s enough interest perhaps I’ll see about “putting it up.” Email me if you’d like to actually play with it and I’ll see what I can do.
UPDATE: There were some issues with the CODEC for the AVI. If you had problems previously, feel free to try again.

ORACLE WAREHOUSE BUILDER : WORKSHOP I

UPDATE : The next workshop will be held July 19th ! Classes are limited to 8 seats (very hands on) Email me to get registered…

Some exerpts from workshop participants:

From my viewpoint, (considerable prior training and preliminary knowledge of data warehousing concepts in general) I found the class to be exactly what I needed. I needed a step-by-step, how-do-I-accomplish-the-tasks-in-Oracle kind of course. Without this course, it would have taken me weeks of time I don’t have, to be able to put together a prototype to show the other teammembers I work with the value of the tools. Now I feel confident I can put something together fairly rapidly, using the project database and real problems we are facing. I needed the hands-on approach to gain the confidence in the toolset. Now, if I can get it loaded on the server in short order, I should be fairly competent with the tool by the time the next release comes out, which will have the full set of capabilities the project needs.


I learned so much in a day that I never found on the OWB documentation. I never thought we can do so much in mapping.

I’ve developed a one day workshop to help those interested in how to get up and running QUICKLY with OWB. If any blog readers are in the greater Seattle area (or have always need an excuse for a weekend here) consider taking this course. Email me directly if you’re interested, and I can help get you registered for the workshop at the PSOUG lab on Mercer Island. Note: the PSOUG has not posted the event on calendar yet, but it’s most certainly scheduled for MARCH 26, 2005

ORACLE WAREHOUSE BUILDER : WORKSHOP I

This course teaches students how to install OWB into their database, and begin building Data Objects and ETL Mappings using OWB. Students will learn the basics of OWB architecture, ETL development, and executing their ETL. The students are required to install their own software and configure their own database to ensure they can leave the workshop with all the requisite skills to continue their OWB education at home, or in Workshop II. Students will be able to WOW their colleagues with their new found OWB skills building quick, optimized solutions to common ETL problems.

Course Objectives
After completing the course, students should:

  • Understand OWB architecture, including the distinction between Design Objects and Runtime Objects (and repositories!)
  • Be able to install OWB onto a database server
  • Install and configure design repositories, runtime repositories, and target schemas
  • Be able to retrieve metadata (table definitions, constraints, etc) and actually source data from remote Oracle databases
  • Be able to create and deploy Tables, Sequences, Cubes, and Dimensions.
  • Be able to create and deploy ETL mappings using multi table joins and custom transformation logic
  • Be able to sample, and load data from flat files for use in OWB
  • Be able to execute and troubleshoot ETL mappings built in OWB
  • Load a star-schema that demonstrates the immense benefit of ETL/dimensional modeling

Text and Materials
All materials are supplied to students at the beginning of each class. The class textbook will be optionally available for purchase from PSOUG at a substantial discount.

Class Format
Classes consist of approximately 1.5 hours of lecture supported by slides and whiteboarding. The balance of the day, approximately 6.5 hours, is hands-on. Each students works on their own Linux server.

Prerequisites
To be successful students must be comfortable with basic Oracle administration. The following is a short list of the suggested but not required skills:

  • Understand what schemas are, how they relate to users
  • Understand basic DDL (create table, alter table, create sequence, create or replace package) and DML (select, insert, update)
  • Know how to connect to Oracle using both Localnames (tnsnames.ora) and JDBC (host/port/SID)
  • Students will benefit most when they have a basic understanding of dimensional modeling (Cubes/Facts/StarSchemas)

RITTMAN ON OWB TUNING

It is likely that most readers of the bayon blog are also readers of rittman.net. However, just in case someone missed his posts on tuning OWB Performance Tuning Framework I highly recommend that you read/bookmark them. For the casual OWB user this might not be necessary; developers who deal either with a great many mappings or mappings that process significant data volumes this is a “tool” to keep in your toolbox.

Mark is on to a great method; the application of battle tested tuning techniques to an often used but rarely fully understood tool. I’m not sure how much time I’ll have over the next few weeks, but if I do perhaps I’ll throw my oar in as well. Things I can think of to contribute to the community dovetailing on Mark’s work include:

  • Unify the approach of monitoring the Clock Time and drill down to the performance data Mark has retrieved. ie, build a set of reports (or even a small data mart!) that consumes the public runtime performance views in the OWB runtime repository. I’m a big fan of quantifying the benefit of a tuning excercise. Perhaps a way to track which processes/mappings are slowing your loads, elapsed/time versus volume (ETL throughput), etc. Consume and understand the basic metrics, then drill down on a particular mapping into the information that Mark has developed.
  • An OMBPlus implementation of the “on/off” trace flag Mark refers to :The way of enabling tracing should either be built in by default, or easily added by the OWB developer. Ideally it should be simple to switch it on or off (perhaps levels of event 10046 tracing?). I bet one can create a script that adds/removes the pre/most mapping process that Mark has added manually to the mapping. In this method we could programmatically turn tracing on for a defined list (from one to entire repository) of OWB mappings. The logic to set the key for the process flow operators using OMBPlus might be a wee bit more difficult (would have to iterate through operators in a process and possibly make changes). The biggest drawback, and I don’t see a clear solution at this point, is how to change tracing at RUNTIME instead of deployment. Making the changes via OMB requires a redeployment but it would be most beneficial if tracing was in the code, and changed at runtime.

A great start of the conversation that’s been long overdue… Perhaps a small OWB coming of age? The technology is now used widely enough to warrant some of it’s own methods, patterns, and strategies for not just building but rather building well.

OWB10G PARIS : SCHEDULING

A bayon blog reader wrote me and asked if I had any additional information on scheduling ETL mappings and process flows. I had not fiddled too much with that part of the product; the Oracle provided run_my_owb_stuff.sql and SQL Templates for Enterprise Manager had usually been sufficient for my customers. There are some new scheduling features in Paris and without the reader request I probably would have just passed them up. So while this covers a somewhat smaller feature of the new version, it’s still interesting and makes a nice installation in my OWB Paris Early Review series.

To address scheduling needs there are entirely new OWB objects, appropriately called “Schedule Modules” and “Schedules.” Those current users of the product are well aware that in previous versions of the product, no such objects existed so it’s helpful to show how these are used. One creates a new Module which is merely a container to hold the actual Schedule objects. Note: I haven’t found anything particular interesting about the module definition in terms of deployment or configuration, but perhaps it’s used in the multi-configuration deployments.

Then one creates the schedule that will contain information on when to run an OWB Mapping Process flow. Note: I tested this feature with a Mapping but not a process flow. All the documentation I’ve seen from Oracle mentions that Schedules can be applied to either Mappings or Process Flows.

Now we see a wizard, which happens to be a conundrum. Wizards are usually helpful in providing the most common use cases with a honed interface, however the actual Schedule editor is far more intuitive than the wizard as you’ll see below. I think it a bit strange that the wizard is less helpful than the actual editor; any other testers out there have an opinion please email me!

We’ll define a start and end date for the Schedule which defines the bounds on which it will run. Perhaps you’ve built a Mapping that only runs for the next 3 months during an OLTP system migration so you can define when this schedule will no longer run.

And now we’ll set our recurrence… Don’t worry, you can see more details and have more control over the Schedule in the actual editor.

I’ve now created my schedule, LOAD_EMP_EVERY_2_MINUTES and have opened it in the editor. You can see this is a much better way of visually editing the Schedule. It projects the schedule in a hierarchical manner in the Schedule Preview window (you can flatten it if you so choose). It also allows you to set clauses here, so you can fine tune when it will execute. You might need to use this if you need to limit the Schedule so that it only runs on say Monday Thru Friday when the system is populating data.

Next step is to CONFIGURE the Mapping and let OWB know that we intend to use a certain schedule when we “Run” our Mapping (or flow). For this example, we’ll assume we’ve already created and tested an OWB mapping named “STAGE_EMP_TABLE” that’s been deployed and runs properly via the Control Center.

Now, when we head back into our Control Center panel we see a new Item that hasn’t been deployed named “STAGE_EMP_TABLE_JOB.” We deploy this object

We know see we have an available job in our Control Center. Remember that in Paris, nearly all (if not all) of the Control Center runs asynchronously now i.e. the client doesn’t block while deploying an application or running a mapping or process flow.

Now, according to OWB my Job has been scheduled and is running! It shows the job, when it last ran was, and what the return was of that run. In my example, the last run was sucessful and finished at 6PM on 2/25/2005.

I did manually verify the job was running, ensuring that STAGE_EMP_TABLE was executed properly each time. You can also see a list of the runs and which job they are associated with. The control center is showing me that it is currently running the job, and that it ran the job previously twice successfully. Notice the seperation of the time is two minutes apart, so it appears as if my schedule was about right. This is completion time, so don’t be worried if it doesn’t match up because while your job may start at the correct time, it may finish with varied elapsed times.

I do believe the only way to change the schedule, Start and Stop the Scheduled jobs is through the proper OWB Client. I took a quick look at the OWB Web Based browser and it does not appear as if you can make changes. My Paris copy mentions that the browser isn’t even going to be in the Beta release so I am probably just seeing some partial build with an incomplete version of the OWB Browser.

For those that this feature matters, this will be very helpful. OWB is really shooting to be a one stop shop for all your Data Warehouse/ETL needs and the scheduling feature is a nice little piece of that. Happy OWB scheduling to ya!

This blog is part of the OWB Paris Early Review series which reviews and comments on several new Paris features.

PSOUG OWB LECTURE : DATAMART IN 90 MINUTES

I recently did some training on OWB at my local PSOUG user group, which is not your “run of the mill” user group. Their focus on peer to peer education and quality, technical presenations makes this group stand out in my mind. If you are in the Puget Sound area (the greater Seattle, WA area for those not familiar with US regional geography) you should definitely check out the meetings. The handout from the meeting is here (PDF, Word). Watch for an email about an OWB course at the PSOUG lab!

I promised those in attendance that I would post some notes so that you can do the same excercise. Here are those promised notes, and I hope you find it useful to get you ready and in a state ready to “recreate” the excercise. Please email me with any questions you might have, or if you are planning a Data Warehouse or Data Mart using OWB. I’d love to help out by providing additional coaching, solution planning, or actual implementation.

Step 1 : Install Oracle Database 10g

We assume you know how to install Oracle software and how to use DBCA to create a database. Download the Oracle DB 10g release 1 from OTN here. Follow the instructions to install the product onto your machine. During the process (or after using DBCA) create a database using the Data Warehouse Template, and make sure and install the sample schemas.

Step 2 : Install Oracle Warehouse Builder 10g

There’s not a lot to do at this point. Download OWB from OTN, and follow the instructions on the screen and install OWB into a SEPERATE ORACLE HOME.

Step 3 : Create an OWB Design Repository

Run the following program from your OWB installation ($ORACLE_HOME/owb/bin/unix/reposinst.sh (or if you’re on windows run %ORACLE_HOME\owb\bin\win32\reposinst.bat).

Click “Next”

Enter your sys username, password, Service name, and port number

Select “Create a new Warehouse Builder Repository”

Create and install into a new user

Call it OWB_REP and give it an easy password

The default tablespaces are usually fine, but if you have installed your db differently feel free to change these.

American English is fine in most circumstances and for this demo

Review the information for correctness and then click “Finish”

Hopefully you’re watching this work and

get one of these. You’ve installed an OWB DESIGN REPOSITORY into your database.

Step 4 : Create an OWB Runtime Repository

Run $ORACLE_HOME/owb/bin/unix/runtimeinst.sh (or corresponding file for windows)
Enter in your sys information (sys/password/singapore/1521/bdev.bayontechnologies.com)

Select Runtime Repository

Select Create New Warehouse Builder Repository

Create and install into a new schema

Enter in owb_rt and an easy password twice

The warnings are nothing major… click OK

Default Tablespaces are fine

Install a new runtime access user

Enter in owb_user and an easy password twice

Port number is fine (very few people actually use the name/address cleansing and it costs extra)

Review and Click Finish

Click “Yes” to create a target schema

Select Target Schema

Create a new OWB Target Schema

This is the Runtime Repository we just created, owb_rt/password

Create and install into New user schema

owb_tst/easy password twice

Click OK again on the warning

Default tablespaces are fine

Review and click Finish

You’ve successfully created a runtime repository and a target schema to deploy your test OWB objects to!

Step 5 : Install OWF 2.6.3

Download OWF Server from OTN here.

Click “Next”

This is really important… You must use your database 10g home for installation

Select “Oracle Workflow”

Review and Click Install

Now run $ORACLE_HOME/wf/install/wfinstall.csh (or corresponding .bat if windows)
Install into OWF_MGR, Install, your connection info (bdev, singapore, singapore:1521:bdev.bayontechnologies.com) (LocalName TNSNames, hostname, JDBC string)

Using the SYSDBA account grant execute any procedure to OWF MGR

SQL> grant execute any procedure to owf_mgr;
Grant succeeded.

Step 6 : Unlock the SCOTT account by running the following script as SYSDBA

SQL> alter user scott account unlock;
User altered.

Step 7 : Allow OWB_TST to select from SCOTT tables

grant select on emp to PUBLIC;
grant select on dept to PUBLIC;
grant select on salgrade to PUBLIC;.

Step 8 : Get the demo materials

Download the following items which were used during the demonstration:

OWB Paris Webcast

I recently gave a presentation at NWOUG on the new features of Paris and thought it worthwhile to post a similar presentation on my blog.

This is my first time producing a rich media presentation and did so rather quickly with the help of a tool called Camtasia Studio. Consider yourself duly warned: the audio is not that great, I didn’t do any editing, and it’s just me chatting over some screenshots of the product. You’ll need flash (1024×768 too), and I’ve split it up by segments for easy viewing. If you’re already familiar with OWB consider skipping ahead to OWB 10gR2 Overview and continuing on from there. They are each a few minutes, and are not in great depth but give a sense for the feature and it’s applicability.

I do hope it’s useful for those eager to get some new information on the Paris release of OWB. Feel free to drop me a line if you have Q’s about OWB Paris or otherwise.

Introduction
Presenter
Format/Objectives
Why OWB?
OWB ETL Leader
OWB Demo
OWB Applicability
OWB Concepts
OWB Installation
OWB 10gR2 Overview
OWB Improved UI
OWB ETL
OWB Pluggable Mappings
OWB Data Profiling 1
OWB Data Profiling 2
OWB Data Profiling 3
OWB Data Correction
OWB Right Time BI
OWB Change Data Capture
OWB Experts
OWB Impact Analysis
OWB Misc
More Information

PSOUG 23-FEB-05 ON OWB

I’ll be giving a lecture at the upcoming PSOUG meeting on the 23rd of this month. Those locally that are interested in a free “crash course” in OWB should attend this meeting. It’s always free and usually has an excellent program aimed at technical users. Hope to see you at Oracle’s Bellevue office 108th Ave NE, Suite 1300 from 5:30 to 7pm.

NWOUG Presentation on OWB Paris

On Monday I gave a presentation at the Northwest Oracle Users Group (NWOUG) Winter conference. This was the first NWOUG conference I’ve attended, and was grateful to the NWOUG board for accepting my presentation.
I’ve included the presentation slides here (Powerpoint and PDF) for those readers interested.
As many of you know, I’m writing a series of articles on the new features of Paris which I’m hoping to add more to shortly.

VMware and Oracle Partnership

While looking for some additional information on Oracle and VMWare I ran across a news story I missed last week. VMWare and Oracle have signed an agreement to make VMWare the standard virtualized Oracle environment.

VMware will become the virtualization development platform for Oracle server technologies, and Oracle plans to deploy 7,000 VMware virtual servers internally.

Prem Kumar, VP of server technologies engineering at Oracle, says the use of VMware virtualization products will enable Oracle to better optimize its products for use in multiple Windows and Linux operating-system environments. “We’ve gotten significant feedback from our customers requesting VMware support,” Kumar says. “We will be using VMware as our standard virtualized environment, including building product, testing, and installing, and it will be certified as a standard configuration.” [emphasis added]

or from another article

Under the agreement, Oracle will develop, test and support its 10g database and applications products to run on VMware’s operating environment virtualization products.

As part of the agreement, Oracle and VMware say they plan to work together to provide customers with problem resolution for Oracle’s 10g database and for 10g versions of its application server, Collaboration Suite and Enterprise Manager running on VMware.

Oracle has posted evaluation kits (VMWare machines with Linux/Oracle to evaluate the products) of which more than 12,000 copies have already been downloaded.
I’ve discussed VMWare before (here, and here) and have mixed thoughts about it. I’ve found it’s greatest strengths are in development, provisioning, and manageability. I’ve had some concerns about deploying Oracle on VMWare for production and high availability environments. This new partnership should only serve to address those concerns and ensure that the “entire stack” is fully supported. Great news for VMWare customers!