Nicholas Goodman on Business Intelligence

For those that don’t know the reference between lilac suit and Pete-s, just google it.  :)  Doesn’t really matter though when compared with the great set of articles that my friend Pete-S is pumping out from the other side of the Atlantic.

Pete has in the trenches practical knowledge building BI and DW systems.  He’s both sharp, and practical (that’s rare you know!).  He’s running a series DW Wisdom and it’s some very useful content.

DW Wisdom

I like that Pete is comfortable enough with his own skills/abilities to question the "age old wisdoms" of DW.  Even if they are found to be true it’s good to see some real scientific "assumption breaking" to prove/disprove reality.

• Use as many small disks as possible – a 1TB disk would be a bad idea for a system that inherently reads large volumes of data, everything would go through a single IO point.

  • Keep all the OLTP tables separate from the DW systems; OLTP has lots of small, fast transactions, DW has slower, big reads. DW loves bitmap indexes, OLTP hates them.

  • Use high degrees of parallel processing

But are these truths still valid?

DI Wisdom (2) - more of the physical

Comparison of OLTP vs DW(OLAP-esque):  Great reference table:

DI Wisdom (3) - departmental business

When people found that their transactional systems were unsuited for BI reporting (perhaps because of the performance impact of running BI on a transactional system, or the transactional system did not hold all of the data required for reporting) they started to look towards dedicated data warehouses.

DI Wisdom (4)

Enterprise DW moves away from the tactical departmental “point solutions” and into something that fits with strategic aspirations of the enterprise. On the face of it having a single solution across the enterprise as distinct advantages:

• there is a single, consistent model of enterprise data

  • there is less duplication of data across the enterprise
  • it is possible to construct a security model such that the right people see all of the data that allows them to their jobs but not the information that is too sensitive for their job role
  • the origins of all of the business data can be traced back to source

In fact these aims are so laudable that they have been hijacked by other IT disciplines such as master data management, risk and compliance management, and business process reengineering.

DW Design (part 1)

I can’t agree with Pete more: A staging, 3NF warehouse, and then presentation layer (marts) I think is a very practical way to seperate concerns, and avoid tight coupling between source and reports.  A la Corporate Information Factory.

For a long while I have favoured a three section data warehouse design: a staging area where raw fact and reference data is validated for referential integrity, a third-normal form layer to hold the reference data and historical fact, and finally a presentation layer to hold denormalised reference data and aggregated fact. The staging layer is ‘private’ to the data warehouse but user query access (subject to business security rules) to other layers is permitted. In some cases it will not be possible to use a denormalised layer; but if you can use one, you should.

DW Design (2) – staging data

As mentioned yesterday, the staging area of the data warehouse has three functional uses:

• It is the initial target for data loads from source systems

  • It validates the incoming data for integrity

  • It is the data source for information to be published the ‘user visible” layers of the data warehouse

Optionally, it may also be where the logic to transform incoming data is applied.

Great series Pete!  Now if only this were in a book that I could tell all my blog readers and colleagues to purchase! 

I recently moved to Wordpress as a blogging engine.  So far I’ve liked it very much, and the WYSIWYG editor in the web browser is pretty good.  Much better, in my opinion than the web based  Movable Type editor.

That being said, I reall do like being able to blog quickly and efficiently from my desktop.  Screen captures, text, copy and paste URLs, all make blogging feel much more like a treat; a thing you get into the rhythym of instead of taking time out of the day to get things "ready to go."

Now, if I ever decide to go Linux flat out I’ll have to do this all over again I suppose.  Until then, I think I rather like it.

One thing I think is missing from nearly all of these desktop blogging products is the ability to do real copy and paste.  Everything has to already be in a file.  For quick screen captures to blog I’d LOVE for a tool to just accept clipboard data and make it into a file.  Anyone know of any product that does that?

I’ve made the switch from Movable Type to Wordpress.  I’ve used Movable Type for more than two years now, quite happily actually, but thought it high time to move to Wordpress.  I’m rather impressed with Wordpress thus far; very competitive product all in all.  Some things in the installation were even EASIER than MT.

I turned on comments/trackbacks on my MT blog approximately 7 weeks ago.  In that 7 weeks, I had received more than 2,000 spam comments, trackbacks, etc.  I plan to use a spam service on this blog to prevent such an abuse again.

If you are encountering any feed or viewing issues please do let me know.

Picked up on this one from Brad Felds blog (VC in beautiful Boulder, home to my alma mater).

This Wednesday (or Tuesday in Europe) will have a sequential read on the clock.

Working from home, and at client sites, I don’t get the chance to enjoy the “casual Friday atmosphere” at software companies.

Someone sent me their office winner of the Ugly Shirt Award for Friday, March 10th.

Reader comments are open… Ugly, Classy, or Who Cares?

I’m not sure how long it will last until some robot computer across the world gets wise to this and starts spamming the comments.

We’ll see!

Rocketboom is beamed to my Tivo regularly. Cool feature and I think it’s a glimpse into the future as well. No cable company, no networks, just content provider and an API to zip it to my Tivo.

Anyhow todays Rocketboom was both odd, and amusing. Usually it’s all business, but crazy day today.

UPDATE: Some people have emailed asking if everything is alright. Overall the changes are very positive but thanks for all those concerned!
UPDATE 2: My 95% figure was apparently, grossly incorrect. Assuming you are an IT making more than 35000 GBP, or 62000 USD, or 52000 EUR annually you are in the richest 99% of the world. Once again… we are so very fortunate, collectively. Happy to update and correct it here!
My life is changing… some professional aspects and personal aspects. Emotions are a part of life and I’ve been facing rational and irrational emotions and thoughts about change. I’m excited and anxious all at once…

And you want to know what? Gratitude and genorosity are the most POWERFUL soothing emotions.

Grateful that you have a roof over your head, food in your cabinet, health, loved ones, and the opportunity to chase dreams professionally. The human condition longs for wanting more, but being thankful for what you have is a powerful idea. Humbling and potent!

Genorosity, or giving with no thought of return, speaks to us all as human beings. Giving some time to strangers in need, random acts of kindness, etc.

So… if you’re reading this blog you are likely an IT professional interested in Oracle, Data Warehousing, Business Intelligence or Open Source; you are better off financially than 95(99)% of the people on planet earth. Consider taking 30 seconds that you spent reading this blog and explore the things you are grateful for. I bet you’ll feel better with that knowledge than any OWB trick or Open Source BI insight you could have grokked in the same 30 seconds.

And now… we return you to regularly scheduled programming.

I received an email for an online webinar on ESB (Enterprise Service Bus) products. The email claims to be a “tough as nails” get to the details type of webinar (usually good in my mind). They even listed the five tough questions they’re gonna grill the vendors with:

The Five Tough ESB Questions Will Get Answered!

1. How do ESB architectures work with existing infrastructures
2. What are ESB’s most popular features
3. Can you show us some real-world user cases
4. Please tell us about ESB Best Practices
5. How are ESBs helping define SOA roadmaps for many firms

What a bunch of softballs! Tough Questions yeah right!

My favorite is number 2. That’s right, we’re going to grill you! What is your biggest selling point and why do users just LOVE your product. Puhhhlease.

I’m not sure I have to convince anyone reading this blog of this, especially if they live in the US. What, you ask, prompted now as a time to blog about it?

Had a look at the details of my most recent Cingular bill and was amazed at the amount of things considered taxes, surcharges, etc. It’s a stretch to regard regulatory fees for operating your business as a special tax. What blows my mind is that Cingular is passing through their Washington State “B and O” tax that are assessed to the BUSINESS and making it sound like some regulatory surcharge. The “B and O” surcharge. The Washington State “B and O” charge is the “Business and Occupation” tax; Washington state has NO income tax (business or personal) and the B and O is really just a unique “income” tax for business (based on gross receipts). In other words, in Washington, B and O is rougly equivalent in practice to corporate income tax.

So, here is a company passing through the cost of operating their business (renting airwaves) AND their taxes (B and O) and we have no choice but to accept this deceptive pricing scheme because of the oligopoly of mobile carriers.

Consider this: Would Amazon.com be in business today if when ordering, customers were to have the following “taxes” added to their orders?

USD 01.25 Sales Tax (1.5%) (yeah… ok, I get this)
USD 02.25 Amazon Utility Manddatory Surcharges (water, electric, and coffee delivery service to headquarters)
USD 01.72 Regulatory and Compliance Surcharge (we don’t want to pay taxes so we’ll pass our tax bill on to you so our prices look lower)

What is the net net? Companies that have to actually compete can’t get away with crappy, deceptive pricing and billing. Companies that don’t, do not (usually).