LucidDB: DynamoBI is running with it

I can think of no better analogy than that of a multi leg race. You know, the races where one sprinter runs as fast as they can, before passing the baton to the next sprinter.

200910240932

First it was Broadbase.
Second it was LucidEra.
Third it was Eigenbase / LucidEra / SQLstream (joint development w/ Eigenbase).

Having purchased commercial rights from LucidEra it’s ours to run with now, alongside Eigenbase and SQLstream.

LucidDB has been described as the “best database no one ever told you about.” That stops today (the telling part, not the best part). Dynamo Business Intelligence Corp will take this great technology to a wider audience and we’ll be telling EVERYONE about it!

Over time, the exceptional features of this open source project will come to light (column store, bit map idxs, drop in java based user plugins, transparent remote JDBC data access, etc). I think it is important to acknowledge how LucidDB arrived to where it is today.

LucidDB is built by smart smart people (people wayyyy smarter than me!). People who’ve written parallel execution engines in Oracle. People who’ve developed Bitmap IDX implementations and helped file those patents. The heritage of LucidDB starts at Broadbase; LucidEra purchased it and brought it to Eigenbase. Eigenbase, and it’s sponsoring companies, have most claim to its current state. Their stewardship and ongoing evolution of the project is a testament to their talents and commitment to open source development. When you pick up LucidDB/DynamoDB and get your first “Ahhhh Cool! 10x Faster than my current database” you have LucidEra/SQLstream/Eigenbase devs to thank. John V. Sichi (lead and main project sponsor), Tai Tran, Julian Hyde, Rushan Chen, Zelaine Fong, Sunny Choi, Steve, Marc, Richard, Hunter, Edan, Damian, Boris, Benny, Stephan, Oscar, …. and the list goes on and on and on. Some of these people will be helping (in small and big ways) with the new company which is great for customers knowing that the people that wrote this stuff will be helping them be successful!

What’s the plan?

  • Open Source.
    Lots of it. Any readers of this blog, or who know me in general, will know I’m a “burn the boats,” open source kind of guy. We’ll be creating some new projects to make using the features/functions already in LucidDB easier. We’ll also be adding new features, which will make their way back into the LucidDB mainline.
  • Commercial in Name Only.
    Mainline DBMS enhancements and development continue, and will continue to be, in LucidDB (Eigenbase). New projects will be available under an OSI approved license. DynamoDB is the prepackaged, assembled, UI included distribution built for customers/evaluators that we’ll offer support on. Should be as easy as we can possibly make it to evaluate, purchase, and use.
  • In Progress.
    We’ve let the announcement ahead of having our website built, or having completed our own DynamoDB QA’ed build. Our open source roots guide us to an “early and often” approach and we’re taking that approach here. Be patient with us as we roll out the business bit by bit over the next few months. Our #1 priority: establish our support/build/qa infrastructure and get an already great piece of software into hands of people who can benefit from it. Hint: If you’ve ever done a star schema on MySQL you need to talk to us!

One thing I am personally looking forward to is getting to work even more extensively with everyone involved at Eigenbase, including the very talented devs at SQLstream (who produce the best real time analytics/integration engine available).

Feel free to join up in taking LucidDB to a whole new level: Download LucidDB and give it a go yourself, since we just released a new version (0.9.2) yesterday! I believe, like others have already mentioned, adding a bit of commercial support behind an already great piece of software is a winning combination!

Drop a line on through to me if you’re interested in getting involved early on (as a charter customer, developer, user, etc). ngoodman at bayontechnologies (with the .COM).

11 thoughts on “LucidDB: DynamoBI is running with it

  1. Mark Callaghan

    How to I get data from my OLTP engine into it? If a stream of SQL statements are available that represent changed rows on the OLTP engine, can I trickle changes to LucidDB so that data is stale for no more than X minutes? And what are reasonable values for X in that case.

    It is much easier to replace or add a new data warehouse engine than replace an OLTP engine, so I am interested in integrating LucidDB for DW queries with MySQL as the OLTP engine.

    Reply
  2. Nicholas Goodman Post author

    Mark,

    Good question, and I agree 100% with you. It’s easier to add / use a new DW engine instead of changing out your OLTP. In fact, you’ll never hear DynamoBI or LucidDB communities come close to suggesting that LucidDB is suitable for widespread OLTP use. Our heritage is BI/OLAP and we embrace it and think a MySQL/Oracle/SQLServer OLTP with a LucidDB backend for reporting is a good solution.

    Now, the ongoing application of changes from OLTP system. Since we aren’t an OLTP system, simply replaying the set of SQL statements from the OLTP db won’t be suitable. LucidDB typically needs to operate on a few hundred rows at a time.

    Currently we suggest people use last_updated_date or sequences to make a pull based update on LucidDB side more efficient by pulling only the records that have changed. Also, we have a PDI streaming loader under development so you can use the CDC methods available in PDI (which admittedly, compare two entire datasets) and only do DML to changed rows. Neither is as complete a solution as you suggest above.

    Something like the MySQL row format bin logs would be promising. A combination of reading these logs and applying them via MERGE and DELETE statements (in batches of several hundred to several thousands of records).

    I’m not all that familiar with format of MySQL row bin logs (records, breakdowns, rolling change numbers, etc). We can explore a general approach on our side (INS/UPD/DEL on ROW w/ VALUES) but would you be willing to act as sherpa through MySQL specifics? I wonder if a MySQLproxy wouldn’t have something to offer here as well.

    A general purpose MySQL to LucidDB replication would be very interesting to a lot of people I think!

    Reply
  3. Roland Bouman

    Hi Nick, Mark 🙂

    Nick: congrats to the purchase, it’s very exciting to see a new future for LucidDB. I am sure it is in good hands with you.

    Mark: I was thinking that it may be worth to try and build a plugin for the Tungsten replicator to load the changes into LucidDB. AFAIK, Tungstten itself and the MySQL log readers are free software, so you’d ‘just’ need to ‘only’ build the plugin that can apply the logevents to LucidDB.

    hth,

    Roland

    Reply
  4. Nicholas Goodman Post author

    Roland,

    The Tungsten approach is interesting. I wonder if there’s a way to incorporate the new REMOTE_ROWS Applib() fcn we’re building for the PDI plugin FarragoStreamingLoader.

    We have an engineer, Ray, working on the REMOTE_ROWS and PDI plugin right now. I’ll ask him to take a look at the Tungsten req’mnt for change applications in row format and make some comments on it.

    Reply
  5. Richard Daley

    Nick – congrats on this new endeavor. I’m looking forward to seeing you make this into a huge success in the open source community and for DynamoBI (love the name). I also know of some rolap technology that you might want to check out…and maybe a front end or two…

    Reply
  6. Pingback: 451 CAOS Theory » 451 CAOS Links 2009.10.27

  7. Pingback: Because 20+ data warehousing vendors is never enough — Too much information

  8. Kirk Wylie

    How about you put in “First and a Half: Radik.” Talk to Sichi about the history, but much of the underlying storage engine was enhanced significantly by John when he was at Radik. Then again, we acquired it from Quadbase in the first place, so there you go.

    Congratulations though! I’m glad to see the code stay out of IP Purgatory!

    Reply
  9. John Sichi

    Yeah, Kirk is right…one day Julian and I need to come up with the full code history diagram starting from primeval days. For example, Fennel wasn’t always Fennel…before that it was Karma, and before that…Quaff! (The “Quadbase File Format”)

    JVS

    Reply
  10. Pingback: More From The Low End: DynamoDB is the New Lucid « Market Strategies for IT Suppliers

  11. Pingback: Calpont opens up: InfiniDB Open Source Analytical Database (based on MySQL) « JZ Talk Blogger

Leave a Reply to Richard Daley Cancel reply

Your email address will not be published. Required fields are marked *