{"id":317,"date":"2009-01-27T17:35:49","date_gmt":"2009-01-28T00:35:49","guid":{"rendered":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/?p=317"},"modified":"2009-01-27T17:35:49","modified_gmt":"2009-01-28T00:35:49","slug":"select-stream-real_time_bi_metric-from-data_in_flight","status":"publish","type":"post","link":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/2009\/01\/27\/select-stream-real_time_bi_metric-from-data_in_flight\/","title":{"rendered":"select stream REAL_TIME_BI_METRIC from DATA_IN_FLIGHT"},"content":{"rendered":"

SQL is great. There are millions of people who know it, it’s semantics are standardized (well, as much as anything is ever I suppose), and it’s a great language for interacting with data. The folks at SQLstream<\/a> have taken a core piece of the technology world, and turbocharged it for a whole new application: querying streams of data. You can think of what SQLstream<\/a> is doing as the ‘calculus’ of SQL – not just query results of static data in databases but query results of a stream of records flowing by.<\/p>\n

Let’s consider some of the possibilities. Remember the results aren’t a one time poll – the values will continue to be updated as the query continues.<\/p>\n

\/\/ Select the stocks Average price over the last 10 ticks
\nselect stream stock, avg(price) over (previous 10) from stock_ticker group by stock
\n\/\/ Get the running averages for the past minute, hour, and day updated
\nselect stream stock, avg(price) over (range interval 1 minute preceding) as “AvgMinute” , avg(price) over (range interval 1 hour preceding) as “AvgHour”, avg(price) over (range interval 1 day preceding) as “AvgDay” from stock_ticker group by stock;<\/p><\/blockquote>\n

For a variety of reasons, I haven’t posted my demo of my real time BI analytics powered by SQLstream<\/a> but figured it’s about time to at least talk about it. The demo is illustrative of a the BASIC capabilities of the SQLstream RAMMS<\/a> (Relational Asynchronous Messaging Management System). Trust me, the simple stream queries I wrote are about as trivial as possible. The more complex the business needs the fancier the SQL becomes<\/a>, but that’s the point where the demo turns into a real customer project.<\/p>\n

You can’t see it in just the still image below, but the bars in the bar chart are “dancing” around as events are flowing from SQLstream<\/a>. It’s very very cool to watch a web dashboard update with no interaction. There’s a huge data set of stock prices being INSERTed into a stream via JDBC from one application and there’s a stream select that gives a windowed “% Change” of stock price. The application simply gets the “Stock \/ % Change” values over JDBC with all the complex windowing, grouping etc being done by SQLstream<\/a>.<\/p>\n

\"200901271717\"<\/a><\/p>\n

SQLstream have announced the 2.0 release of their produc<\/a>t. Some of you may be curious as to where the 1.x version of their product has been. To date, SQLstream<\/a> hasn’t made much fanfare about their products. They’ve been selling to customers, working with the Eigenbase<\/a> foundation, and have released several versions of their product over that past few years. I’ve had the chance to work with the 1.x series of their product and an early release candidate of their 2.x series and think it’s very very compelling. I very much look forward to digging into the new 2.x features soon.<\/p>\n

In particular, and what I continue to find most fascinating, is the new aggregation features in 2.x. This will allow users of ROLAP type query systems (or simply companies with summary\/aggregate tables they spend time updating) to keep these tables up to date throughout the day. This is a huge win for companies that have to update summary tables from huge transaction data sets.<\/p>\n

Consider a common problem batch loading window. You load a million records \/ day. You do period batch updates of your transactions throughout the day and at midnight you kick off the big huge queries to update the “Sales by Day”, “Sales by Month”, “Sales by Qtr”, “Sales by Year”, “Sales All Time” summary tables that take hours to rebuild<\/strong> (going against huge fact tables). Missing the batch window means reports\/business is delayed. <\/strong> SQLstream<\/a> can ease this burden. SQLstream<\/a> is efficiently calculating these summaries throughout the day (period batch loads) and at midnight simply “spits out” the summary table data. That’s a huge win for customers with a lot of data.<\/p>\n

If you’re loading millions of records per day and you need to keep summaries up to date or have time sensitive BI applications you should definitely consider SQLstream<\/a>. If you’re interested in discussing real time BI or the benefits of streaming aggregation don’t hesitate to contact Bayon<\/a> ; we can help you prototype, evaluate and build a SQLstream<\/a> solution<\/p>\n","protected":false},"excerpt":{"rendered":"

SQL is great. There are millions of people who know it, it’s semantics are standardized (well, as much as anything is ever I suppose), and it’s a great language for interacting with data. The folks at SQLstream have taken a core piece of the technology world, and turbocharged it for a whole new application: querying […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[6],"tags":[],"_links":{"self":[{"href":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/wp-json\/wp\/v2\/posts\/317"}],"collection":[{"href":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/wp-json\/wp\/v2\/comments?post=317"}],"version-history":[{"count":0,"href":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/wp-json\/wp\/v2\/posts\/317\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/wp-json\/wp\/v2\/media?parent=317"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/wp-json\/wp\/v2\/categories?post=317"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.nicholasgoodman.com\/bt\/blog\/wp-json\/wp\/v2\/tags?post=317"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}