Eye Tracking, ML and Web Analytics – Correlated Concepts? Absolutely … Not just a Laura-ism, but Confirmed by a Professor at Carnegie

Eye Tracking Studies Required Expensive Hardware in the Past

 

 

  Anyone who has read my blog (shameless self-plug: http://www.lauraedell.com) over the years will know, I am very passionate about drinking my own analytical cool-aid. Whether during my stints as a Programmer, BI Developer, BI Manager, Practice Lead / Consultant or Senior Data Scientist, I believe whole-heartedly in measuring my own success with advanced analytics.  Even my fantasy football success (more on that in a later post) …But you wouldn’t believe how often this type of measurement gets ignored. ​

Introduce Eye-Tracking Studies-Daunting little set of machines in that image above, I know…But this system has been a cornerstone in the measurement practices of advertisement efficacy for eons, and something I latched onto into my early 20’s, in fact, ad-nauseam. I was lucky enough to work for the now uber online travel company who shall go nameless (okay, here is a hint: remember a little ditty that ended with some hillbilly singing “dot commmm” & you will know to whom I refer). This company believed so wholeheartedly in the user experience that they allowed me, young ingénue of the workplace, to spend thousands on eye tracking studies against a series of balanced scorecards that I was developing for the senior leadership team. This is important because you can ASK someone whether a designed visualization is WHAT THEY WERE THINKING or WANTING, even if done iteratively with the intended target, yet 9x out of 10, they will nod ‘yes’ instead of being honest, employing conflict avoidance at its best. Note, this applies to most, but I can think of a few in my new role at MSFT who are probably reading this and shaking their head in disagreement at this very moment <Got ya, you know who you are, ya ol’ negative Nelly’s; but I digress…AND… now we’re back –>

Eye tracking studies are used to measure efficacy by tracking what content areas engage users’ brains vs. areas that fall flat, are lackluster, overdesigned &/or contribute to eye/brain fatigue. It measures this by “tracking” where & for how long your eyes dwell on a quadrant (aka visual / website content / widget on a dashboard) and by recording the path & movement of the eyes between different quadrants’ on a page. It’s amazing to watch these advanced, algorithmic-tuned systems, measure a digital, informational message, in real-time, as it’s relayed to the intended audience, all while generating the statistics necessary to either know you “done a good job, son” or go back to the drawing board if you want to achieve the ‘Atta boy’. “BRILLIANT, I say.”

What I also learned which seems a no-brainer now, but people tend to read from Left to Right & from top to bottom. <duh> So, when I see anything that doesn’t at LEAST follow those two simple principles, I just shake my head and tisk tisk tisk, wondering how these improperly designed <insert content here> will ever relay any sort of meaningful message, destined for the “oh that’s interesting to view once” sphere instead of raising to the levels of usefulness it was designed for. Come on now, how hard is it to remember to stick the most important info in that top left quadrant and the least important in the bottom right, especially when creating visualizations for use in the corporate workplace by senior execs. They have even less time & attention these days to focus on even the most relevant KPIs, those they need to monitor to run their business & will get asked to update the CEO on each QTR, with all those fun distractions that come with the latest vernacular du-jour taking up all their brain space: “give me MACHINE LEARNING or give me death; the upstart that replaced mobile/cloud/big data/business intelligence (you fill in the blank).

But for so long, it was me against the hard reality that no one knew what I was blabbing on about, nor would they give me carte blanche to re-run those studies ever again <silently humming, “Cry me a River”>, And lo and behold, my Laura-ism soapbox has now been vetted, in fact, quantified by a prestigious University professor from Carnegie, all possible because a little know hero named

 

Edmond Huey, now near and dear to my heart, grandfather of the heatmap, followed up his color-friendly block chart by building the first device capable of tracking eye movements while people were reading. This breakthrough initiated a revolution for scientists but it was intrusive and readers had to wear special lenses with a tiny opening and a pointer attached to it like the 1st image pictured above.

heatmapFast-forward 100 years, and combine all ingredients into the cauldron of innovation and technological advancement, sprinkle in my favorite algorithmic pals:  CNN and LSTM, and the result is that grandchild now known as heat mapping. It’s eye tracking analytics without all the cost, basically a measure of the same phenomena (at a fraction of the cost).

Cool history lesson, right?

So, for those non-believers, I say, use some of the web analytic trends of the future (aka Web Analytics 3.0). Be a future-thinker, forward mover, innovator of your data science sphere of influence, and I tell you, you will become so much more informed and able to offer more information to others based on…MEASUREMENT (Intelligent MEASUREMENT in a digital transformational age).

 

Microsoft Data AMP 2017

Aside

Data AMP 2017 just finished and some really interesting announcements came out specific to our company-wide push into infusing machine learning, cognitive and deep learning APIs into every part of our organization. Some of the announcements are ML enablers while others are direct enhancements.

Here is a summary with links to further information:

  • SQL Server R Services in SQL Server 2017 is renamed to Machine Learning Services since both R and Python will be supported. More info
  • Three new features for Cognitive Services are now Generally Available (GA): Face API, Content Moderator, Computer Vision API. More info
  • Microsoft R Server 9.1 released: Real time scoring and performance enhancements, Microsoft ML libraries for Linux, Hadoop/Spark and Teradata. More info
  • Azure Analysis Services is now Generally Available (GA). More info
  • **Microsoft has incorporated the technology that sits behind the Cognitive Services inside U-SQL directly as functions. U-SQL is part of Azure Data Lake Analytics(ADLA)
  • More Cortana Intelligence solution templates: Demand forecasting, Personalized offers, Quality assurance. More info
  • A new database migration service will help you migrate existing on-premises SQL Server, Oracle, and MySQL databases to Azure SQL Database or SQL Server on Azure virtual machines. Sign up for limited preview
  • A new Azure SQL Database offering, currently being called Azure SQL Managed Instance (final name to be determined):
    • Migrate SQL Server to SQL as a Service with no changes
    • Support SQL Agent, 3-part names, DBMail, CDC, Service Broker
    • **Cross-database + cross-instance querying
    • **Extensibility: CLR + R Services
    • SQL profiler, additional DMVs support, Xevents
    • Native back-up restore, log shipping, transaction replication
    • More info
    • Sign up for limited preview
  • SQL Server vNext CTP 2.0 is now available and the product will be officially called SQL Server 2017:

Those I am most excited about I added ** next to. This includes key innovations with our approach to AI and enhancing our deep learning compete against Google TensorFlor for example. Check out the following blog posting: https://blogs.technet.microsoft.com/dataplatforminsider/2017/04/19/delivering-ai-with-data-the-next-generation-of-microsofts-data-platform/ :

  1. The first is the close integration of AI functions into databases, data lakes, and the cloud to simplify the deployment of intelligent applications.
  2. The second is the use of AI within our services to enhance performance and data security.
  3. The third is flexibility—the flexibility for developers to compose multiple cloud services into various design patterns for AI, and the flexibility to leverage Windows, Linux, Python, R, Spark, Hadoop, and other open source tools in building such systems.

 

Azure ML + AI (Cognitive Services Deep Learning)Most recent documents

23 items in the toolbar. Use Left or Right to navigate. Use Enter to add the selected web part.

Action Bulleted list performed.

Wonderful World of Sports: Hey NFL, Got RFID?

Aside

As requested by some of my LinkedIn followers, here is the NFL Infographic about RFID tags I shared a while back:

nfl_tech_infographic-100612792-large.idge

I hope @NFL @XboxOne #rfid data becomes more easily accessible. I have been tweeting about the Zebra deal for 6 months now, and the awesome implications this would have on everything from sports betting to fantasy enthusiasts to coaching, drafting and what have you. Similarly, I have built a fantasy football (PPR) league bench/play #MachineLearning model using #PySpark which, as it turns out, is pretty good. But it could be great with the RFID stream.
nfl-tagged-shoulder-pads-100612790-large.idge

This is where the #IoT rubber really hits the road because there are so many more fans of the NFL than there are folks who really grok the “Connected Home” (not knocking it, but it doesn’t have the reach tentacles of the NFL). Imagine measuring the burn-rate output vs. performance degradation of these athletes mid game and one day, being able to stream that on the field or booth for game course corrections. Aah, a girl can only dream…

Is Machine Learning the New EPM Black?

Aside

I am currently a data scientist & am also a certified lean six sigma black belt. I specialize in the Big Data Finance, EPM, BI & process improvement fields where this convergence of skills has provided me the ability to understand the interactions between people, process and technology/ tools.

I would like to address the need to transform traditional EPM processes by leveraging more machine learning to help reduce forecast error and eliminate unnecessary budgeting and planning rework and cycle time using a  3 step ML approach:

1st, determine which business drivers are statistically meaningful to the forecast (correlation) , eliminating those that are not.

2nd, cluster those correlated drivers by significance to determine those that cause the most variability to the forecast (causation).

3rd, use the output of 1 and 2 as inputs to the forecast, and apply ML in order to generate a statistically accurate forward looking forecast.

 ml

Objection handling, in my experience, focuses on the cost,  time and the sensitive change management aspect- how I have handled these, for example, is as such :

  1. Cost: all of these models can be built using free tools like R and Python data science libraries, so there is minimal to no technology/tool capEx/opEx investment.   
  2. Time: most college grads with either a business, science or computer engineering degree will have undoubtedly worked with R and/or Python (and more) while earning their degree. This reduces the ramp time to get folks acclimated and up to speed. To fill the remaining skill set gap, they can use the vast libraries of work already provided by the R / Python initiatives or the many other data science communities available online for free as a starting point, which also minimizes the time due to unnecessary cycles and rework trying to define drivers based on gut feel only. 
  3. Change: this is the bigger objection that has to be handled according to the business culture and openness to change. Best means of handling this is to simply show them. Proof is in the proverbial pudding so creating a variance analysis of the ML forecast, the human forecast and the actuals will speak volumes, and bonus points if the correlation and clustering analysis also surfaced previously unknown nuggets of information richness.

Even without the finding the golden nugget ticket, the CFO will certainly take notice of a more accurate forecast and appreciate the time and frustration savings from a less consuming budget and planning cycle.

Utilizing #PredictiveAnalytics & #BigData To Improve Accuracy of #EPM Forecasting Process

Aside

I was amazed when I read the @TidemarkEPM awesome new white paper on the “4 Steps to a Big Data Finance Strategy.” This is an area I am very passionate about; some might say, it’s become my soap-box since my days as a Business Intelligence consultant. I saw the dawn of a world where EPM, specifically, the planning and budgeting process was elevated from gut feel analytics to embracing #machinelearning as a means of understanding which drivers are statistically significant from those that have no verifiable impact , and ultimately using those to feed a more accurate forecast model.

Big Data Finance

Traditionally (even still today), finance teams sit in a conference room with Excel spreadsheets from Marketing, Customer Service etc., and basically, define the current or future plans based on past performance mixed with a sprinkle of gut feel (sometimes, it was more like a gallon of gut feel to every tablespoon of historical data). In these same meetings just one quarter later, I would shake my head when the same people questioned why they missed their targets or achieved a variance that was greater/less than the anticipated or expected value.

The new world order of Big Data Finance leverages the power of machine learned algorithms to derive true forecasted analytics. And this was a primary driver for my switching from a pure BI focus into data science. And, I have seen so many companies embrace the power of true “advanced predictive analytics” and by doing so, harness the value and benefits of doing so; and doing so, with confidence, instead of fear of this unknown statistical realm, not to mention all of the unsettled glances when you say the nebulous “#BigData” or “#predictiveAnalytics” phrases.

But I wondered, exactlyBig Data Finance, Data Types, Process Use Cases, Forecasting, Budgeting, Planning, EPM, Predictive, Model how many companies are doing this vs. the old way? And I was very surprised to learn from the white-paper that  22.7% of people view predictive capabilities as “essential” to forecasting, with 52.2% claiming it nice to have.  Surprised is an understatement; in fact, I was floored.

We aren’t just talking about including weather data when predicting consumer buying behaviors. What about the major challenge for the telecommunications / network provider with customer churn? Wouldn’t it be nice to answer the question: Who are the most profitable customers WHO have the highest likelihood of churn? And wouldn’t it be nice to not have to assign 1 to several analysts xx number of days or weeks to be able to crunch through all of the relevant data to try to get to an answer to that question? And still probably not have all of the most important internal indicators or be including indicators that have no value or significance to driving an accurate churn outcome?

What about adding in 3rd party external benchmarking data to further classify and correlate these customer indicators before you run your churn prediction model? To manually do this is daunting and so many companies, I now hypothesize, revert to the old ways of doing the forecast. Plus, I bet they have a daunting impression of the cost of big data and the time to implement because of past experiences with things like building the uber “data warehouse” to get to that panacea of the “1 single source of truth”…On the island of Dr. Disparate Data that we all dreamt of in our past lives, right?

I mean we have all heard that before and yet, how many times was it actually done successfully, within budget or in the allocated time frame? And if it was, what kind of quantifiable return on investment did you really get before annual maintenance bills flowed in? Be honest…No one is judging you; well, that is, if you learned from your mistakes or can admit that your pet project perhaps bit off too much and failed.

And what about training your people or the company to utilize said investment as part of your implementation plan? What was your budget for this training and was it successful,  or did you have to hire outside folks like consultants to do the work for you? And by doing so, how long did it actually take the break the dependency on those external resources and still be successful?

Before the days of Apache Spark and other Open Source in-memory or streaming technologies, the world of Big Data was just blossoming into what it was going to grow into as a more mature flower. On top of which, it takes a while for someone to fully grok a new technology, even with the most specialized training, especially if they aren’t organically a programmer, like many Business Intelligence implementation specialists were/are. That is because those who have past experience with something like C++, can quickly apply the same techniques to newer technologies like Scala for Apache Spark or Python and be up and running much faster vs. someone who has no background in programming trying to learn what a loop is or how to call an API to get 3rd party benchmarking data. We programmers take that for granted when applying ourselves to learning something new.

And now that these tools are more enterprise ready and friendly with new integration modules with tools like R or MATLib for the statistical analysis coupled with all of the free training offered by places like University of Berkeley (via eDX online), now is the time to adopt Big Data Finance more than ever.

In a world where the machine learning algorithm can be paired with traditional classification modeling techniques automatically, and said algorithms have been made publicly available for your analysts to use as a starting point or in their entirety for your organization, one no longer needs to be daunted by thought of implementing Big Data Finance or testing out the waters of accuracy to see if you are comfortable with the margin of error between your former forecasting methodology and this new world order.

2015 Gartner Magic Quadrant Business Intelligence – Mind Melding BI & Data Science, a Continuing Trend…

2015 Magic Quadrant Business intelligence

2015 Magic Quadrant Business intelligence

IT WAS the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us…

–Charles Dickens

Truer words were never spoken, whether about the current technological times or deep in our past (remember the good ole enterprise report books, aka the 120 page paper weight?)

And, this data gal couldn’t be happier with the final predictions made by Gartner in their 2015 Magic Quadrant Report for Business Intelligence. Two major trends / differentiators fall right into the sweet spot I adore:

New demands for advanced analytics 

Focus on predictive/prescriptive capabilities 

Whether you think this spells out doom for business intelligence as it exists today or not, you cannot deny that these trends in data science and big data can only force us to finally work smarter, and not harder (is that even possible??)

What are your thoughts…?

KPIs in Retail and Store Analytics

I like this post. While I added some KPIs to their list, I think it is a good list to get retailers on the right path…

KPIs in Retail and Store Analytics (continuation of a post made by Abhinav on kpisrus.wordpress.com:
A) If it is a classic brick and mortar retailer:

Retail / Merchandising KPIs:

-Average Time on Shelf

-Item Staleness

-Shrinkage % (includes things like spoilage, shoplifting/theft and damaged merchandise)

Marketing KPIs:

-Coupon Breakage and Efficacy (which coupons drive desired purchase behavior vs. detract)

-Net Promoter Score (“How likely are you to recommend xx company to a friend or family member” – this is typically measured during customer satisfaction surveys and depending on your organization, it may fall under Customer Ops or Marketing departments in terms of responsibility).

-Number of trips (in person) vs. e-commerce site visits per month (tells you if your website is more effective than your physical store at generating shopping interest)

B) If it is an e-retailer :

Marketing KPIs:

-Shopping Cart Abandonment %

-Page with the Highest Abandonment

-Dwell time per page (indicates interest)

-Clickstream path for purchasers (like Jamie mentioned do they arrive via email, promotion, flash sales source like Groupon), and if so, what are the clickstream paths that they take. This should look like an upside down funnel, where you have the visitors / unique users at the top who enter your site, and then the various paths (pages) they view in route to a purchase).

-Clickstream path for visitors (take Expedia for example…Many people use them as a travel search engine but then jump off the site to buy directly from the travel vendor – understanding this behavior can help you monetize the value of the content you provide as an alternate source of revenue).

-Visit to Buy %

-If direct email marketing is part of your strategy, analyzing click rate is a close second to measuring conversion rate. 2 different KPIs, one the king , the other the queen and both necessary to understand how effective your email campaign was and whether it warranted the associated campaign cost.

Site Operations KPIs / Marketing KPIs:

-Error % Overall

-Error % by Page (this is highly correlated to the Pages that have the Highest Abandonment, which means you can fix something like the reason for the error, and have a direct path to measure the success of the change).

Financial KPIs:

-Average order size per transaction

-Average sales per transaction

-Average number of items per transaction

-Average profit per transaction

-Return on capital invested

-Margin %

-Markup %

I hope this helps. Let me know if you have any questions.

You can reach me at mailto://lauraedell@me.com or you can visit my blog where I have many posts listing out various KPIs by industry and how to best aggregate them for reporting and executive presentation purposes ( http://www.lauraedell.com ).

It was very likely that I would write on KPIs in Retail or Store Analytics since my last post on Marketing and Customer Analytics. The main motive behind retailers looking into BI is ‘customer’ and how they can quickly react to changes in customer demand, rather predict customer demand, remove wasteful spending by target marketing, exceeding customer expectation and hence improve customer retention.

I did a quick research on what companies have been using as a measure of performance in retail industry and compiled a list of KPIs that I would recommend for consideration.

Customer Analytics

Customer being the key for this industry it is important to segment customers especially for strategic campaigns and to develop relationships for maximum customer retention. Understanding customer requirements and dealing with ever-changing market conditions is the key for a retail industry to survive the competition.

  • Average order size per transaction
  • Average sales per transaction

View original post 278 more words

Awesome Article “Views from the C-Suite: Who’s Big on Big Data” from The Economist

This is an awesome article discussing the whole “big data” thing from the C-level point of view. It is easy to get mired down in the technical weeds of big data, especially since it generates a ton of different definitions depending on who you ask and usually, where they work *department, wise*.

http://pages.platfora.com/rs/platfora/images/Economist-Intelligence-Unit-Big-Data-Exec-Summary.pdf

Let me know what you think.

Big shout out to @platfora for sharing this!

Finance is the Participation Sport of the BI Olympics

IT is no longer the powerhouse that it once was, and unfortunately for CIOs who haven’t embraced change, much of their realm was commoditized by cloud computing powered by the core principles of grid computational engines and schema-less database designs. The whole concept of spending millions of dollars to bring all disparate systems together into one data warehouse has proven modesty beneficial but if we are being truly honest, what has all that money and time actually yielded, especially towards the bottom line?
And by the time you finished with the EDW, I guarantee it was missing core operational data streams that were then designed into their own sea of data marts. Fast forward a few years, and you probably have some level of EDW, many more data marts , probably one or more cube (ROLAP/MOLAP) applications and n-number of cubes or a massive 1+ hyper-cube(s) and still, the business depends of spreadsheets to sit on top of these systems, creating individual silos of information under the desk or in the mind of one individual.

Wait<<<rewind<<< Isn’t that where we started?

Having disparate, ungoverned and untrusted data sources being managed by individuals instead of by enterprise systems of record?

And now we’re back>>>press play to continue>>>

When you stop to think about the last ten years, fellow BI practitioners, you might be scared of your ever-changing role. From a grass-roots effort to a formalized department team, Business Intelligence went from the shadows to the mainstream, and brought with it reports then dashboards, then KPIs and scorecards, managing by exception, proactive notifications and so on. And bam! We were hit by the first smattering of changes to come when Hadoop and others hit the presses. But we really didnt grok what the true potential and actual meaning of said systems unless you come from a background like myself, either competitively, or from a big data friendly industry group like telecommunications, or from a consultant/implementation p.o.v.
And then social networking took off like gang busters and mobile became a reality with the introduction of the tablet device (though, I hate to float my boat as always by mentioning my soap box dream spewed at a TDWI conference about the future of mobile BI when the 1st generation iPhone released).

But that is neither here nor there. And, as always, I digress and am back…

At the same time as we myopically focused on the technological changing landscape around us, a shifting power paradigm was building wherein the Finance organization, once relegated to the back partition of cubicles, where a pin drop was heard ’round the world (or at least, the floor), was growing more and more despondent with not being able to access the data they needed without IT intervention in order to update their monthly forecasts and produce their subsequent P&L, Balance Sheet and Cash Flow Planning statements. And IT’s response was to acquire (for additional millions of dollars) a “BI tool” aka an ad-hoc reporting application that would allow them to pull their own data. But it had been installed and the data had been pulled, and validated and by the time of completion, the Finance team had either found an alternate solution or found the system useful for a very small sliver of analysis but went outside of IT to get additional sources of information that wanted and needed to adapt to the changing business pressures from the convergence of social, mobile and unstructured datasets. And suddenly those once, shiny BI tools, seemed like antiquated relics, and simply could not handle the sheer data volumes that were now expected from it or would crash (unless filtered beyond the point of value). Businesses need not adapt their queries to the tool but need a tool that can adapt to their ever-changing processes and needed.

Drowning in data but starving for information...

Drowning in data but starving for information…

So if necessity if the mother of invention, Finance was its well deserving child. And why? The business across the board is starving for information but drowning in data. And Finance is no longer a game of solitaire, understood by few and ignored by many. In fact, Finance has become the participation sport of the BI Olympics, and rightfully so, where departmental collaboration at the fringe of the organization has proven as the missing link that before prevented successful top-down planning efforts. Where visualizations demands made dashboards a thing of the past, and demanded and better story, vis-a-vie storylines / infographics, to help disseminate more than just the numbers, but the story behind the numbers to the rest of the organization, or what I like to call the “fringe”.

I remember a few years ago when the biggest challenge was getting data, and often, we joked about how nice it would be to have a sea of data to drown in; an analysts’ buffet-du-jour; a happy paralysis-induced-by said analysis plate was the special of the day, yet only for a few, while the rest was but a gleam in our data-starved eyes.

Looking forward from there, I ask, dear reader, where do we go from here…If it’s a Finance party and we are all invited, what do we bring to the party table as BI practitioners of value? Can we provide the next critical differentiator?

Well, I believe that we can, and that critical differentiator is forward-looking data. Why?

Gartner Group stated that “Predictive data will increase profitability by 20% and that historical data will become a thing of the past” (for a BI practitioner, the last part of that statement should worry you, if you are still resisting the plunge into the predictive analytics pool).

Remember, predictive is a process that allows an organization to get true insight and has been executed amongst a larger group of people to drive faster, smarter business users. This is perfect for enterprise needs because by definition, they offer a larger group of people to work with.

Smooth sailingIn fact, it was Jack Welch would said  An organization’s ability to learn, and translate that learning into action rapidly, is the ultimate competitive advantage” 

If you haven’t already, go out and started learning one of the statistical application packages. I suggest “R” and in the coming weeks, I will provide R and SAS scripts (I have experience with both) for those interested in growing their chosen profession and remaining relevant as we weather the sea of business changes

.

Futures According to Laura… Convergence of Cloud and Neural Networking with Mobility and Big Data

It’s been longer and longer between my posts and as always, life can be inferred as the reason for my delay.

But I was also struggling with feeling a sense of “what now” as it relates to Business Intelligence.

Many years ago, when I first started blogging, I would write about where I thought BI needed to move in order to remain relevant in the future. And those futures have come to fruition lately. Gamuts ranging from merging social networking datasets into traditional BI frameworks to a more common use case of applying composite visualizations to data (microcharts, as an example). Perhaps more esoteric was my staunch stance on the Mobile BI marriage which when iPhone 1 was released was a future many disputed with me. In fact, most did not own the first release of the iPhone, and many were still RIM subscribers. And it was hard for the Blackberry crowd to fathom a world unbounded by keyboards and scroll wheels and how that would be a game changer for mobile BI. And of course, once the iPad was introduced, it was a game over moment. Execs everywhere wanted their iPads to have the latest and greatest dashboards/KPIs/apps. From Angry Birds to their Daily Sales trend, CEOs and the like had new brain candy to distract them during those drawn out meetings. And instead of wanting that PDF or PowerPoint update, they wanted to receive the same data on their iPad. Once they did, they realized that having the “WHAT” is happening understanding was only the crack to get them hooked for a while. Unfortunately, the efficacy of KPI colors and related numbers only satisfies the one person show – but as we know, it isn’t the CEO who analyzes why a RED KPI indicator shows up. Thus, more levels of information (beyond the “WHAT” and  “HOW OFTEN”)  were needed to answer the “WHY” and “HOW TO FIX” the underlying / root cause issue.

The mobile app was born.

It is the reborn mobile dashboard that has been transformed into a new mobile workflow, more akin to the mobile app. 

But it took time for people to understand the marriage between BI dashboards, the mobile wave, especially the game change that Apple introduced with it’s swipe and pinch to zoom gestures, the revolution of the App Stores for the “need to have access to it now” generation of Execs, the capability to write-back from mobile devices to any number of source systems and how functionally, each of these seemingly unrelated functions would and could be weaved together to create the next generation of Mobile Apps for Business Intelligence. 

But that’s not what I wanted to write about today. It was a dream of the past that has come to fruition. 

Coming into 2013, cloud went from being something that very few understood to another game changer in terms of how CIOs are thinking about application support of the future. And that future is now.

But there are still limitations that we are bound by. Either we have a mobile device or not, either it is on 3 or 4G or wifi. Add to that our laptops (yes, something I believe will not dominate the business world in a future someday). And compound that with other devices like smartphones, eReaders, desktop computers et al. 

So, I started thinking about some of the latest research regarding Neural Networks (another set of posts I have made about the future of communication via Neural networks) published recently by Cornell University here (link points to http://arxiv.org/abs/1301.3605).

And my nature “plinko” thought process (before you ask, search for the Price is Right game and you will understand “Plinko Thoughts”) bounced from Neural Networks to Cloud Networks and from Cloud Networks to the idea of a Personal Cloud. 

A cloud of such personal nature that all of our unique devices are forever connected in our own personal sphere and all times when on our person. We walk around and we each have our own personal clouds. Instead of a mass world wide web, we have our own personal wide area network and our own personal wide web.

When we interact with other people, those people can choose to share their Personal networks with us via Neural Networking or some other sentient process, or in the example, where we bump into a friend and we want to share details with them, all of our devices have the capability to interlink to each other via our Personal Clouds. 

Devices are always connected to your Personal Cloud which is authenticated to your person, so that passwords which are already reaching their shelf life (see: article for more information on this point), are no longer the annoying constraint when we try to seamlessly use our mobile devices while on the go. Instead, they are authenticated to our Personal Cloud following similar principles as where IAM (Identity and Access Management) is moving towards in future. And changes in IAM are not only necessary for this idea to come to fruition but are on the horizon.

In fact, Gartner published an article in July 2012, called “Hype Cycle for Identity and Access Management Technologies, 2012” in which Gartner recognized that the growing adoption of mobile devices, cloud computing, social media and big data were converging to help drive significant changes in the identity and access management market.

For background purposes, IAM processes and technologies work across multiple systems to manage:

■ Multiple digital identities representing individual users, each comprising an identifier (name or key) and a set of data that represent attributes, preferences and traits

■ The relationship of those digital identities to each user’s civil identity

■ How digital user identities communicate or otherwise interact with those systems to handle
information or gain knowledge about the information contained in the systems

If you extrapolate that 3rd bullet out, and weave in what you might or might not know/understand about Neural Networking or brain-to-brain communication (see recent Duke findings by Dr. Miguel Nicolelis here) (BTW – the link points to http://www.nicolelislab.net/), one can start to fathom the world of our future. Add in cloud networking, big data, social data and mobility, and perhaps, the Personal Cloud concept I extol is not as far fetched as you initially thought when you read this post. Think about it.

My dream like with my other posts is to be able to refer back to this entry years from now with a sense of pride and “I told you so.” 

Come on – any blogger who makes predictions which come true years later deserves some bragging rites. 

Or at least, I think so…

MicroStrategy World 2012 – Miami

Our internal SKO (sales kick off) meeting was the beginning of this years’ MSTR World conference ( held in Miami, FL at the Intercontinental Hotel located on Chopin Plaza). As with every year, the kickoff meeting is the preliminary gathering of the salesforce in an effort to “rah-rah” the troops who work the front lines around the world ( myself included).

What I find most intriguing is the fact that MicroStrategy is materializing for BI all of those pipe dreams we ALL have. You know the ones I mean : I didn’t buy socialintelligence.co for my health several years ago. It was because I saw the vision of a future where business intelligence and social networking were married. Or take cloud intelligence, aka BI in the cloud. Looking back in 2008, I remember my soapbox discussion of BI mashups, ala My Google, supported in a drag and drop off premises environment. And everyone hollered that I was too visionary, or too far ahead. That everyone wanted reporting, and if I was lucky, maybe even dashboards.

But the acceleration continued, whether adoption grew or not.

Then, i pushed the envelope again: I wanted to take my previous thought of the mashup a morph it into an app integrated with BI tools. Write back to transactional systems or web services was key.

What is a dashboard without the ability within the same interface to take action? Everyone talks about actionable metrics/KPIs. Well, I will tell you that to have a KPI BY DEFINITION OF WHAT A KPI IS, means it is actionable.

But making your end users go to a separate ERP or CRM, to make the changes necessary to affect a KPI, will drive your users away. What benefit can you offer them in that instance ? Going to a dashboard or an excel sheet is no different. It is 1 application to view and if they are lucky, to analyze their data. If they were using excel before , they will still be using excel, especially if your dashboard isn’t useful to day to day operations.

Why? They still have to go to a 2nd application to take action.

Instead, integrate them into one.

Your dashboard will become meaningful and useful to the larger audience of users.
Pipe dream right?

NO. I have proved this out many times now and it works.

Back in 2007-2008, it was merely a theory I pontificated with you, my dear readers.

Since then, I have proved it out several times over and proven the success that can be achieved by taking that next step with your BI platforms.

Folks, if you haven’t done it, do it. Don’t waste anymore time. It took me less then 3 days to write the web services code to consume the salesforce APIs including chatter, ( business “twitter” according to SFDC), into my BI dashboard ( mobile dashboard in fact).

And suddenly, a sales dashboard becomes relevant. No longer does the salesforce team have to view their opportunities and quota achievement in one place, only to leave or open a new browser, to access their salesforce.com portal in order to update where they are at mid quarter.

But wait, now they forgot which KPIs they need to add comments to because they were red on the dashboard which is now closed, and their sales GM is screaming at them on the phone. Oh wait…they are on the road while this is happening and their data plan for their iPad has expired and no wireless connection is found.

What do you do?

Integrating salesforce.com into their dashboard eliminates at least one step (opening a new browser) in the process. Offering mobile offline transactions is a new feature of MicroStrategy’s mobile application. This allows those sales folks to make the comments they need to make while offline, on the road , which will be queued until they are online again.

One stop, one dashboard to access and take action through, even when offline, using their mobile ( android, iPad/iPhone or blackberry ) device.

This is why I’m excited to see MicroStrategy pushing the envelope on mobile BI futures.

MicroStrategy Personal Cloud – a Great **FREE** Cloud-based, Mobile Visualization Tool

Have you ever needed to create a prototype of a larger Business Intelligence project focused on data visualizations? Chances are, you have, fellow BI practitioners. Here’s the scenario for you day-dreamers out there:

Think of the hours spent creating wire-frames, no matter what tool you used, even if said tool was your hand and a napkin (ala ‘back of the napkin’ drawing) or the all-time-favorite white board, which later becomes a permanent drawing with huge bolded letters to the effect of ‘DO NOT ERASE OR ITS OFF WITH YOUR HEAD’ annotations dancing merrily around your work. Even better: electronic whiteboards which yield you hard copies of your hard work (so aptly named), which at first, seems like the panacea of all things cool (though it has been around for eons) but still, upon using, deemed the raddest piece of hardware your company has, until, of course, you look down at the thermal paper printout which has already faded in the millisecond since you tore it from machine to hand, which after said event, leaves the print out useless to the naked eye, unless you have super spidey sense optic nerves, but now I digress even further and in the time it took you to try to read thermal printout, it has degraded further because anything over 77 degrees is suboptimal (last I checked we checked in at around 98.6 but who’s counting), thus last stand on thermal paper electronic whiteboards is that they are most awesome when NOT thermoregulate ;).

OK, and now We are back…rewind to sentence 1 –

Prototyping is to dashboard design or any data visualization design as pencils and grid paper are to me. Mano y mano – I mean, totally symbiotic, right?

But, wireframing is torturous when you are in a consultative or pre-sales role, because you can’t present napkin designs to a client, or pictures of a whiteboard, unless you are showing them the process behind the design. (And by the way, this is an effective “presentation builder” when you are going for a dramatic effect –> ala “first there were cavemen, then the chisel and stone where all one had to create metrics –> then the whiteboard –> then the…wait!

This is where said BI practitioner needs to have something MORE for that dramatic pop, whiz-AM to give to their prospective clients/customers in their leave behind presentation.

And finally, the girl gets to her point (you are always so patient, my loving blog readers)…While I biased, if you forget whom I work for, and just take into account the tool, you will see the awesomeness that the new MicroStrategy Personal Cloud is for (drum roll please) PROTOTYPING a new dashboard — or just building, distributing, mobilizing etc your spreadsheet of data in a highly stylized, graphical means that tell a story far better than a spreadsheet can in most situations. (Yes, neighseyers, I know that for the 5% of circumstances which you can name, a spreadsheet is more àpropos, but HA HA, I say: this cloud personal product has the ability to include the data table along with the data visualizations!)

Best of all it is free.

I demoed this recently and was able to time it took to upload and spreadsheet, render 3 different data visualizations, generate the link to send to mobile devices (iPads and iPhones), network latency for said demo-ees to receive the email with the link and for them to launch the dashboard I created, and guess what the total time was?

Next best of all, it took only 23.7 minutes from concept to mobilization!

Mind you, I was also using data from the prospect that I had never seen or had any experience with.

OK, here is how it was done:

1) Create a FREE account or login to your existing MicroStrategy account (by existing, I mean, if you have ever signed up for the MicroStrategy forums or discussion boards, or you are an employee, then use the same login) at https://www.microstrategy.com/cloud/personal

Cloud Home

Landing Page After Logged in to Personal Cloud

2) Click the button to Create New Dashboard:

Create Dashboard Icon

  • Now, you either need to have a spreadsheet of data OR you can choose one of the sample spreadsheets that MicroStrategy provides (which is helpful if you want to see how others set up their data in Excel, or how others have used Cloud personal to create dashboards; even though it is sample data , it is actually REAL data that has been scrub-a-dub-dubbed for your pleasure!) If using a sample data set, I recommend the FAA data. It is real air traffic data, with carrier, airport code, days of the week, etc, which you can use to plan your travel by; I do…See screenshot below. There are some airports and some carriers who fly into said airports whom I WILL not fly given set days of the week in which I must travel. If there is a choice, I will choose to fly alternate carriers/routes. This FAA data set will enable you to analyze this information to make the most informed decision (outside of price) when planning your travel. Trust me…VERY HELPFUL! Plus, you can look at all the poor slobs without names sitting at the Alaska Air gate who DIDNT use this information to plan their travel, and as you casually saunter to your own gate on that Tuesday between 3 – 6 PM at SeaTac airport , you will remember that they look so sad because their Alaska Air flight has a 88% likelihood of being delayed or cancelled. (BTW, before you jump on me for my not so nice reference to said passengers), it is merely a quotation from my favorite movie ‘Breakfast at Tiffany’s’ …says Holly Golightly: “Poor cat…poor old slob without a name”.

On time Performance (Live FAA Data)

If using your own data, select the spreadsheet you want to upload

3) Preview your data; IMPORTANT STEP: make sure that you change any fields which to their correct type (either Attribute or Metric or Do Not Import).

Cloud Import - Preview Data

Keep in mind the 80/20 rule: 80% of the time, MicroStrategy will designate your data as either an Attribute or Metric correctly using a simple rule of thumb: Text or VarChar/NVarChar if using SQL Server, will always be designated as an Attribute (i.e. your descriptor/Dimension) and your numerals designated as your Metrics. BUT, if your spreadsheet uses ID fields, like Store ID, or Case ID, along with the descriptor like Store DESC or Case DESC, most likely MicroStrategy will assume the Store ID/Case ID are Metrics (since the fields are numeric in the source). This is an Easy Change! You just need to make sure ahead of time to make that change using the drop down indicator arrows in the column headings – To find them, hover over the column names with your mouse icon until you see the drop down indicator arrow. Click on the arrow to change an Attribute column to a Metric column and vice-versa (see screenshot):

Change Attribute to Metric

Once you finish with previewing your data, and everything looks good, click OK at the bottom Right of your screen.

In about 30-35 seconds, MicroStrategy will have imported your data into the Cloud for you to start building your awesome dashboards.

4) Choose a visualization from the menu that pops up on your screen upon successfully importing your spreadsheet:

Dashboard Visualization Selector
Change data visualization as little or as often as you choose

Here is the 2010 NFL data which I uploaded this morning. It is a heatmap showing the Home teams as well as any teams they played in the 2010 season. The size of the box is HOW big the win or loss was. The color indicates whether they won or lost (Green = Home team won // Red = Home team lost).

For all you, dear readers, I bid you a Happy New Year. May your ideas flow a plenty, and your data match your dreams (of what it should be) :). Go fearlessly into the new world order of business intelligence, and know that I , Laura E. your Dashboard Design Diva, called Social Intelligence the New Order, in 2005, again in 2006 and 2007. 🙂 Cheers, ya’ll.

http://tinyurl.com/ckfmya8

https://my.microstrategy.com/MicroStrategy/servlet/mstrWeb?pg=shareAgent&RRUid=1173963&documentID=4A6BD4C611E1322B538D00802F57673E&starget=1

Continue reading

Business Intelligence Clouds – The Skies the Limit

I am back…(for now, or so it seems these days) – I promise to get back to one post a month if not more.

Yes, I am known for my frequent use of puns, bordering on the line between cheesy and relevant. Forgive the title. It has been over 110 days since I last posted, which for me is a travesty. Despite my ever growing list of activities both professional and personally, I have always put my blog in the top priority quadrant.

Enough ranting…I diverged; and now I am back.

Ok, cloud computing (BI tools related) seems to be all the rage. Right up there with Mobile

BI, big data and social. I dare use my own term coined back in 2007 ‘Social Intelligence’ as now others have trade marked this phrase (but we, dear readers, know the truth –> we have been thinking about the marriage between social networks / social media data sets and business intelligence for years now)…Alas, I diverge again. Today, I have been thinking a lot about cloud computing and Business Intelligence.

Think about BI and portals, like Sharepoint (just to name 1)…It was all of the rage (or perhaps, still is)…”Integrate my BI reporting with my intranet / portal /Sharepoint web parts…OK, once that was completed successfully, did it buy much in terms of adoption or savings or any number of those ROI / savings catch – “Buy our product, and your employees will literally save so much time they will be basket weaving their reports into TRUE analysis'” What they didnt tell you, was that more bandwidth meant less need for those people, which in turn, meant people went into scarcity mode/tactics trying to make themselves seem or be relevant…And I dont fault them for this…Companies were not ready or did not want to think about what they were going to do with the newly freed up resources that they would have when the panacea of BI deployments actually came to fruition…And so, the wheel turned. What was next…? Reports became dashboards; dashboards became scorecards (became the complements for the former); Scorecards introduced proactive notification / alerting; alerting introduced threshold based notification across multiple devices/methods, one of which was mobile; mobile notification brought the need for mobile BI –> and frankly, and I will say it: Apple brought us the hardware to see the latter into fruition…Swipe, tap, double tap –> drill down was now fun. Mobile made portals seem like child’s play. But what about when you need to visualize something and ONLY have it on a spreadsheet?

(I love hearing this one; as if the multi-billion dollar company whose employee is claiming to only have the data on a spreadsheet didnt get it from somewhere else; I know, I know –> in the odd case, yes, this is true…so I will play along)…

The “only on a spreadsheet” crowd made mobile seem restrictive; enter RoamBI and the likes of others like MicroStrategy (yes, MicroStrategy now has a data import feature for spreadsheets with advanced visualizations for both web and mobile)…Enter Qlikview for the web crowd. The “I’m going to build-a dashboard in less than 30 minutes” salesforce “wait…that’s not all folks….come now (to the meeting room) with your spreadsheet, and watch our magicians create dashboards to take with you from the meeting”

But no one cared about maintenance, data integrity, cleanliness or accuracy…I know…they are meant to be nimble, and I see their value in some instances and some circumstances…Just like the multi-billion dollar company who only tracks data on spreqadsheets…I get it; there are some circumstances where they exist…But, it is not the norm.

So, here we are …mobile offerings here and there; build a dashboard on the fly; import spreadsheets during meetings; but, what happens when you go back to your desk and have to open up your portal (still) and now have a new dashboard that only you can see unless you forward it out manually?

Enter cloud computing for BI; but not at the macro scale; let’s talk , personal…Personal clouds; individual sandboxes of a predefined amount of space which IT has no sanction over other than to bless how much space is allocated…From there, what you do with it is up to you; Hackles going up I see…How about this…

Image representing Salesforce as depicted in C...
Image via CrunchBase

Salesforce.com –> The biggest CRM cloud today. And for the last many years, SFDC has

enbraced Cloud Computing. And big data for that matter; and databases (database.com in fact) in the cloud…Lions and tigers and bears, oh my!

So isnt it natural for BI to follow CRM into cloud computing ?? Ok, ok…for those of you whose hackles are still up, some rules (you IT folks will want to read further):

Rules of the game:

1) Set an amount of space (not to be exceeded; no matter what) – But be fair and realistic; a 100 MB is useless; in today’s world, a 4 GB zip drive was advertised for $4.99 during the back to school sales, so I think you can pony up enough to help make the cloud useful.

2) If you delete it, there is a recycling bin (like on your PC/Mac); if you permanently delete it, too bad/so sad…We need to draw the line somewhere. Poor Sharepoint admins around the world are having to drop into STSADM commands to restore Alvin Analyst’s Most Important Analysis that he not only moved into recycling bin but then permanently deleted.

3) Put some things of use in this personal cloud at work like BI tools; upload a spreadsheet and build a dashboard in minutes wiht visualizations like the graph matrix (a crowd pleasure) or a time series slider (another crowd favorite; people just love time based data 🙂 But I digress (again)…

4) Set up BI reporting on the logged events; understand how many users are using your cloud environment; how many are getting errors; what and why are they getting errors; this simple type of event based logging is very informative. (We BI professionals tend to overthink things, especially those who are also physicists).

5) Take a look at what people are using the cloud for; if you create and add meaningful tools like BI visualizations and data import and offer viewing via mobile devices like iPhone/iPad and Android or web, people will use it…

This isnt a corporate iTunes or MobileMe Cloud; this isnt Amazon’s elastic cloud (EC2). This is a cloud wiht the sole purpase of supporting BI; wait, not just supporting, but propelling users out of the doldrums of the current state of affairs and into the future.

It’s tangible and just cool enough to tell your colleagues and work friends “hey, I’ve got a BI cloud; do you?”

BIPlayBook.Com is Now Available!

As an aside, I’m excited to announce my latest website: http://www.biplaybook.com is finally published. Essentially, I decided that you, dear readers, were ready for the next step.  What comes next, you ask?

After Measuring BI data –> Making Measurements Meaningful –> and –>Massaging Meaningful Data into Metrics, what comes next is to discuss the age-old question of ‘So What’? & ‘What Do I Do About it’?

BI PlayBook offers readers the next level of real-world scenarios now that BI has become the nomenclature of yesteryear & is used by most to inform decisions. Basically, it is the same, with the added bonus of how to tie BI back into the original business process, customer service/satisfaction process or really any process of substance within a company.

This is quite meaningful to me because so often, as consumers of goods and services, we find our voices go unheard, especially when we are left dissatisfied. Can you muster the courage to voice your issue (dare I say, ‘complain’?) using the only tools provided: poor website feedback forms, surveys or (gasp) relaying our issue by calling into a call center(s) or IVR system (double gasp)? I don’t know if I can…

How many times do we get caught in the endless loop of an IVR, only to be ‘opted-out’ (aka – hung up on) when we do not press the magical combination of numbers on our keypads to reach a live human being, or when we are sneaky, pressing ‘0’ only to find out the company is one step ahead of us, having programmed ‘0’ to automatically transfer your call to our friend:  ‘ReLisa Boutton’ – aka the Release Button().

Feedback is critical, especially as our world has become consumed by social networks. The ‘chatter’ of customers that ensues, choosing to ‘Like’ or join our company page or product, or tweet about the merits or demerits of one’s value proposition, is not only rich if one cares about understanding their customer. But, it is also a key into how well you are doing in the eyes of your customer. Think about how many customer satisfaction surveys you have taken ask you whether or not your would recommend a company to a friend or family member.

This measure defines one’s NPR, or Net Promoter Rank, and is a commonly shared KPI or key performance indicator for a company.

Yet, market researchers like myself know that what a customer says on a survey isn’t always how they will behave. This discrepancy between what someone says and what someone does is as age-old as our parents telling us as children “do not as I do, but as I say.” However, no longer does this paradigm hold true. Therefore, limiting oneself by their NPR score will restrict the ability to truly understand one’s Voice of the Customer. And further, if you do not understand your customer’s actual likelihood to recommend to others or repeat purchase from you, how can you predict their lifetime value or propensity for future revenue earnings? You can’t.

Now, I am ranting. I get it.

But I want you to understand that social media content that is available from understanding the social network spheres can fill that gap. They can help you understand how your customers truly perceive your goods or services. Trust me, customers are more likely to tweet (use Twitter) to vent in 140 characters or less about a negative experience than they are to take the time to fill out a survey. Likewise, they are more likely to rave about a great experience with your company.

So, why shouldn’t this social ‘chatter’ be tied back into the business intelligence platforms, and further, mined out specifically to inform customer feedback loops, voice of the customer & value stream maps, for example?

Going one step further, having a BI PlayBook focuses the attention of the metric owners on the areas that needs to be addressed, while filtering out the noise that can detract from the intended purpose.

If we are going to make folks responsible for the performance of a given metric, shouldn’t we also help them understand what is expected of them up front, as opposed to when something goes terribly wrong, signified by the “text message” tirade of an overworked CEO waking you out of your slumber at 3 AM?

Further, understanding how to address an issue, who to communicate to and most importantly, how to resolve and respond to affected parties are all part of a well conceived BI playbook.

It truly takes BI to that next level. In fact, two years ago, I presented this very topic at the TDWI Executive Summit in San Diego (Tying Business Processes into your Business Intelligence). While I got a lot of  stares ala ‘dog tilting head to the side in that confused glare at owner look’, I hope people can draw back on that experience with moments of ‘ah ha – that is what she meant’ now that they have evolved ( a little) in their BI maturation growth.

Gartner BI Magic Quadrant 2011 – Keeping with the Tradition

Gartner Magic Quadrant 2011

Gartner Magic Quadrant 2011

I have posted the Gartner Business Intelligence ‘BI’ Magic Quadrant (in addition to the ETL quadrant) for the last several years.  To say that I missed the boat on this year’s quadrant is a bit extreme folks, though for my delay, I am sorry. I did not realize there were readers who counted on me to post this information each year.  I am a few months behind the curve on getting this to you, dear readers.  But, what that said, it is better late, than never, right?

Oh, and who is really ‘clocking’ me anyway, other than myself? But that is a whole other issue for another post, some other day.

As an aside, am excited to say that my latest websites http://www.biplaybook.com is finally published. Essentially, I decided that the next step after Measuring BI data, Making the Measurements Meaningful, and Modifying Meaningful Data into Metrics was to address the age old question of ‘So What’? Or ‘What Do I Do About it’?

BI PlayBook offers readers real-world scenarios that I have solved using BI or data visualizations of sorts, but with the added bonus, of how to tie it back into the original business process you were reporting on or trying to help with BI, or tie back into the customer services/satisfaction process. This latter one is quite meaningful to me, because so often, we find our voices go unheard, especially when we complain to large corporations via website feedback, surveys or (gasp) calling into their call center(s). Feedback should be directly tied back into the performance being measured whether it is operational, tactical, managerial, marketing, financial, retail , production and so forth. So, why not tie that back into your business intelligence platforms using feedback loops and voice of the customer maps /value stream maps to do so.

Going one step further, having a BI PlayBook allows end users of your BI systems who are signed up and responsible for metrics being visualized and reported out to the company to know what they are expected to do to address a problem with that metric, who they are to communicate both the issue and the resolution to, and what success looks like.

Is it really fair of us, BI practitioners, to build and assign responisble ownership to our leaders of the world, without giving them some guidance (documented of course), on what to do about these new responsibilities? We are certainly the 1st to be critical when a ‘red’ issue shows up on one of our reports/dashboards/visualizations. How cool would it be to look at these red events, see the people responsible getting alerted to said fluctation, and further, seeing said person take appropriate and reasonable steps towards resolution? Well, a playbook offers the roadmap or guidance around this very process.

It truly takes BI to that next level. In fact, two years ago, I presented this very topic at the TDWI Executive Summit in San Diego (Tying Business Processes into your Business Intelligence). The PlayBook is the documented ways and means to achieve this outcome in a real-world situation.

To Start Quilting, One Just Needs a Set of Patterns: Deconstructing Neural Networks (my favorite topic de la journée, semaine ou année)

 

How a Neural Network Works:

Neural NetworkA neural network (#neuralnetwork) uses rules it “learns” from patterns in data to construct a hidden layer of logic. The hidden layer then processes inputs, classifying them based on the experience of the model. In this example, the neural network has been trained to distinguish between valid and fraudulent credit card purchases.

This is not your mom’s apple pie or the good old days of case-based reasoning or fuzzy logic. (Although, the latter is still one of my favorite terms to say. Try it: fuzzzzyyyy logic. Rolls off the tongue, right?)…But I digress…

And, now, we’re back.

To give you a quick refresher:

image

Case based reasoning represents knowledge as a database of past cases and their solutions. The system uses a six-step process to generate solutions to new problems encountered by the user.

We’re talking old school, folks…Think to yourself, frustrating FAQ pages, where you type a question into a search box, only to have follow on questions prompt you for further clarification and with each one, further frustration. Oh and BTW, the same FAQ pages which e-commerce sites laughably call ‘customer support’ –

“ And, I wonder why your ASCI customer service scores are soo low Mr. or Mrs. e-Retailer :),” says this blogger facetiously, to her audience .

 

 

 

And, we’re not talking about fuzzy logic either – Simply put, fuzzy logic is fun to say, yes, and technically is:

fuzzy logic

–> Rule-based technology with exceptions (see arrow 4)

–> Represents linguistic categories (for example, “warm”, “hot”) as ranges of values

–> Describes a particular phenomenon or process and then represents in a diminutive number of flexible rules

–> Provides solutions to scenarios typically difficult to represent with succinct IF-THEN rules

(Graphic: Take a thermostat in your home and assign membership functions for the input called temperature. This becomes part of the logic of the thermostat to control the room temperature. Membership functions translate linguistic expressions such as “warm” or “cool” into quantifiable numbers that computer systems can then consume and manipulate.)

 

Nope, we are talking Neural Networks – the absolute Bees-Knees in my mind, right up there with social intelligence and my family (in no specific order :):

–> Find patterns and relationships in massive amounts of data that are too complicated for human to analyze

–> “Learn” patterns by searching for relationships, building models, and correcting over and over again model’s own mistakes

–> Humans “train” network by feeding it training data for which inputs produce known set of outputs or conclusions, to help neural network learn correct solution by example

–> Neural network applications in medicine, science, and business address problems in pattern classification, prediction, financial analysis, and control and optimization

 

Remember folks: Knowledge is power and definitely an asset. Want to know more? I discuss this and other intangibles further in part 1 of a multi-part study I am conducting called:

weemee Measuring Our Intangible Assets, by Laura Edell

Gartner VP Addresses Prerequisites for Developing Corporate Social Media Policies

Carol Rozwell might be my personal hero, well-respected and distinguished Gartner analyst. "Social media offers tempting opportunities to interact with employees, business partners, customers, prospects and a whole host of anonymous participants on the social Web," said the analyst and vice-president recently,  "However, those who participate in social media need guidance from their employer about the rules, responsibilities, ‘norms’ and behaviors expected of them, and these topics are commonly covered in the social media policy."

Gartner has identified seven critical questions that designers of social media policy must ask themselves:

What Is Our Organization’s Strategy for Social Media?
There are many possible purposes for social media. It can be used for five levels of increasingly involved interaction (ranging from monitoring to co-creation) and across four different constituencies (employees, business partners, customers and prospects, and the social Web). It is critical that social media leaders determine the purpose of their initiatives before they deploy them and that those responsible for social media initiatives articulate how the organization’s mission, strategy, values and desired outcomes inform and impact on these initiatives. A social media strategy plan is one means of conveying this information.

Who Will Write and Revise the Policy?
Some organizations assign policy writing to the CIO, others have decided it’s the general counsel’s job, while in other cases, a self-appointed committee decides to craft a policy. It’s useful to gain agreement about who is responsible, accountable, consulted and involved before beginning work on the policy and, where possible, a cross-section of the company’s population should be involved in the policy creation process. It’s important to remember that there is a difference between policy — which states do’s and don’ts at a high level — and operational processes, such as recruitment or customer support — which may use social media. These operational processes need to be flexible and changeable and adhere to the policy, but each department/activity will need to work out specific governance and process guidelines.

How Will We Vet the Policy?
Getting broad feedback on the policy serves two purposes. First, it ensures that multiple disparate interests such as legal, security, privacy and corporate branding, have been adequately addressed and that the policy is balanced. Second, it increases the amount of buy-in when a diverse group of people is asked to review and comment on the policy draft. This means that the process by which the policy will be reviewed and discussed, along with the feedback, will be incorporated into the final copy. A vetting process that includes social media makes it more likely that this will occur.

How Will We Inform Employees About Their Responsibilities?
Some organizations confuse policy creation with policy communication. A policy should be well-written and comprehensive, but it is unlikely that the policy alone will be all that is needed to instruct employees about their responsibilities for social media. A well-designed communication plan, backed up by a training program, helps to make the policy come to life so that employees understand not just what the policy says, but how it impacts on them. It also explains what the organization expects to gain from its participation in social media, which should influence employees in their social media interactions.

Who Will Be Responsible for Monitoring Social Media Employee Activities?
Once the strategy has been set, the rules have been established and the rationale for them explained, who will ensure that they are followed? Who will watch to make sure the organization is getting the desired benefit from social media? A well-designed training and awareness program will help with this, but managers and the organization’s leader for social media also need to pay attention. Managers need to understand policy and assumptions and how to spot inappropriate activity, but their role is to be more of a guide to support team self-moderation, rather than employ a top-down, monitor-and-control approach.

How Will We Train Managers to Coach Employees on Social Media Use?
Some managers will have no problem supporting their employees as they navigate a myriad of social media sites. Others may have more trouble helping employees figure out the best approach for blogs, microblogs and social networking. There needs to be a plan for how the organization will give managers the skills needed to confront and counsel employees on this sensitive subject.

How Will We Use Missteps to Refine Our Policy and Training?
As with any new communications medium, some initiatives go exceptionally well, while others run adrift or even sink. Organizations that approach social media using an organized and planned approach, consistent with the organization’s mission, strategy and values, will be able to review how well these initiatives meet their objectives and use that insight to improve existing efforts or plan future projects better.

More information is available in the report "Answer Seven Critical Questions Before You Write Your Social Media Policy," which can be found on the Gartner website at http://www.gartner.com/resId=1522014.

 

In addition, I wanted to add the following points:

I am all about the process – And a process for establishing a social media strategy (internal or externally facing) have several process steps which flow sequentially for the varying audience members who will consume or provide this information.

 

First, It is important to understand your corporate strategic goals. And even if social media isn’t explicitly defined, it is certainly an input to several common objectives like acquire/retain new/existing customers (Marketing), World-Class operations (real-time fodder is a great tool for customer service complaints in real time), etc.

Second, you need to functionally understand the impact domains and what purpose a social strategy will provide: which groups will be impacted by a social media strategy, and what, if anything are they already doing to address? Characteristics of a good purpose according to Carol Rozwell:

 

1. Magnetic
2. Aligned
3. Properly-scoped
4. Promotes Evolution
5. Low risk
6. Measurable
7. Community-driven

 

Third, connecting the corporate goals from the strategic plans to the social media purpose / strategy is key – that is what is defined by Aligned and Properly Scoped. All strategic plans evolve over time so why wouldn’t your social purpose evolve as well?

 

Fourth, Measurement. This is near and dear to my heart : Measuring what matters; business intelligence tools are starting to realize the value of offering real-time capabilities to track the chatter across the social sphere; think about my Wynn Hotel examples from previous posts to validate the power this can provide towards improving customer experience, and ultimately affecting long-term retention of your customers.

Community-driven is self-explanatory. You cannot tell a customer what their voice should be, it is what it is.

You as an organization need to understand that word of mouth from your customers is worth its weight in gold; more than the millions spent on advertising budgets and huge marketing campaigns. Communities offer the soap box that so many customers want to stand upon to share their experiences.

You reap the benefits of understanding this voice, and consuming this information in a meaningful and metrics driven approach that can provide context to your strategic goals without augmenting them with cost laden initiatives or proposals.

“LAURA” Stratification: Best Practice for Implementing ‘Social Intelligence’

Doing an assessment for how and where to learn social media to better understand your business drivers can be daunting, especially when you want to overlay how those drivers affect your goals, customers, suppliers, employees, partners…you name it.

I came up with this process which happens to mimic my name (shameless self-persona plug) to ease the assessment process while providing a guided assessment plan.

First, ‘Learn’ to Listen: learning from the voice of the customer/supplier/partner is an extremely effective way to understand how well you are doing retaining, acquiring or losing your relationships with those who you rely on to operate your business.

Second, Analyze what matters, ignore or shelve (for later) what doesn’t; data should be actionable, (metrics in your control to address), reporting key performance indicators that are tied to corporate strategies and goals to ensure relevancy.

Third, Understand your constituent groups; it isn’t just your customers, but also your shareholders, employees, partners, and suppliers who can make or break a business through word of mouth and social networking.

Fourth, Relate your root causes to your constituents value perceptions, loyalty drivers and needs to ensure relevancy flow through from step 2. Map these to your business initiatives and goals exercise from step 2. Explore gaps between initiatives, value perceptions, loyalty drivers and corporate goals.

Lastly, create Action plans to address the gaps discovered in Step 4. If you analyzed truly actionable data in step 2, this should be easy to do.

To apply this to social media in order to turn it into social intelligence, you need to make the chatter of the networks meaningful and actionable.

To do this, think about this example:

 

A person tweets a desire to stop using a hotel chain because of a bad experience. In marketing, this is known as an “intent to churn” event; when social intelligence reporting systems ferrets out this intent based on scouring the web commentaries of social networks, an alert can be automatically forwarded to your customer loyalty, marketing/social media or customer response teams to respond, address and retain said customer.

A posting might say “trouble with product or service” – That type of message can be sent to customer operations (service) or warranty service departments as a mobile alert.

And a “having trouble replenishing item; out of stock” question on a customer forum can be passed along to your supply chain or retail teams — all automatically.

The Wynn has a great feedback loop using social media to alert them in real-time of customers who are dissatisfied with their stay who Tweet or comment about this during their stay.

The hotel manager and response time will find this person to address and rectify the situation before they check out. And before long, the negative tweet or post is replaced by an even more positive response, and best of all, WORD of MOUTH to friends and family.

Its sad to say, in this day and age, we are often left without a voice or one that is heard by our providers of services / products. When good service comes, we are so starved that we rejoice about it to the would. And why not? That is how good companies excel and excellent companies  hit the echelon of amazing companies!

‘Social Intelligence’, the bridge between social networking and business intelligence, Starts To Build Momentum

Several years ago (in early 2009), I blogged about two of my passions, social networking and business intelligence. It was about the time that business folks starting building their profiles on LinkedIn, extending their networks via Twitter and started realizing that FaceBook wasn’t just a tool for their children to build their socialization skills but was a vehicle for networking with other professionals within and outside of their own personal networks. Grasping the power of the social network was still this abstruse almost arcane concept in its theoretical potential for corporate America. And while there were those visionaries, like the Wynn in Las Vegas, about whom I shared an anecdote within my TDWI presentation on Social Intelligence (one I will share in a moment) later that year, most companies saw social networking websites as distractions and often, banned them from use during the work day.

Why was Wynn different?

As a frequent corporate traveler, I have had many “check-in” line experiences: from the car rental counter to the hotel check-in line, I have had both good and bad experiences. On one somewhat lackluster experience, I was standing in line to check into the Wynn Hotel in Vegas. Several people ahead of me was a gentleman, fairly polished but obviously frustrated by his conversation with the desk clerk. As a highly perceptive observer (or at least, that is how I am spinning being nosy), I listened in on the situation. This gentleman had reserved a junior suite, since he and a colleague were sharing the room, a common occurrence as companies started to tighten their belts around corporate travel expenses. And, the suite was not available. The clerk seemed to want to help but was strapped by her computer system telling her no suites were available until the following night in the category booked. It turns out, she was new.

Quite gruffly, this gentlemen left the line, and proceeded to stand in the lobby, talking to his colleague about the disappointment, and commented that he was going to Tweet (post a message to his Twitter account) that buyer beware when it came to staying at the Wynn. Now, in a city like Las Vegas where capacity can exceed occupancy rates, combined with a name like the Wynn, combined with the sheer reach of a site like Twitter, this kind of negative word of mouth can really hurt a vendor. And more often than not, comments like this are over looked, or at least, were overlooked in the past, because of the lack of technology or reporting to alert such vendors to such disturbances in real time. In a travel situation, do you want to know that your issue was addressed after your trip with a gift and apology in the form of a coupon for choosing the stay there in future?

No…In fact, the breakage rate on such post-trip coupons is 70-80% (remember, I used to work for the largest online travel consortium) :). Thus, granting coupons is ineffective at winning the customer back. And it is because your trip, whether for business or pleasure, was ruined. And no, I am not being dramatic. You might not think a rooming issue can ruin a trip but it can. In fact, just being placed on the wrong floor or near an elevator or merely any event that is different that you were expecting can ruin a trip from a customer’s perspective.

But, I digress…

Back to my story: As soon as the customer finished posting his Tweet to Twitter, he turned to his colleague and walked to a cafe and sat down to order some refreshments. By the time I reached the front of the check in line, I noticed what appeared to be someone who appeared to be in charge (dark suit, name plate, piece of paper in his hand) approach the gentlemen and start a dialogue with him. Within moments, the two shook hands and the paper (which turned out to be room keys and an invoice) were swapped and the authority figure left about his business.

Intrigued, I walked up and asked the gentlemen what had happened. He was so excited by what had happened that he asked me to wait while he posted a note to Twitter. Since I had heard the original part of the story, I started to deduce what was happening. When he was finished, he said that gentleman was the hotel manager. He had been alerted to the room situation via a Twitter application which alerted management to travel disruptions as they occurred in real time to his smart phone. It was his job to make sure the customers were found in the hotel and the situation fixed to the betterment of the customer, no matter the situation. In this case, the customer was treated to an upgraded full suite, which was available, at no additional cost and given vouchers for the show that evening. The customer was so pleased, he had to go back to Twitter to recant his previous post, and to alert people to how well the situation was handled not days after the fact, but within the hour of it occurring.

I was floored.

You hear about the concept of the customer feedback loop but rarely do you see it implemented well or in a way that can affect overall customer loyalty or perception of the brand. In this case, it not only affected the customer and his colleague, but his entire social network.

Later, I found that same manager and asked his what he used to alert him to the Twitter incident from earlier.

He smiled and said we are in the business of pleasure, thus, it is our job to know when we fail. Alerting in real time is not as hard as you think with the right tools and technology. And left it at that.

Ok, so Vegas is a pretty secretive world of proprietary tools and technology, and are often market leaders when it comes to adoption.

And that is where Social Intelligence comes in: the ability to understand the Voice of the Customer as expressed within the intricate web of the social network via tools and technology. What better tools for alerting and reporting on incidents in real-time than those offered by the Business Intelligence suite of tools (at its most generalized state).

I am so happy to also report that in 2011, BI technology is taking an even larger footprint into the Social Intelligence space. When I can say more, I will. Just know I am really excited about the future ahead of us folks!

Happy New Year readers.

Applying the Bing Decision-Engine Model to “Business Intelligence” and Other Musings

Yes, folks, I am back. Wait, didn’t I write that before.

Well, after having my 1st child, I spent many months (just shy of 10, to be exact), noodling business intelligence, and the concepts I had previously discussed on my blog. For the last 5 years, I have been touting the need for better search integration, offering up the BI mashup concept before people really understood what a plain vanilla dashboard was, and was met by glazed stares and confusion. Now that folks are catching on to the iGoogle experience, and the ability to “mashup” or integrated several points of interest or relevance into a dashboard, I want to discuss this topic again. But, this time, I want to apply the concept of the Decision Engine instead of just the Search Engine when it comes to ways to make BI content more meaningful, more relevant and more useful to end users.

Side note: “mashup” is still not a recognized word in the spell-check driven dictionary lists for the greater population of enterprise applications.

Coupled with my mashup passion was my belief in eye-tracking studies. Eye-tracking measures the human behavior of looking at something and measuring the concentration of the eyes on a particular area of a particular object of interest, say a website for example. In the case of business intelligence, I applied eye-tracking studies to the efficacy of dashboard design in order to better understand the areas where the human brain focused concentration vs. those ignored (despite what the person might say was of interest to them).

Advertisers have known about eye-tracking studies for years, and have applied the results to their business. For example, the eyes will focus on the top left corner first. Whether a TV screen, a book, a piece of paper or a dashboard. It is the area of the greatest concentration. Therefore, special importance has been paid to the piece of advertising real estate. And since the popularity rise of folks like Stephan Few of recent or Edward Tufte, whose design principles for effective dashboard design have driven many a BI practitioner to rethink the look and feel of what they are designing, this concept of top left is more important has become commonplace.

And, the handful of other book grade principles have risen to the surface too: less is more when it comes to color, overuse of lines in graphs is distracting, save pie for desert (pie charts, that is), etc.  But tying it all together is another story all together. Understanding how human perception, visual perception and dashboard design meet is a whole other can of worms, and usually requires a specialized skill set to fully “grok” (sorry, but I love Heinlein’s work). 🙂

Excuse my digression…


Take a look at this image which shows eyetracking results from the three most popular search engines in 2006:

 

Notice the dispersion of color measured in the Yahoo and MSN examples vs. Google. This is correlated to the relevancy of the results and content presented on the page. And 4 years ago, Google’s search engine was a popular go-to tool for many when it came to finding related websites to help answer questions. Fast forward 4 years, and MSN is now Bing, and what was the search engine is now the dubbed “decision engine.”

The advent of the decision-engine in my eyes is because of the dilution of search engine effectiveness based on the flood of results presented to end users. In fact, I am sure the results of an eye tracking study from 2010 would be vastly different as a result of the exponential growth of web-based content available for crawling.

The same has occurred within enterprise business-intelligence platforms. What was introduced as powerful has really become inundated with content, in the form of reports, objects, dimensions, attributes, attribute elements, actual metrics, derived metrics and the list goes on and on.

Superficially, search was introduced as an add-on to the enterprise BI platforms. An add-on; really, an afterthought.

To the credit of the solutions on the market (grouped into a collective unit), people didn’t realize what they didn’t or better put, needed to know when building the technology behind their solution offerings. And they needed to start somewhere. It was only after BI became more mass-adopted in corporate America, and the need grew pervasive into even the smallest Mom and Pop shop for some level of reporting, that people began to realize that getting the visualizing the data was one thing; finding the results of those visualizations or data extractions was an entirely different can of worms.

At the same time as this was happening, the search giants started innovating and introducing the world to the concepts of real-time search and the “decision engine” named Bing. Understanding the statistical models behind how search algorithms work, even simplistically, understanding enough to be dangerous, is a key that any reader of this blog and any BI practitioner would be smart to invest their own time into doing. 

In a nutshell, my belief? Applying those principles and eons of dollars thrown at optimizing said models (by the search giants) is an effective way for BI solutions at any level to leverage the work done to advance search research and technology, instead of just patching BI platforms with ineffective search add-ons. Just look back at the Golden Triangle study graphic above, and remember that long before BI design experts like Tufte and Few said it, advertising gurus knew that the Top Left real estate of any space is the most important space to reach end users. So, instead of thinking of search as a nice add-on for your BI platforms, why not see it as a necessity. if a report is loaded into a repository and no one knows about it, was it ever really done? Let alone meaningful or valuable enough to be adopted by your end users? Think about it…

Gartner 2010 Business Intelligence Tools Magic Quadrant

For those of you who prefer not to register to receive this information, here is the 2010 Gartner Magic Quadrant rating the latest and greatest BI Platforms.

 

I love how many of the pure play newbies of last season like QlikTech moved from visionaries into the challengers role giving the big dogs on campus, MS, SAP/Business Objects and Oracle a run for their money. And while, value can be shown easily using a product which can consume and spit out dashboards as easily as making scrambled eggs in the morning, one has to wonder how much value it provides over time when the data to support such dashboards often still requires much manual intervention, ie. acquisition from source systems, cleansing, transformation and loading into a consumable format. Where’s the ROI in that? Most systems boast on the time savings achieved with implementation when calculating a BI system’s ROI.

 

But, not to knock them. I find them a great alternative for proof of concept work or when the manual nature of compiling the data isnt a concern or is someone’s role, and all that is needed is the icing to tell the cunning story (“Once upon a time, there was a SKU…And this SKU had many family members who lived in different houses in different regions of the world”)

 

Aah yes, if only all BI could be told as such a happy little anecdote of a story…A girl can wish can’t she?

Download Gartner 2010 Report Click Here

Report is also available in my SkyDrive library.

Data Visualization: Looking vs. Seeing

For many years vision researchers have been investigating how humans use theiw own visual cortex and other perceptions based systems to analyze images. An important initial result was the discovery of a very small subset of visual properties detectable very quickly & for a large part, very accurately, by the lowest of these systems, aptly referred to as low-level visual system.

These properties were initially called “preattentive”, since their detection seemed to come before one actually focused their attention.

Since then, we have a better understand. As it stands today, attention plays a critical role in what we see, even at this early stage of vision. The term preattentive continues to be used, however, since it conveys the speed and ease of visual identification of these properties.

Anne Treisman determined two types of visual search tasks: 1 which is preattentive known as Feature search, and the other which requires conscious attention or Conjunction search. Feature search can be performed fast and pre-attentively for targets defined by primitive features. 

The features or properties can be broken into 3 areas: color, orientation and intensity.

And as you might have guessed, Conjunction search is slower and requires the participant’s full attention, something we humans have a hard time giving in certain situations, which is only worsening with the advent of hand held devices and other mobile smart phones to distract us. 

“Typically, tasks that can be performed on large multi-element displays in less than 200 to 250 milliseconds (msec) are considered preattentive. Eye movements take at least 200 msec to initiate, and random locations of the elements in the display ensure that attention cannot be prefocused on any particular location, yet viewers report that these tasks can be completed with very little effort. This suggests that certain information in the display is processed in parallel by the low-level visual system.” (“Perception in Visualization by Christopher Healey”)

What does this mean: well, given a target, say a red circle, and distractors being everything else, which is this case are blue objects,one can quickly see in this example which is which, i.e. in < 200 msec, you can glance at these two pictures and define the target from the distractors, right?

  As in this example, it seems introducing preattentive cognition to dashboards would result in a healthy and loving relationship and one when carried over time (ie – employed by BI practitioners during design phase of any BI / data visualization project) would result in more meaningful, & less cluttered dashboards, right? 

Now, think about your dashboards and BI visualizations – Think about how many of them tell a good and clean story, where the absolute most important information “pops” out to end viewer. One requiring little explanatory text, contextual help or other mechanisms we BI practitioners employ to explain our poorly designed dashboards. And, I am by no means claiming everything I have designed to be fault free– We all learn as we go. But I can say that those designs of today vs. yesterday are better because of my understanding of visual perception, neural processing / substrates and cognitive sciences and how to apply these fields to business intelligence in order to drive better data visualizations.

Why is it that some who work in BI think the more gauges or widgets pushed into a screen, the better?

Instead, I contend that the application of this principle to dashboard design, report design, website design or any type of design would point out that much in our world today is poorly designed, fitting with non complementary colors, over use of of dristractor objects, thus, rendering the user confused or “distracted” from the target object, which could be something as important as revenue of a company, or number of death in an ER wing of a hospital, both of which so important as one might question how such numbers could get lost.

 

Try it for yourself by reading anything by Stephen Few or Edward Tufte as a starting place.

Talking about Getting Started \ Processing 1.0

 

Quote

Getting Started \ Processing 1.0

Gotta ask my audience for commentary on this one…How many of you are using Processing 1.0 environment/language to build your complex data visualizations?

Processing.org quotes it as "Processing is a simple programming environment that was created to make it easier to develop visually oriented applications with an emphasis on animation and providing users with instant feedback through interaction." (http://processing.org/learning/gettingstarted/)

I have been using this app since college and being a BI professional services/developer now, I tend to overlook the simplicity and ease of use of the Processing language, functions and environment (PDE).

Has anyone else used it for building data visualizing?

“BOTHER” (Before Offering To Heavily Expense, R$Invent): Unearthing BI Insights in the Most Unlikely of Places: the Existing Pools of Information Within Your Workplace

Yes. As many of you know, I took a hiatus from my blog to really get down to the nuts and bolts of business intelligence. As a BI solutions architect working for a leading BI consultancy, I get the wonderful benefit of getting to experience many different perspectives of BI in the workplace. Each year, I also get to watch how the industry grows in it’s implicit pervasiveness (and no, I do not mean BI pervasiveness, as in the latest “catch phrase” // one I would release BTW if you have added it to your vocabulary of late, but I digress)…No, what I mean, is the textbook definition : I get to watch how business intelligence is infecting the lives of countless employees, all with the promise of making lives easier, better, faster, eliminating manual practices without eliminating their jobs, enhancing decision making with data, empowering, illuminating, targeting, and the list goes on and on. But the reality of what I get to witness is that business intelligence has become the CRM that we all wanted to avoid; the nomenclature of late; the popular trend in the workplace, where one gets to order a dashboard on the side with their reporting platform. Oh, and for just a few dollars more, you can super size your order and get those ridiculous pie charts, with not just 3-D rendering capabilities but also in every hue and shade of the color spin-wheel of life…Spin a red, and lose your job, spin a green, and move up the proverbial corporate hierarchy – Opening up the paradigm of pandora’s BI box is not for the light hearted…Stop reading now if you are already getting queasy.

 

What was once a new area of interest for me, the uncovering of KPIs or key performance indicators, and building strategically targeted performance management programs around them, has sadly manifested into the CRM nightmare I predicated over 3 years ago in this very blog. When any one thing gets overexposed (think of that infamous hotel heiress), we are systematically programmed to shift focus elsewhere; the new, being more interesting than the old. And the ‘WHAT’ that dashboards tell (*it’s Red, it’s Green*) gets very old, very quickly. And before one measures the efficacy of their measurement system, (i.e. – metadata measurements of the usefulness *qualitatively* of their BI program, practitioners leap the quantum conclusion that BI isn’t effective anymore /wasn’t ever effective, in their respective work environments. 9 x out of 10, it is the leaders, who thought they wanted that extra helping of BI, after attending that year’s conference, flavored with some Norton or Kaplan or more generalized BI user conferences sponsored by the software vendors (slight shudder thinking about all of those workplace leaders who don’t know what they don’t know, and are invited by self-interested ‘teachers’ whose altruism stops as soon the invoice is signed and dated.) But again I digress.

And really, when it is all boiled down, besides being what you may perceive to be a rant on my part, is an impassioned plea for the next wave of BI to begin…A call to arms, for the vets of our industry to stop self-promoting for one second and to start helping those build better with what they already have. To stop buying the new flavors of this month, and stick with the vanilla or chocolate or strawberry . That’s what is so great about keeping it simple. Not only can you slice something individual and enjoy its flavor for a lifetime of richness personally; but if and when the flavor stops providing what you need, you can always layer on to the basics and actually create something new, like 2-tiered Swirls or for those even more risky, the 3-tiered über swirl, known to us ice cream aficionados as Neapolitan. 

Now, I realize one must crawl before they walk, and walk before they run, or at least, so I am told whenever I step onto my soapbox and herald for change in the way BI is being implemented today. Whether I champion change on the street or in this case, the cubicle aisles of the workplace, beating my drum, that yes, Virginia, there is a Santa Claus, and he likes it when BI delivers what it promises AND CAN deliver. It is up to the practitioners to hang up their green-colored glasses and start thinking about all of those reasons they got into BI in the 1st place: and it is a powerful step back to take; one I can personally speak to having just returned from my sojourn, now with a greater understanding that understanding ‘WHAT’ about a business is the surface level cut that indicators or operational reporting will yield. But looking further at the usefulness of one’s metrics, asking those painful questions like “what are you going to do with that data” instead of just becoming a reporting jockey, driving the ‘WHAT’ down to the ‘WHY’ is only half the battle; it is the ‘WHAT CAN BE DONE ABOUT IT’ that takes it to a whole other level. And only those analysts, truly inundated with the data from all areas of the company, not just finance, or operations, but market research, retail, development, etc., who can truly answer the timeless question of ‘So what do we do now;’ because they have the data to steer the powers that be in that direction –

This isn’t a tool that I am prescribing; it might be spreadsheets and hours of analyst bandwidth to finally arrive where you need to be to make your BI programs and platforms useful. And the only way to get there, is to take a step back and examine your business frankly, ask the right people the right questions, and finally, question (with respect) the answers you get or keep asking the FIVE WHY’s, until you get to root cause of an existing platforms efficacy. Otherwise, if you don’t change your approach, you will always get what you have always got’ and trust me, there are only a handful (< 5% of companies) doing this today; start now, by pulling of the band-aid, or kicking the crutch of expenditure away, and use what you got to explore what you have in your data stores, manual as it may be, to find the nuggets of gold you want to be successful. No, let me rephrase. The nuggets you NEED to be successful. Check back over the next few days for actual steps to achieve success. I will prescribe a 5 step DON’T BOTHER plan of attach for reinventing your BI program before you reinvest starting with where to start getting down and dirty with your existing analytics – This is not for the faint hearted – you’ll find many of today’s business intelligence practitioners tend to avoid, not know about, or are too intimidated to uncover what I will reveal – As we move into the new year, why not shift the paradigm of your existing BI mindset by taking a bite from the beefiest side of all: the analytic?!!!

More to come tomorrow…

A Data Architect’s Brain on Drugs, or Your Worst Nightmare

Cold Sweats as you begin testing cardinality – with palms shaking, you open the data architecture of what others’ have called a spaghetti coded mess…

You are instructed to look for the blue line;

Come on…it isn’t that hard to see they say…

Did you catch it, the one you need to test the correct relationship on? 😉

Talk about a data architect’s spaghetti coded nightmare!!!

Business intelligence: Adding value to daily decisions

 

Business intelligence in hospitality: Adding value to daily decisions

Insight you can act on equals business success


Related Links


Microsoft Business Intelligence Web site


Microsoft Hospitality Web site


IDC vendor share 2005


Integrated IT platform integrates, personalizes guest experience


Related Products

Microsoft Windows Server "Longhorn"

Microsoft SQL Server

2007 Microsoft Office system

Microsoft Office PerformancePoint Server 2007

After more than 10 years, business intelligence (BI) is catching on. In many organizations, everyone from C-level executives to the controller to the chef rely on dashboards, scorecards, and daily reports to provide information about their business and the entire enterprise.

A recent IDC Research study ("Worldwide Business Intelligence Tools 2005 Vendor Shares," October 2006, #202603) found that organizations are looking for more than just tools for queries and reports. People want insight from their BI solution to support collaborative analysis, forecasting, and decision-making, so that BI can help drive better business processes—and results. Microsoft BI solutions can provide such support—and have helped companies such as Hilton and Expedia save money, provide superior guest service, and improve business performance and the bottom line. In this article, we’ll discuss how the Microsoft Business Intelligence platform can help your company.

On This Page


Insight you can act on


Keeping scorecards to track BI


A system everyone can use


It’s all about forecasting


Business intelligence and beyond


The Microsoft Business Intelligence solution


Next steps

Insight you can act on

"The trend now is to move from reporting about the past to studying targeted information about how key metrics or key performance indicators (KPIs) compare to current goals," says Sandra Andrews, industry solutions director in the Retail & Hospitality group at Microsoft. "Delivering the right information to the right people in the right format at the right time is critical. Empowering employees with real-time views of where the business is now and where it’s headed adds value to daily decisions."

To accurately manage and forecast, you need an integrated system that provides one version of the truth, and then you need that information to be easily accessible to your teams. But many organizations in the hotel industry are still using different BI tools in different departments. Complicating the matter more—companies use separate systems for different locations. As a result, it can be extremely difficult to standardize information and reports, forecast staffing and supply needs, let alone provide real-time analytics. However, the business benefits for delivering information to people in a format they can use to take action or make better business decisions far outweighs the costs.

Top of page

Keeping scorecards to track BI

Scorecarding is an efficient, immediate way to capture the key data you need. Recently, Expedia implemented a scorecard solution to better serve online customers and put complex Web performance metrics and KPI at its analysts’ fingertips. The result? Automated data collection saved time and effort, allowing analysts to spend their time developing answers rather than crunching numbers.

"Customer satisfaction is essential to helping make Expedia a great company. With scorecarding, we have the means to evaluate how well we are doing to make the company even greater," says Laura Gibbons, manager of Customer Satisfaction & Six Sigma at Expedia. "And if scorecarding is adopted throughout the company, I believe we are that much closer to becoming the largest and most profitable seller of travel in the world."

Top of page

A system everyone can use

Making sense of enormous quantities of rapidly changing data, visualizing and prioritizing that information, and holding the organization accountable for specific performance metrics is essential for success. If you have insight that you can act on, then you can align those activities with corporate goals and forecasts. And by empowering people through familiar tools, you make it easier for your employees to access the information they need to build relationships with guests.

The Microsoft Business Intelligence platform leverages the Microsoft Office system on the front end, helping you create a BI solution that your people can use easily, without a steep learning curve. "Managers and executives can create reports in Excel, link them to PowerPoint, and easily update their reports and presentations. Hotel managers are already using Excel," Andrews says. "No matter what BI tool organizations adopt, ultimately the user extracts the data into an Excel file to manipulate it. By giving your people the information they need in the Office system right from the start, you reach all employees and increase collaboration. You change the way your company works."

Top of page

It’s all about forecasting

To provide the type of service that generates customer loyalty, you need to be able to pull data from multiple systems to analyze guest profiles, forecast trends, determine occupancy rates, or predict food and beverage sales. The right BI solution can help you manage your business, increase productivity, and provide the excellent service that builds customer loyalty.

For example, Hilton Hotels wanted an adaptable, scalable solution that would include demand-based pricing and improve forecasting for group, catering, and public-space sales. Hilton leveraged Microsoft’s Business Intelligence platform, deploying Microsoft SQL Server 2005 and using SQL Server Analysis and Reporting Services all running on the Microsoft Windows Server 2003 operating system. As a result of the Microsoft BI solution, Hilton increased their data processing rate by 300 percent. They reduced catering forecast time by 25 percent. And they improved customer service by accommodating more catering requests, all with a 15-percent reduction in deployment time. Kathleen Sullivan, vice president, Sales and Revenue Management Systems at Hilton Hotels, says, "SQL Server 2005 provides Hilton with the power and extensibility to deliver revenue analysis and forecasting capabilities."

Top of page

Business intelligence and beyond

One trend that’s already changing revenue and channel management is how BI is fueling a better understanding of convention space and catering needs to generate revenue for sales and catering.

Organizations are integrating customer relationship management (CRM) sales tools with business intelligence to help book their conventions and catering events. The sales department can determine which event will bring in the most all-property revenue. Harrah’s Entertainment, a forerunner in innovative use of BI, is using customer intelligence and CRM strategies for tracking and increasing customer loyalty. Harrah’s hands out credits to their guests each time they visit the casino and play games. Harrah’s then tracks visits and the more the guest visits, the greater the value of the reward. Harrah’s can predict the value of each guest, their habits, and how to increase each guest’s total revenue per available room (REVPAR).

Top of page

The Microsoft Business Intelligence solution

IDC’s competitive analysis report, "Worldwide Business Intelligence Tools 2005 Vendor Shares," found that Microsoft’s BI tools revenue growth was more than twice that of the other leading database management systems (DBMS) and legacy pure-play BI vendors.

The Microsoft Business Intelligence platform is a complete and integrated solution. Whether you use it as your data warehouse platform, your day–to-day user interface, or as an analysis and reporting solution, Microsoft provides the fastest growing business intelligence platform to support your needs. The Microsoft BI solution includes the following servers and client tools to enhance your business:

Microsoft SQL Server 2005 (along with Visual Studio 2005 and BizTalk Server 2006) provides advanced data integration, data warehousing, data analysis, and enterprise reporting capabilities to help ensure interoperability in heterogeneous environments and speed the deployment of your BI projects.

Microsoft SQL Server Reporting Services is a comprehensive, server-based reporting solution designed to help you author, manage, and deliver both paper-based and interactive Web-based reports.

Microsoft SQL Server Integration Services (SSIS) is a next generation data integration platform that can integrate data from any source. SSIS provides a scalable and extensible platform that empowers development teams to build, manage, and deploy integration solutions to meet unique integration needs.

Microsoft SQL Server Analysis Services (SSAS) provides tools for data mining with which you can identify rules and patterns in your data, so that you can determine why things happen and predict what will happen in the future – giving you powerful insight that will help your company make better business decisions. SQL Server 2005 Analysis Services provides, for the first time, a unified and integrated view of all your business data as the foundation for all of your traditional reporting, online analytical processing (OLAP) analysis, KPI scorecards, and data mining.

End-user tools build on the BI platform capabilities of Microsoft SQL Server.

Microsoft Office Excel 2007 helps you to securely access, analyze, and share information from data warehouses and enterprise applications. And maintain a permanent connection between their Office Excel spreadsheet and the data source.

Microsoft Office SharePoint Server 2007 becomes a comprehensive portal for all of the BI content and end-user capabilities in SQL Server Reporting Services and the Microsoft Office 2007 release, providing secure access to business information in one place. Excel Services allow customers to more effectively share and manage spreadsheets on the server.

Microsoft Office PerformancePoint Server 2007 offers an easy to use performance management application spanning business scorecarding, analytics, and forecasting to enable companies to better manage their business.

Business intelligence: Adding value to daily decisions

Libra – a week of house cleaning and independence is in store for you!

Your Horoscope – This Week (April 26 – May 2, 2009)

Don’t start or decide on anything of matter on Monday or Tuesday – Moon and Mars are in your ruling house where the Hermit is deriving a joyous solitude that may surprise you dear Libra, if you choose to listen. Your key word is independent, so break the chains of codependency now. Your love life looks hotter than ever. You can’t escape the demands and desires of your lover. The presence of Mars in your relationship zone indicates it’s time to clear the air. If there are any issues that have been pushed under the carpet, they’re about to be exposed. You may find your partner a lot more argumentative than usual. Talking over difficulties will help the energy in the relationship to move instead of stagnate. There may be some turbulence, but you’ll also feel a lot better for having shared your feelings.

 

Your Horoscope – April 2009

You’ll be torn between work and personal matters at home and need to find balance on April 1 and 2. Don’t let others push you into making decisions quickly. You’ll have the chance to establish warm and affectionate bonds and enjoy life on the weekend of April 4. Catch up on a work project on the evening of April 5. Take your obligations very seriously on April 6 and 7. The Moon in your sign on April 8 and 9 will relax you and help you take it easy for a couple of days. You’ll have to be proactive and assertive in some situations, though, and not wait for developments. Relationships may be strained on the weekend of April 11 if you don’t deal with issues immediately. April 13 and 14 would be a good time to take a day trip if you feel the need for a change of scene. Burn off some excess energy by hitting the gym. Work will be intense on April 15 and 16 and you may need to put in some extra hours to get things finished. You may want to get involved in a humanitarian cause or at least be of help to someone in need on the weekend of April 18. A burst of energy on April 22 and 23 may be short lived but will motivate you to start new projects. Do some reading or call a friend on the evening of April 26. You’ll feel frustrated by a critical person in authority on April 27 and 28.

Attn: Northwest BI Professionals – Register now for the next TDWI NW Chapter Meeting

 

Date: May 14, 2009

Time: 5:30–8:00PM, with billiards and networking to be held at the Parlor immediately following event

Location: Lincoln Square, 700 Bellevue Way NE, Bellevue, WA 98004 (see map below)


Lincoln Square

Speaker: Dave Wells, TDWI Research Director and Avid Conference Speaker

Customer Speaker: Vincent Ippolito, Washington Dental Services’ Director of BI

Topic: How to Deploy BI Programs in Time of Economic Hardship

Registration is free. Food is free to attendees. And best of all (unlike those other Data Organizations), you DONT have to be a member to attend, nor pay to attend even if you ARE NOT currently a member.

Space is extremely limited and advanced registration is recommended

Link to Register: http://1105media.inquisiteasp.com/cgi-bin/qwebcorporate.dll?P5RVKQ

TDWI NW Page: http://www.twdi.org/northwest

Strategic Business Intelligence in Times of Economic Turmoil

Ideas for business intelligence practitioners to forge ahead with their BI initiatives in times of economic turmoil – To pursue best-in-class business intelligence and data management without incurring the wrath of the monolithic centralized platforms built when times were marked by economic growth in most revenue bearing verticals – Can this same race hold its pace with its same velocity and momentum when the economy shifts winds against the runners? In this, larger than life uber-BI applications race, marked in the last 13 – 24 months by a high rate of BI mergers and acquisitions, one has to wonder what will happen when the dust settles and the acquired folks realize they are no more a part of the organization that was the little guy you bought out way back when. Imagine how ProClarity felt when giant Microsoft came a callin’ – Or, when SAP acquired BusinessObjects; did it become: B-I-C ERP meet B-I-C, well, BI.

What about the true R & D exploratory labs like Google Labs, who churns out some interesting advancements in the technology space, offering APIs and SDKs for free to the bold and daring willing to take them up on their offer (oh, and that’s no wilting flower of a number folks…Google had a cap of 10000 invitations when their App Store went live in the Spring of 2008. Some of the cool new data warehousing appliances or the process changes that came about with Master Data Management came about from the die harder open source fans who wanted to bring some structure to the masses without the cost of enterprise platforms with their clunky deployment paths and costly upgrades. Let’s not forget, the Adobe flash-frenzied dashboard users now introduced to the presentation layer worthy interactive dashboard gauges that made mouths salivate the first through third time viewing it… As was expected, vendors tried to update their applications to mirror such interactivity and integration with the MS Office stacks, though Xcelsius still corners the presentation layer market by far. (Open source has some cool contenders especially when it comes to data visualizations) as the race to the dashboard 2.0 space moves into the collaborative world of social networks.

No, there really is a Santa Claus, and I, too, am still smitten with PerformancePoint Planning !! PPS truly rocks the product placement in this arena, much harder to appeal to that broad category of stuffy Financial Budgeting and Planning CFOs, and the likes.

So I ask, with the downturn in the economy, can such advancements be made in, as clearly demonstrated, a capitalized business intelligence software industry , whose recent growth spurts marked a growing sense of entitlement yet with subpar execution and results upon implementation, where services and solutions costs drove positive spikes in software sales, and vice versa? In fact, an interdisciplinary and while highly debated, interdependency exists between the BI, social network, collaboration and portals with custom embedded BI apps, web services and more all geared with one goal in mind: to optimize in a cost effective manner in an effort to drive better, more data driven decision making ? Or is this another blue skies and apple pie dream manifested by one girl’s love for business intelligence

Enterprise Architecture Got You Down?

Try this simplified toolkit approach based on standards defined by the Federal Enterprise Architecture (FEA) board, along with NASCIO:

 

Performance Reference Model (PRM) •Inputs, outputs, and outcomes •Uniquely tailored performance indicators  

In this category, you should immediately think Scorecard (Balanced Scorecard and otherwise),:

–Each scorecard have 4-6 perspectives which are logical / categorical groupings of key indicators, or what I like to call ‘affinitized’ KPIs.

–Each perspective has < 6-7 KPIs (Key Performance Indicators) (if you receive pushback, and you will, as people would define a KPI for the percent of time the Express Grocery queue contains purchasers with more than the specified limit of 12 – 15 items, doll it up for the BBB as a complaint, and deliver it with such ferver one almost winces when they realize said complaint is recycled faster than their next trip to the grocery store

    –Remember the 3 keys to success with defining KPIs: are they actionable, are they measureable (not in a future state, but today can you measure it) and will they drive a performance based behavior change (think incentives, as they represent a perfect example of a performance based behavior changer).

– So, ask the question now… "That’s a long list, John or Jane Exec…First, what are you going to do with that information [the so-what question distinguishing actionable from interesting]…can you really drive performance improvements in your business with more than 5 indicators in the case where all 5 go red at the same time or if you don’t want to be direct, you can ask "how do those KPIs link to your performance objectives personally – Any leader worth their title will take the time to align the activities and work tasks that they personally, or through delegation, list in their performance reviews.

–This process can be started at any layer in an organizational hierarchy and is called Goals / Objectives Alignment or Cascaded KPIs

–Most leaders you ask have no more than a few KPIs that they ACTUALLY care about – just over the hurt feelings now, or feeling that you have wasted x number of hours measuring and reporting on metrics that top leadership doesn’t care about; if you have been in BI long enough, you will have experienced this at least once in your career.

 

Business Reference Model (SRM) • Lines of Business (functions and sub-functions) • Agencies, customers, partners

 

How will the performance KPIs cascade down to the individual contributor from the CEO’s goals? Easy – take this example:

CEO sets a goal of wanting to increase revenue by growing the Sales line of business, specifically new customer sales. He sets a goal of 35% growth of new sales revenue, which the VP of Sales is tasked to drive. They , in turn, assign the goal to the account managers in charge of new customer accounts who then add the same goal to their salesforce in the field. The KPI becomes New Sales Growth >= 35%, frequency is set to weekly with hierarchical rollups to monthly and quarterly aggregations.

 

–Now, you may ask yourself, what about the Operations department where Customer Sales and Service (aka Telesales) lives, and bingo! You’re getting it now…Even though it was tasked to the VP of Sales, they, or that same CEO, should have realized that the Telesales department can also generate revenue from new sales, just those that come through a different channel. Instead of the typical route of calculating in field sales and measuring the sales department for a goal of this nature, the Operations VP should have the same goal on their performance review as the VP of Sales, which they, in turn, delegate to their Telesales Center Managers, who delegate to the Supervisors who delegate it to the agents on the phone – While it is an implicit delegation as one who is hired to man a Telesales line understands their job is to answer the phone and make sales (thus, encouraging sales growth), it is still an action that is mandated by a supervisor, who received the mandate from their manager, who likely received the goal from the corporate office VP of Operations or person responsible for the Telesales center.

–It flows from top or bottom (vertically) as well as horizontally since in this example, it covers two horizontal business units (Sales and Operations).

–This is why starting with objectives or goals makes this process, that is, cascading KPIs, that much easier because you have a definitive starting point and end point which is that same objective/goal.

 

New Sales Growth >= 35% is the same whether you start with the call center agent whose awesome sales performance on the phones contributes to making her supervisor meet their goal which was to grow sales by 35% who enabled their manager who enabled the VP of Operations and the VP of Sales objectives who then met or exceeded their CEO’s original mandate.

 

Service Component Reference Model (SRM) • Service domains, service types • Business and service components

–A service component is defined as "a self contained business process or service with predetermined functionality that may be exposed through a business or technology interface.

 

Data Reference Model (DRM) • Business-focused data standardization • Cross-agency information exchanges

 

Technical Reference Model (TRM) • Service component interfaces, interoperability • Technologies, recommendations

Today I dub ‘Data Services Oriented Architecture’ for a Web 2.0 and beyond World

As David Besemer wrote in his May 2007 article for DMReview, ‘SOA for BI’, "It took Michelangelo nearly five years to complete his famous works at the Sistine Chapel. Your transition to SOA for BI can go much faster if you start with data services."

What are data services? According to Wikipedia, wait, there isnt an existing definition on Wikipedia. First, a definition with I share with the Internet users of the world vis-a-vie WikiPedia:

"A Data Services Oriented Architecture or ‘DSOA’ framework consists of a combination of schemas, classes and libraries that facilitate and provide the ability to create and consume data services for the web. DSOA reveals the consumer data underlying architecture, exposed using Data Services’ Entity Data Model, and provides reusability of your service when developed correctly," Laura Edell-Gibbons, Mantis Technology Group Inc.


portalnavbar
And by correctly, I would highly recommend not getting bogged down by the concept of plug and chug, dubbed by my colleague Tip, or making your code reusable. It is all a balance, remember, young BI padawan.

Select best-in-class data services middleware to help you model, develop and implement your back-end BI services. PowerDesigner is a pretty rocking modeling tool, which covers everything from data element impact analysis to facilitating requirements gathering. Simplistically speaking, I am a big fan of the simplicity of SQL Server Integration Services and the new Data Services, both Microsoft products, though this opinion is certainly one that doesnt necessarily represent the populas vote. I am a big fan of Infomatica and Data Integrator (now called Data services, funnily enough under the SAP/BusinessObjects brand).

During the first and second projects, be sure to track all productive working hours to deliver each phase of your solution and costs savings for the efficiencies I expect you designed your system with the unescapable expectation of being the ROI-generator, a widely accepted expectation that all BI systems have high ROI, and many due. Start small, grow enterprise once the concept has been leaned out and efficiencies expected and beyond are gained. Then, as you expand your deployment from project to enterprise, you can easily self-fund additional licenses and other required resources with the savings or other benefits gained on those 1st two, somewhat painful, ‘initiation’ projects. We all have to go through the process and while painful at times, the learning experiences offered outweigh any of the difficulties while learning.

It is better to build the new services project by project, always making the predecessor available to other projects in a unified data services tier as you go. You and your team can then choose whether to rhttp://scorecardstreet.spaces.live.com/mmm2008-11-07_18.20/#euse a data service, extend an existing service, buy or to build something from scratch, my least favorite BTW.

Over time, these will change and I suspect ‘reuse’ will become the greatest portion of the proverbial pie, whereas today, I believe the paradigm shifts more in the direction of ‘build from scratch.’

Starting to plan front-end BI services up front, even while deploying your backend BI services, will enable you to make small but meaningful steps without much noticable downtime to the organization, something especially important for those of us working with a ‘4 x 9s’ uptime SLA for our data centers. Plus, remember, if you build these on a powerful data services foundation, you will reduce your time to market, and your TCO over time. By providing the business with their much anticipated and needed operational reports, tactical and strategic dashboards and performance management analytics while infusing the lot into your SOA, one will reep rewards greater than my words could ever portray, dear reader…Til then, remember ‘what will come sooner than you think is no more when than how’.

References:

  • David Besener. "SOA for BI." DMReview, May 2007.
  • SWF Search-ability Announcement from Adobe and How It Relates to Xcelsius 2008

    Imagine Xcelsius dashboards especially built in 2008 with its flexible add-on component manager making it that much easier to customize components (think objects / widgets like scatterplots which are offered out of the box as a chart type)..

     

    Now, we have a best practice for monetizing the SWF content that is part of your Xcelsius 2008 dashboard…here is what Adobe had to say:

     

    Adobe is teaming up with search industry leaders to dramatically improve search results of dynamic web content and rich Internet applications (RIAs). Adobe is providing optimized Adobe Flash Player technology to Google and Yahoo! to enhance search engine indexing of the Flash file format (SWF) and uncover information that is currently undiscoverable by search engines. This will provide more relevant automatic search rankings of the millions of RIAs and other dynamic content that run in Adobe Flash Player. Moving forward, RIA developers and rich web content producers won’t need to amend existing and future content to make it searchable—they can now be confident that it can be found by users around the globe.

    Why is this news important?

    Adobe is working with Google and Yahoo! to enable one of the largest fundamental improvements in web search results by making the Flash file format (SWF) a first-class citizen in searchable web content. This will increase the accuracy of web search results by enabling top search engines to understand what’s inside of RIAs and other rich web content created with Adobe Flash technology and add that relevance back to the HTML page.

    Improved search of SWF content will provide immediate benefits to companies leveraging Adobe Flash software. Without additional changes to content, developers can continue to provide experiences that are possible only with Adobe Flash technology without the trade-off of a loss in search indexing. It will also positively affect the Search Engine Optimization community, which will develop best practices for building content and RIAs utilizing Adobe Flash technologies, and enhance the ability to find and monetize SWF content.

    Why is Adobe doing this?

    The openly published SWF specification describes the file format used to deliver rich applications and interactive content via Adobe Flash Player, which is installed on more than 98 percent of Internet-connected computers. Although search engines already index static text and links within SWF files, RIAs and dynamic web content have been generally difficult to fully expose to search engines because of their changing states—a problem also inherent in other RIA technologies.

    Until now it has been extremely challenging to search the millions of RIAs and dynamic content on the web, so we are leading the charge in improving search of content that runs in Adobe Flash Player. We are initially working with Google and Yahoo! to significantly improve search of this rich content on the web, and we intend to broaden the availability of this capability to benefit all content publishers, developers, and end users.

    Which versions of the SWF file format will benefit from this improved indexing and searching?

    This solution works with all existing SWF content, across all versions of the SWF file format.

    What do content owners and developers need to do to their SWF content to benefit from improved search results?

    Content owners and developers do not have to do anything to the millions of deployed SWF files to make them more searchable. Existing SWF content is now searchable using Google search, and in the future Yahoo! Search, dramatically improving the relevance of RIAs and rich media experiences that run in Adobe Flash Player. As with HTML content, best practices will emerge over time for creating SWF content that is more optimized for search engine rankings.

    What technology has Adobe contributed to this effort?

    Adobe has provided Flash Player technology to Google and Yahoo! that allows their search spiders to navigate through a live SWF application as if they were virtual users. The Flash Player technology, optimized for search spiders, runs a SWF file similarly to how the file would run in Adobe Flash Player in the browser, yet it returns all of the text and links that occur at any state of the application back to the search spider, which then appears in search results to the end user.

    How are Google and Yahoo! using the Adobe Flash technology?

    Google is using the Adobe Flash Player technology now and Yahoo! also expects to deliver improved web search capabilities for SWF applications in a future update to Yahoo! Search. Google uses the Adobe Flash Player technology to run SWF content for their search engines to crawl and provide the logic that chooses how to walk through a SWF. All of the extracted information is indexed for relevance according to Google and Yahoo!’s algorithms. The end result is SWF content adding to the searchable information of the web page that hosts the SWF content, thus giving users more information from the web to search through.

    When will the improved SWF searching solutions go live?

    Google has already begun to roll out Adobe Flash Player technology incorporated into its search engine. With Adobe’s help, Google can now better read the SWF content on sites, which will help users find more relevant information when conducting searches. As a result, millions of pre-existing RIAs and dynamic web experiences that utilize Adobe Flash technology, including content that loads at runtime, are immediately searchable without the need for companies and developers to alter it. Yahoo! is committed to supporting webmaster needs with plans to support searchable SWF and is working with Adobe to determine the best possible implementation.

    How will this announcement benefit the average user/consumers?

    Consumers will use industry leading search engines, Google now and Yahoo! Search in the future, exactly as they do today. Indexed SWF files will add more data to what the search engine knows about the page in which it’s embedded, which will open up more relevant content to users, and could cause pages to appear at a higher ranking level in applicable search results. As a result, millions of pre-existing rich media experiences created with Adobe Flash technology will be immediately searchable without the need for companies and developers to alter content.

    When will the new results register on Google?

    Google is using the optimized Adobe Flash Player technology now, so users will immediately see improved search results. As Google spiders index more SWF content, search results will continue to get better.

    How will this announcement benefit SWF content producers?

    Organizations can now dramatically improve the rich web experiences they deliver to customers and partners by increasing the use of Adobe Flash technology, which is no longer impeding the ability for users to find those experiences in highly relevant search results. RIA creators and other web content producers can now be confident that their rich media and RIA experiences leveraging Adobe Flash technology are fully searchable by users around the globe who use the dominant search engines. Furthermore, the ability to index information extracted throughout the various states of dynamic SWF applications reduces the need to produce an HTML or XHTML backup for the RIA site as a workaround for prior search limitations.

    Does this affect the searchability of video that runs in Adobe Flash Player?

    This initial rollout is to improve the search of dynamic text and links in rich content created with Adobe Flash technology. A SWF that has both video and text may be more easily found by improved SWF search.

    Will Adobe Flex applications now be more easily found by Google search, including those that access remote data?

    Yes, any type of SWF content including Adobe Flex applications and SWF created by Adobe Flash authoring will benefit from improved indexing and search results. The improved SWF search also includes the capability to load and access remote data like XML calls and loaded SWFs.

    Does Adobe recommend a specific process for deep-linking into a SWF RIA?

    Deep-linking, in the case of SWF content and RIAs, is when there is a direct link to a specific state of the application or rich content. A variety of solutions exist today that can be used for deep-linking SWF content and RIAs. It’s important that sites make use of deep links so that links coming into a site will drive relevance to the specific parts of an application.

    To generate URLs at runtime that reflect the specific state of SWF content or RIA, developers can use Adobe Flex components that will update the location bar of a browser window with the information that is needed to reconstruct the state of the application.

    For complex sites that have a finite number of entry points, you can highlight the specific URLs to a search spider using techniques such as site map XML files. Even for sites that use a single SWF, you can create multiple HTML files that provide different variables to the SWF and start your application at the correct subsection. By creating multiple entry points, you can get the benefits of a site that is indexed as a suite of pages but still only need to manage one copy of your application. For more information on deep-linking best practices, visit www.sitemaps.org/faq.php.

    Is Adobe planning on providing this capability to other search vendors too?

    Adobe wants to help make all SWF content more easily searchable. As we roll out the solution with Google and Yahoo!, we are also exploring ways to make the technology more broadly available.

    Where to go from here

    For for more information from Google on SWF search, read Improved Flash indexing on the Official Google Webmaster Central Blog.

    .NET vs. Java Consumer SDK – BusinessObjects Enterprise

    The Java and .NET versions of the consumer SDK are identical in functionality. The two versions of the SDK are generated from a common set of Web Service Definition Language (WSDL) files. As a result, they possess identical class names and inheritance patterns. There are differences between the two, however, that are addressed in this section.

    Note:    For more information on the Platform Web Services WSDL, see Using the WSDL instead of the consumer API.

    Organization of plugin classes

    It is the goal of this SDK to provide the same organizational structure of plugin classes as provided in the traditional, non-web services Enterprise SDK.

    In Java, classes are organized in packages where the name of the plugin is part of the package. For example, the CrystalReport class is located in the com.businessobjects.enterprise.crystalreport package, while the Folder class is located in the com.businessobjects.enterprise.folder package.

    In .NET, classes are organized in namespaces based on its plugin type. There are separate namespaces for destination, authentication, desktop, and encylopedia plugin classes. For example, both the CrystalReport and Folder classes are desktop plugins, so they are located in the BusinessObjects.DSWS.BIPlatform.Desktop namespace.

    There is also a separate namespace for system rights in .NET.

    Representation of class properties

    WSDL class properties are generated differently in Java and .NET. In Java, properties are generated as getX and setX methods, where X is the name of the property. In .NET, properties are generated as fields.

    In this guide, the term "property" refers to both the class method in Java and its field equivalent in .NET.

    Capitalization of method names

    In Java, method names begin with a lowercase character. In .NET, method names begin with an uppercase character.

    In this guide, the convention is to refer to a method by its Java case.


    Graphical Representation of the Process Towards Business Intelligence Enlightenment

    It all begins with an idea…whether an idea that came to you or one which is derived on a senior executive’s whim, all business intelligence initiatives start with a single thought: how can I drive more data into our decision making and business processes in order to drive better more accurate decisions for the business, thus enabling world class operations and growth potential. Whew – that was a mouthful.
    In all reality, let this graphical representation flow as organically as the thought I am trying to emphasize here – BI is a thought process and is a human relative need – So, we, as technologists need to start building software applications that meet that germanely simple conceptual need – to create software that not only improves my efficiencies at home or at work but that marries those efficiencies into human adaptive and behavioral neurological processes. When synapsis’ fire in ones brain, and neurological circuitry moves to pass one synapsis into another cortex from an origination point, one can visualize how to tie this metaphor into the systems we use everyday – Take the process of searching the Internet using your favorite search engine. We enter key words or metadata tags that represent nouns, verbs or contextual fragments that represent natural neural processing of the human brain. If search engines were constructed, likewise, BI systems, to more mirror this reflexive neural network within their enterprise application architecture, one might find more usefulness in the long run in terms of end user adoption and sustainment of said adoption after the 1st year after implementation. Let this pictorial represent that behavior marriage between BI technologies and human neural networks.

    Increasing Business Value vs. Insights Provided – a Business Intelligence Roadmap

     

    While many companies feel they have strong BI programs, most, in my experience, have operational reporting systems and sometimes, if you are lucky, they also have strategies that go with those systems or even better, are driving those systems (fueled by requirements gleaned from business needs and actual usage scenarios vs. the "way it has always been done/reported on").

    As you can see in figure 1, that level merely tells you the ‘WHAT’ – it doesn’t answer the ‘WHY’ it happened (root cause), or predict the ‘HOW’ it might affect you in future, nor the ‘WHEN’ in terms of monitoring if it is happening currently or just a past event.

     (img source TDWI Research at http://tdwi.org, 2007)

    Your BI roadmap should have a similar long term plan. If you want to provide increasing value to your organization, one must get out of the business of operational reporting and move towards the real meat and value of BI, which lay in the analysis, monitoring and predicting in terms of how the business views their needs from BI, not how BI believes they can deliver information.

    Capturing Metrics and Their Measurements – A Data Quality Perspective

     

    The techniques that exist within the organization for collecting, presenting, and validating metrics must be evaluated in preparation for automating selected repeatable processes.

    Cataloging existing measurements and qualifying their relevance helps to filter out processes that do not provide business value as well as reducing potential duplication of effort in measuring and monitoring of critical data quality metrics. Surviving

    measurements of relevant metrics are to be collected and presented in a hierarchical manner within a scorecard, reflecting the ways that individual metrics roll up into higher level characterizations of compliance with expectations while allowing for drill-down to isolate the source of specific issues.

    As is shown in Figure 1, collecting the measurements for a data quality scorecard would incorporate:

    1. Standardizing business processes for automatically populating selected metrics into a common repository

    2. Collecting requirements for an appropriate level of design for a data model for capturing data quality metrics

    3. Standardizing a reporting template for reporting and presenting data quality metrics

    4. Automating the extraction of metric data from the repository

    5. Automating the population of the reporting and presentation template, or a data quality scorecard

     

    Figure 1:

    Dimensional World – Understanding Modeling Techniques and Approaches

     

    As a data architect, I am often amazed at how many with the same title really do not understand the core differences between dimensional model structures in my industry.

    In fact, it is a rarity to find the data architect willing to be challenged without personalities coming into the mix. As as we all know, when you fight the person and not the problem, you end up with hurt feelings and resentments in the workplace. For an EDW, this can lead to people resigned and not willing to stand up for what they think, which leads to the ‘sheep’-like syndrome called ‘Group Think’, thus resulting in an EDW that is built off of emotion and not educated beliefs or best practices.

    I thought I would take the time to explain some of the differences between dimensions to enable you, reader, to stand up for the correct approach when you are faced with a contended data model designed by you.

    Remember, they are not attacking you, just the model. Look at it as a chance to grow and learn from others. Maybe, just maybe, having 4 or 6 eyes is better than just your 2 and your model, already attributed to you from a recognition perspective, will be that much better.

    Here we go:

    Slowly Changing Dimensions (SCD – of the Type II variety is where I become a Type II gal)

    • What in the world is a SCD anyway?
    • Ralph Kimball defined this his 1996 book as the following:

    (Kimball, 1996), a slowly changing dimension is a dimension table in which a new row is created each time the underlying component entity changes some important characteristic. Its purpose is to record the state of the dimension entity at the time each transaction took place.

    This concept is hard for for those who have primarily dealt with changes as handled in operational systems: no matter how a customer changes, we want to ensure that we have only one customer row in the customer table.

    Thus each row in a slowly changing dimension does not correspond to a different entity but a different “state” of that entity—a “snapshot” of the entity at a point in time.

    To create a slowly changing dimension table, the following design steps are required:

     

    • Define which attributes of the dimension entity need to be tracked over time. This defines the conditions for creating each new dimensional instance.
    • Generalize the key of the dimensional table to enable tracking of state changes. Usually this involves adding a version number to the original key of the dimension table.

    Apart from the generalized key, the structure of the slowly changing dimension is the same as the original dimension. However, insertion and update processes for the table will need to be modified significantly.

    Splitting Dimensions: “Tiny-Dimensions”
    In practice, dimension tables often consist of millions of rows, making them unmanageable for browsing purposes. To address this issue, the most heavily used attributes (e.g., demographic fields for customer dimensions) may be separated into a mini-dimension table. This can improve performance significantly for the most common queries. The mini-dimension should contain a subset of attributes that can be efficiently browsed. As a rule of thumb, there should be fewer than 100,000 combinations of attribute values in a mini-dimension (i.e., fewer than 100,000 rows) to facilitate efficient browsing (Kimball, 1996). The number of attribute value combinations in a tiny dimension can be limited by:

    • Including attributes in the mini-dimension that have discrete values (i.e., whose underlying domains consist of a fixed set of values).
    • Grouping continuously valued attributes into “bands.” For example, age could be converted to a set of discrete ranges such as child (0-17), young (18-29), adult (30-45), mature (45-64), and senior (65+).

     Figure 4. Mini-Dimensional Table (Detailed Level Design)

    Figure 4 shows how customer demographics in the Order Item star could be stratified out into a mini-dimension table. Some of the attributes in the original table have been transformed to reduce the number of rows in the mini-dimension table: 

  • # Employees and Revenue have been converted to ranges.

  • Date of First Order has been converted to Years of Service, reducing the number of combinations to years’ multiples rather than all dates combos that are possible.

    Rule of Thumb:

    Rather than create a lot of very small dimension tables, these may be combined into a single dimension, with each row representing a valid combination of values. As a rule of thumb, there should be no more than seven dimensions in each star schema to ensure that it is cognitively manageable in size (following the “seven, plus or minus two” principle).

    Dealing with Non-Hierarchical Data

    A major source of complexity in dimensional modeling is dealing with non-hierarchically structured data.

    Dimensional models assume an underlying hierarchical structure and therefore exclude data that is naturally non-hierarchical.

    So what do we do if important decision-making data is stored in the form of many-to-many relationships? In this section, we describe how to handle some particular types of non-hierarchical structures that commonly occur in practice:

    1. Many-to-many relationships: these define network structures among entities, and cause major headaches in dimensional modeling because they occur so frequently in ER models.
    2. Recursive relationships: these represent “hidden” hierarchies, in which the levels of the hierarchy are represented in data instances rather than the data structure.
    3. Generalization hierarchies: subtypes and supertypes require special handling in dimensional modeling, because of the issue of optional attributes. These represent hierarchies at the meta-data level only—data instances are not hierarchically related to each other so they cannot be treated as hierarchies for dimensional modeling purposes.

    First, #1:

    1. Many-To-Many Relationships
      Many-to-many relationships cause major headaches in dimensional modeling for two reasons. Firstly, they define network structures and therefore do not fit the hierarchical structure of a dimensional model. Secondly, they occur very commonly in practice. Here we consider three types of many-to-many relationships which commonly occur in practice:
      • Time-dependent (history) relationships
      • Generic (multiple role) relationships
      • Multi-valued dependencies (“true” many-tomany relationships)
  • To include such relationships in a dimensional model generally requires converting them to one-to-many (hierarchical) relationships.

    Type 1. Time-Dependent (Historical) Relationships
    A special type of many-to-many relationship that occurs commonly in data warehousing applications is one which records the history of a single-valued relationship or attribute over time. That is, the attribute or relationship has only one value at a specific point in time, but has multiple values over time. For example, suppose that the history of employee positions is maintained in the example data model. As shown in Figure 5, the many-to-many relationship which results (Employee Position History) breaks the hierarchical chain and Position Type can no longer be collapsed into its associated component entity (Employee).

    Figure 5. Time-Dependent (Historical) Relationship

    There are two ways to handle this situation:

    • Ignore history: Convert the historical relationship to a “point in time” relationship, which records the current value of the relationship or attribute. In the example, this would mean converting Employee Position History to a one-to-many relationship between Employee and Position Type, which records the employee’s current position. Position Type can then be collapsed into the Employee entity, and the Employee dimension will record the employee’s current position (i.e., at the time of the query). The history of previous positions (including their position at the time of the order) will be lost. A disadvantage of this solution is that it may result in (apparently) inconsistent results to queries about past events.
    • Slowly changing dimension: Define Employee as a slowly changing dimension and create a new instance in the Employee dimension table when an employee changes position. This means that Position Type becomes single valued with respect to Employee, since each instance in the Employee table now represents a snapshot at a point in time, and an employee can only have one position at a point in time. Position Type can again be collapsed into the Employee dimension. The difference between this and the previous solution is that the position recorded in the dimension table is the employee’s position at the time of the order, not at the time of the query.

    Type 2. Generic (Multiple Role) Relationships
    Another situation that commonly occurs in practice is when a many-to-many relationship is used to represent a fixed number of different types of relationships between the same two entities. These correspond to different roles that an entity may play in relation to another entity. For example, in the example data model, an employee may have a number of possible roles in an order:

    • They may receive the order
    • They may approve the order
    • They may dispatch the order

    This may be represented in an ER model as a many-to-many relationship with a “role type” entity used to distinguish between the different types of relationships (Figure 6).

    Figure 6. Generic (Multiple Role) Relationship converted to Specific Relationships

    The intersection entity between Employee and Order means that Employee cannot be considered as a component of the Order transaction, and therefore orders cannot be analyzed by employee characteristics. To convert such a structure to dimensional form, the different types of relationships or roles represented by the generic relationship need to be factored out into separate one-to-many relationships (Figure 6). Once this is done, Employee becomes a component of the Order transaction, and can form a dimension in the resulting star schema.

    Type 3. Multi-Valued Dependency (True Many-To-Many Relationship)
    The final type of many-to-many relationship is when a true multi-valued dependency (MVD) exists between two entities: that is, when many entities of one type can be associated in the same type of relationship with many entities of another type at a single point in time. For example, in Figure 7, each customer may be involved in multiple industries. The intersection entity Customer Industry “breaks” the hierarchical chain and the industry hierarchy cannot be collapsed into the Customer component entity.

    Figure 7. Multi-Valued Dependency (MVD)

    One way of handling this is to convert the Customer Industry relationship to a 1:M relationship, by identifying the main or “principal” industry for each customer.

    While each customer may be involved in many industries, there will generally be one industry in which they are primarily involved (e.g., earn most of their revenue). This converts the relationship into a one-to-many relationship, which means it can then be collapsed into the Customer table (Figure 8). Manual conversion effort may be required to identify which is the main industry if this is not recorded in the underlying production system.

    Figure 8. Conversion of Many-To-Many Relationships to One-To-Many Recursive Relationships

     

     

    Second, #2 from above:


  • Recursive Relationships

    Hierarchies are commonly represented in ER models using recursive relationships. Using a recursive relationship, the levels of the hierarchy are represented as data instances rather than as entities. More flexible for sure…However, such structures are less useful in a data warehousing environment, as they reduce understandability to end users and increase complexity of queries.

    In converting an ER model to dimensional form, recursive relationships must be converted to explicit hierarchies, with each level shown as a separate entity. To convert this to dimensional form, each row in the Industry Classification Type entity becomes a separate entity. Once this is done, the levels of the hierarchy (which become classification entities) can be easily collapsed to form dimensions.

    For example, the industry classification hierarchy in the example data model may be shown in ER form as a recursive relationship (Figure 9).

    Figure 9. Conversion of Recursive Relationship to Explicit Hierarchy

     

     

     

    Lastly, #3 from above:


  • Generalization Hierarchies: Subtypes and Supertypes
    In the simplest case, supertype/subtype relationships can be converted to dimensional form by merging the subtypes into the supertype and creating a “type” entity to distinguish between them. This can then be converted to a dimensional model in a straightforward manner as it forms a simple (two level) hierarchy. This will result in a dimension table with optional attributes for each subtype. This is the recommended approach when there are a relatively small number of subtype specific attributes and/or relationships. In the more complex case—when there are many subtype-specific attributes and when different transaction-entity attributes are applicable for different subtypes—separate dimensional models may need to be created for the supertype and each of the subtypes. These are called heterogeneous star schemas, and are the dimensional equivalent of subtypes and supertypes. In general, this will result in n+1 star schemas, where n is the number of subtypes (see Figure 10):

    Figure 10. Heterogeneous Star Schemas ("Dimensional Subtyping")

     

    • One Core Star Schema (“dimensional supertype”): This consists of a core fact table, a core dimension table plus other (non-subtyped) dimension tables. The core dimension table will contain all attributes of the supertype, while the core fact table will contain transaction attributes (facts) that are common to all subtypes.
    • Multiple Custom Star Schemas (“dimensional subtypes”): A separate Customer Star Schema should be optionally created for each subtype in the underlying ER model. Each custom star schema will consist of a custom fact table, a custom dimension table plus other (non-subtyped) dimension tables.
    • Each custom dimension table will contain all common attributes plus attributes specific to that subtype. The custom fact table will contain all common facts plus facts applicable only to that subtype.
  • Dimensional Structures: A Look Into the Outliers that Often Get Overlooked When Designing Data Structures

     

    Think Star Schema is the only way to go with your EDW? Think again…StarFlake is a Best in Breed Approach that Might Make BI on top of the EDW that much more capable and robust.  Read on… I will cover the following concepts:

     

  • Alternative dimensional structures: snowflake schemas and starflake schemas
  • Slowly changing dimensions
  • Mini-dimensions
  • Heterogeneous star schemas (dimensional subtypes)
  • Dealing with non–hierarchically structured data in the underlying ER model:
    – Many-to-many relationships
    – Recursive relationships
    – Subtypes and supertypes

    Figure 1: Snowflake (the anti-lord of all things good in the data world)

    Kimball (1996) argues that “snowflaking” is undesirable because it adds unnecessary complexity, reduces query performance, and doesn’t substantially reduce storage space.

    However, performance impact of snowflaking depends on the DBMS / OLAP structures in place.  Advantages of the snowflake : explicitly shows the hierarchical structure of each dimension, which can help in understanding how the data can be analyzed.

    Thus, whether a snowflake or a star schema is used at the physical level, views should be used to enable the user to see the structure of each dimension as required.

    Being a star schema girl myself, I am quite interested in the power of the starflake model and thus being a starflake girl <she is reminded of the Tori Amos song ‘Cornflake Girl’ with this reference>.

     

    Figure 2: Starflake model

    A starflake schema is a star schema in which shared hierarchical segments are separated out into sub-dimension tables. These represent “highest common factors” between dimensions. A starflake schema is formed by collapsing classification entities from the top of each hierarchy until they reach either a branch entity or a component entity. If a branch entity is reached, a subdimension table is formed. Collapsing then begins again after the branch entity. When a component entity is reached, a dimension table is formed.

     

     

    Dimensional Design Trade-Offs

    The alternative dimensional structures considered here represent different trade-offs between complexity and redundancy:

    • The star schema is the simplest structure, as contains the least number of tables—eight tables in the example. However, it also has the highest level of data redundancy, as the dimension tables all violate third normal form (3NF). This maximizes understanding and simplifies queries to pairwise joins between tables.
    • The snowflake schema has the most complex structure and consists of more than five times many tables as the star schema representation (41 in the example). This will require multitable joins to satisfy queries.
    • The starflake schema has a slightly more complex structure than the star schema—nine tables in the example. However, while it has redundancy within each table (the dimension tables and sub-dimension tables all violate 3NF), redundancy between dimensions is eliminated, thus reducing the possibility of inconsistency between them. All of these structures are semantically equivalent: they all contain the same data and support the same set of queries. As a result, views may be used to construct any of these structures from any other.
  •  

    Bottom’s Up Data Mart and Enterprise Data Warehouse Project Implementation Plan

    SUMMARY: In my consulting practice, I recommend an incremental, ‘bottom-up’ implementation methodology, similar to that advocated by Ralph Kimball. This has proven to be a successful deployment approach to ensure a short-term return on investment with minimal project risk, while still delivering a data warehousing architecture that provides a standardized, enterprise wide view of information. This column describes a typical project plan based on a bottom-up implementation methodology.

    * REQUIREMENT FOR BOTTOM-UP DEVELOPMENT

    I receive numerous e-mails from both technical and business managers who have become extremely frustrated by the inordinate amount of time and effort required to build data warehouses using traditional, top-down development techniques. Due to the large amount of effort required up front to perform user interviews and enterprise data modeling, organizations using top-down techniques often wait 12 to 14 months to get any return on investment. In some cases, managers complain that they have worked with a systems integrator on a top-down development project for over two years without obtaining any solution to their business problem.

    As described in previous articles, a bottom-up development approach directly addresses the need for a rapid solution of the business problem, at low cost and low risk. A typical requirement is to develop an operational data mart for a specific business area in 90 days, and develop subsequent data marts in 60 to 90 days each. The bottom-up approach meets these requirements without compromising the technical integrity of the data warehousing solution. Data marts are constructed within a long-term enterprise data warehousing architecture, and the development effort is strictly controlled through the use of logical data modeling techniques and integration of all components of the architecture with central meta data.

    * PROJECT PLAN

    The representative project plan described below is based on an incremental, ‘bottom-up’ implementation methodology. In my experience, this has been the most successful deployment approach to ensure a short-term return on investment with minimal project risk, while still delivering a data warehousing architecture that provides a standardized, enterprise-wide view of information.

    The project plan breaks down into three major phases, as follows:

    1. Data Warehouse Requirements and Architecture – gather user requirements through a series of requirements interviews, assess the current IT architecture and infrastructure, along with current and long-range reporting requirements, and develop an enterprise data warehousing framework that will provide maximum flexibility, generality and responsiveness to change. Define functional requirements for the initial data mart within the enterprise architecture. Optionally, conduct a Proof-of-Concept demonstration to select enterprise-class ETL and/or BI tools.

    2. Implement Initial Data Mart under the New Architecture – prove the new architecture works by implementing a fully operational data mart for a selected subject area within a 90-day time box.

    3. Develop Additional Data Marts under the New Architecture – on a phased basis, develop additional architected data marts within the new framework.

    Timeboxes are placed around each phase of the project plan to strictly limit the duration of the development effort. Phase 1 (specification of architecture and functional requirements) is limited to four weeks. If proof-of-concept testing of ETL and BI tool is conducted, these tests occur outside the timebox for Phase 1, due to the uncertain timing of scheduling vendors to perform the tests. Phase 2 (development of initial data mart) is limited to 90 days. Phase 3 (development of additional data marts) is limited to 60 to 90 days for each subsequent data mart.

    * TASKS AND DELIVERABLES BY PHASE

    The list below summarizes representative tasks, timeframe, and deliverables for each of these three phases.

    **Phase / Task
    1. Requirements and Architecture
    —Conduct workshop to review/define strategic business drivers, review current application architecture, and define strategic data warehousing architecture.
    —Elapsed Week :1
    —Deliverables: strategic requirements, document specifying the recommended long-term enterprise data warehousing architecture

    —Conduct requirements interviews with personnel from multiple subject areas and document the results of the interviews.
    —Elapsed Week: 1
    —Deliverables: Interview summaries, requirements inventory

    —Conduct workshop to define functional specifications of initial data mart,
    —Elapsed Weeks: 2
    —Deliverables: scope statement, reporting and analysis specifications

    —Develop high-level dimensional data model for initial data mart
    —Elapsed Weeks: 2
    —Deliverables: logical data model

    —Conduct workshop to prepare for a Proof-of-Concept test of leading ETL tool(s)
    —Elapsed Weeks: 3
    —Deliverables: specifications for the ETL POC

    —Conduct workshop to prepare for a Proof-of-Concept test of leading BI tool(s)
    —Elapsed Weeks: 3
    —Deliverables: Specifications for the BI POC

    —Conduct workshop to develop Phase 2 project plan
    —Elapsed Weeks: 4
    —Deliverables: Phase 2 project plan

    **Phase / Task
    2. Implement Initial Data Mart in 90-Day Time-Box,
    —Elapsed Weeks: 12
    —Deliverables: live architected data mart, live OLAP cubes and reports, central meta data repository, local meta data repository, reusable ETL objects and conformed dimensions

    **Phase / Task
    3. Implement Additional Data Marts
    —Elapsed Weeks: 8-12 each
    —Deliverables: Additional architected data marts

    * PHASE 1 IMPLEMENTATION

    The initial task in Phase 1 is to conduct two on-site workshops, limited to 1 to 2 days each. The function of the first workshop is to bring all members of the development team up-to-speed on alternative enterprise data warehousing architectures and ‘best practices’ in data warehousing. The second workshop is used to achieve consensus on the specification of an enterprise data warehousing architecture capable of meeting the long-term business requirements of the organization.

    Interviews with personnel from multiple subject areas are held to define high-level functional requirements for each subject area. Subject areas often correlate with proposed data marts, in areas such as finance, sales, marketing, HR, supply-chain management, customer touchpoints, etc. As described in a previous column, the interviews are kept deliberately short (one day or less per subject area). The deliverables from each interview include a short, concise requirement specification for the subject area and a top-level dimensional data model representing the data sources, source-to-target mappings, target database, and reports required for a specific subject area. The top-level data models from all subject areas are then synthesized to identify common data sources, conformed dimensions and facts, common transformations and aggregates, etc.

    Following synthesis of functional requirements from all subject areas, a workshop is held to define the functional requirements for the initial data mart, lay out the project plan for the development of the initial data mart, identify required skill sets, and personnel assignments to the project.

    The next task is to define a high-level dimensional data model for the initial data mart, representing an expansion of the model prepared as part of the user interview process.

    If the organization has decided to conduct a proof-of-concept test of competitive ETL tools and BI tools, the next two tasks represent preparation of functional specifications for the tests. Proof-of-concept testing may be required if multiple ETL or BI tools are being evaluated and it is politically expedient to conduct rigorous testing of competitive products prior to making a selection. I have found that an intensive two-day workshop is sufficient to specify the functionality of the tests to be conducted for a proof of concept for either an ETL or BI tool.

    The final task in Phase 1 is to specify a detailed project plan for the implementation of the initial data mart.

    * PHASE 2 IMPLEMENTATION

    In the recommended bottom-up development methodology, the process of implementing the initial data mart is limited to 90 calendar days. Although 90 days is arbitrary, it fits the needs of business managers for a rapid solution of the business problem and meets the needs of CFOs for a 90-day Return-On-Investment. The 90-day timebox starts on the day that the ETL tool, the target DBMS, and the BI tool are successfully installed. To meet the challenge of a 90-day implementation process, utilization of an ETL tool, rather than hand-coding the extractions and transformations, is strongly recommended.

    Implementation of the initial and subsequent data marts is ideally performed by a small team of data warehousing professionals, typically consisting of a data modeling specialist, an ETL specialist, and a BI tool specialist. In my own organization, I emphasize cross-training of personnel, which minimizes the number of personnel that need to be assigned to a project and maximizes their efficiency.

    The first task for the development team is to design the target data base for the initial data mart. Modeling of the target data base for the initial data mart proceeds through three steps: design of an entity-relationship diagram, then a logical dimensional model, and finally a physical model of the target database schema. Although the E-R diagram is not required for the initial data mart, it is required for subsequent data marts to ensure that all physical data models for multiple data marts are derived from a common logical specification.

    The next major task is to specify and implement data mapping, extraction, transformation, and data cleansing rules. The data mapping and transformation rules are defined first in natural language, and then implemented using only the transformation objects supplied with the ETL tool. The objective is to avoid coding any ETL processes. It is hard to predict how long this task will take. For many applications, specification and implementation of the transformation rules should not take more than 3 to 4 weeks. However, some applications may require many months to specify and implement complex transformations. These applications are not likely to fit within a 90-day implementation timebox.

    In parallel with implementation of the transformation logic, developers build aggregation, summarization, partition, and distribution functions. The ETL tool may be used to compute aggregates in one pass of the source data, using incremental aggregation techniques. Pre-computed aggregates are recommended for most data warehousing applications to provide rapid response to predictable queries, reports, and analysis.

    The final task in Phase 2 is delivery of a fully operational, architected data mart for the initial subject area, using an exact subset of the enterprise data warehousing architecture. Ideally, the development team delivers all of the functionality that was specified at the beginning of the 90-day timebox. However, the team is permitted to defer low priority functions in order to make the 90-day timebox. The team is self-managed and organizes its resources to deliver as much functionality as possible within the 90-day development window.

    * PHASE 3 IMPLEMENTATION

    The objective of Phase 3 is to build additional architected data marts. Additional data marts are built by the primary development team using common templates and components, such as conformed dimensions and facts, common transformation objects, data models, central meta data definitions, etc.

    In the bottom-up methodology, a central data warehouse, an operational data store, and a persistent staging file are optional. Data marts are typically developed using a data warehousing backplane schema design, as described by Ralph Kimball in his book ‘The Data Warehouse Toolkit, Second Edition.’ There may be good technical reasons to incorporate a central data warehouse and an operational data store in the enterprise data warehousing architecture. However, in the bottom-up methodology, development of the central data warehouse and ODS are deferred until they are clearly required. A central data warehouse is often required and when detailed, atomic data from multiple data marts must be accessed to generate cross-business reports, and when there is a low percentage of conformed dimensions across the data marts.

    Maintenance and administration of the data warehousing application is an ongoing function. A secondary team may be used to enhance and maintain completed data marts. The primary team transfers transformation templates, data models, conformed dimensions, conformed facts, meta data, etc. to the secondary team to simplify the enhancement and administration of completed data marts.

    My organization has used this methodology successfully for many clients and has a good deal of experience with it. However, successful implementation of the methodology depends on several critical success factors:
    -a dedicated implementation team;
    -consulting help from an experienced organization at the beginning of the project;
    -backing of a business manager who is hungry for a solution to a painful business problem; and
    -integration of all components of the architecture with central meta data.

    Guerilla Marketing Meet Business Intelligence; The Result? Social BI Networks or BI 2.0

    Casual users who become adopters compound the “viral” tipping point of the “usage effect”.  When a system eases the user’s experience, rather than the reverse that never leads nowhere good <insert the person who best fits ‘I deleted it from the registry with a shrug’ who inadvertently deletes their O/S here>, why not try to these stages to gain a better understanding of your BI efficacy from your users.

     

    I dubbed this measurement technique 3XEI (in order) : educate, illustrate, elaborate, integrate, extrapolate, iterate. The follow down process is as defined as follows:

    1. Seek first to understand, then to be understood; EDUCATE yourself on the pain points from the eyes of the customer (“Voice of the Customer”),
    2. ILLUSTRATE to prior adopters, detractors and neutralities the benefits of BI by mapping out or illustrating a visualization map of their current process circling all pain points in the process.
    3. In order your dependence on IT to support business-driven application acquisitions, you should continue to interview candidates, walk the process and mine as much as you can in order to ELABORATE on the process map until you’ve captured all process metrics.
    4. INTEGRATE your approach to educating, illustrating and elaborating on your BI program, you will gain end-to-end insights previously obscured by not factoring in the multi co-linearity of inputs on each other as well as the intended output.
    5. EXTRAPOLATE to pain points and use information to drive decisions or changes to the process at large.
    6. ITERATE on the illustrated processes as extrapolation exercise dictates or requires in order to remain vital and relevant to the organization.

    These phases have no definitive start/stop points. They are as mutable as the changing technology landscape in corporations today. They are tightly embedded within for the life of the process at the broad organizational level. Very few companies understand the power of mapping all efforts to the core business process supported, nor can they visualize the power of the optimized, end-to-end process on driving the bottom line. Those that do, reap the rewards; and those that don’t, well, they need not be mentioned.