Salesforce’s analytics strategy is to start simple and grow from there.  Donovan counted dashboards and reports as analytics (in contrast to others who define analytics as what you have left when you subtract ETL and reporting from BI). He noted that Salesforce users have built three quarters of a million dashboards and dashboards are viewed 700,000 times per day.  There are over 12 million reports and 2.5 million reports are run each day.  In addition to dashboards and reports, Donovan included list views and search in the collection of analytics tools currently available.  Future additions to analytics will build from this collection instead of adding relatively high end analytics.

Donovan showed a nice architecture slide contrasting the traditional datawarehouse based approach to the new cloud-based approach taken by Salesforce and claimed that the datawarehouse approach is out of date, too complicated, and too rigid.  Salesforce’s new approach takes advantage of their cloud-based force.com API, multitenant architecture and platform so it is easy to use, realtime, and flexible.

Donovan gave a brief review of Salesforce’s multi-tenant architecture. Craig Weissman, Salesforce’s CTO, gave a presentation on the multi-tenant architecture for SDForum’s Software Architecture SIG back when I helped organize that group. Click here for more on this topic.

Salesforce achieves realtime analytics and simplification by not having a datawarehouse.  But unlike many companies using NoSQL data stores Salesforce does use a traditional RDBMS: Oracle. Currently, there are around fifteen databases for production each containing around ten terabytes. Salesforce relies on ACID transactions because they are supporting companies doing business on their platform.  When transactions are committed, the changes are available immediately everywhere and this supports the goal of realtime analytics.

Interestingly, Salesforce does not cache results of queries and so on because things change so frequently.  Salesforce is able to get away without caching and does without a datawarehouse in part by making query execution efficient.  Donovan went into some depth on how queries are optimized.  Salesforce has its own optimizer and does its own indexing in part because the multi-tenant architecture of their data often makes it impossible to directly use Oracle’s indexing and optimization.

In closing, Donovan said his top priorities now are adding analytical capabilities, improving scalability and usability, and supporting collaboration.  You can try out analytics for free as it is available in Salesforce’s sandbox along with a new report builder.  Donovan’s slides are available here and at SDForum.org.

Agility was a key theme of the presentation and part of James’ approach is intended to support quick actions. When an event occurs, in some cases it is important to notice quickly and act almost immediately as the value of action based on the information about the event decreases rapidly as time passes. With other events, you have more time to consider options. But the key is to notice and act appropriately in a timely manner. It isn’t enough just to make information available or notice things quickly or even in real time. Sometimes noticing immediately isn’t even necessary or useful (e.g., when the required action would take a long time anyway).

James uses a three stage approach to improving operational decisions:

1. discover the important decisions
2. build (automate) decision services
3. analyze decisions and create a closed loop between analytics and decision services.

The slides for James’s presentation are available at the BI SIG’s web page at http://SDForum.org/BISIG and here.

Over 250 people attended. Some leading analytics gurus and data scientists gave presentations and participated in panel discussions.  This blog post links to summaries of the keynotes and panels.

There were four panel discussions at the conference:

• Competing on Analytics
• Analyzing Big Data
• New Frontiers
• The Investor Perspective

Keynote speakers representing an array of large companies with strong analytics capabilities gave presentations on a wide range of analytics efforts:

• Ronny Kohavi (Microsoft) “Online Controlled Experiments:  Listening to the Customers, Not to the HiPPO“
• Sanjay Poonen (SAP) “Leading the Analytics Revolution“
• Peter Norvig (Google) “The Unreasonable Effectiveness of Data“
• Jeff Kreulen (IBM) “Analytics:  An Applied Researcher’s Perspective“
• Jaap Suermondt (HP) “Research in Analytics for Operational Impact at HP“

You can click on the links above to get quick summaries.  The keynote presentations and panel discussions were recorded and the audio and video recordings are available individually from the pages above or at dynno.com or at SDForum here.  The conference was sponsored by Accel Partners, Dynno, IBM, Impetus, Microsoft and SAP and the organizing committee included Stacey Bishop (Scale Venture Partners), Jim Claussen (IBM), Lars Leckie (Hummer Winblad), Sonja London (Summit Software), Paul O’Rorke (Kraken Data), and Julia Queck (SAP).

David began by asking:  how do the panelists use analytics to get a competitive advantage?  What is an example of something they found that surprised them?

Ken Rudin and Zynga use analytics to achieve their goals of increasing revenue, improving user retention and increasing viral spread for their online social games including Farmville.  They collect 3-4 terabytes of data from their 40-50 million daily users.  Initially they were reactive:  producing reports in response to requests.  Now they use analytics as an integral part of game design:  they use AB testing and experiments in development just as QA is an integral part of development.  Analysts are part of the design team.  An example of something that surprised them occurred when they analyzed Mafia game players.  There are two groups of players, the “crime jobbers” and the “fighters.”  They discovered that fighters spend twice as much because they are trying to compete with their friends so they purchase weapons to arm their mafia members so they can fight and defeat their friend’s mafia gang.  Since they discovered this, they have changed the game to encourage players to fight more instead of just doing crime jobs.

Kevin Weil mentioned that a key accomplishment of the analytics team at Twitter has been to get everyone to consider data (and to make it possible for them to do so) in all important decisions.  He described an example of how analytics surprised them that involved the social network underlying Twitter.  Many people use Twitter largely as an information source and their network of “following” links says something about their interests.  But the bidirectional links can be ignored when trying to determine their interests.  The unidirectional links (e.g., the people they follow who don’t follow them) are the ones that carry the most important incoming information.

David Steier asked the panelists:  How do you organize people to achieve the company’s goals?  The panelists companies all have analytics teams and a team responsible for the company’s analytics platform and analysts who work with other teams but the way they work within the companies is different and several companies are adapting innovative approaches.

DJ Patil and LinkedIn started out by looking at other companies such as Google and Yahoo.  Yahoo had analytics in a separate research organization and it was difficult or impossible to get it into products.  Google is driven by technology and bolts products on top (see also http://techcrunch.com/2010/05/15/facebook-google/).  So to ensure that analytics is integrated into products at LinkedIn, the Analytics team is a substantial part (1/4th) of the product team and has its own designers and developers so it is easier to go straight to production or to integrate with existing products.

Neel Sundaresan (eBay) claimed that “everybody should be an analytics scientist.”  eBay has an analytics platform team that provides data to the rest of the company and “the data tells you what the product should be.”  With 200 million users and a billion searches per day, eBay gets tremendous amounts of data and product managers and developers and even some machine learning scientists need to learn to look at the data.

Ken Rudin (Zynga) argued that the whole idea of having an “analytics team” is flawed.  He asked:  ”Does Microsoft have an internet division?”  Like the internet, analytics is or should be fundamental to everything in the company.  So his goal is to work himself out of a job by putting analysts in development and product teams and by training almost everyone in the company in analytics starting with product managers, then engineers, and then quality assurance.

In summary, one of the key themes of the panel was that companies competing on analytics are finding innovative ways to integrate analytics throughout their companies.  One of the biggest problems companies face these days is that it is difficult to find good analytics people or data scientists.  Ken Rudin is looking for different kinds of people now as compared to five years ago.  Now he is looking for people with analytics and business abilities.  So for example, instead of just taking data that is given to you and looking for interesting patterns, you should be able to take a business goal like “increase player retention” and figure out how to do it, what data you need, and so on.

A recording of this panel is available at dyyno.com.

Amr Awadalla defined “big data” as 10 terabytes or more and noted that when Cloudera talks to prospective customers they often have “medium data” (less than 10 terabytes). Amr noted that individual nodes can have 4, 12, or soon even 24TB nodes.  So “medium data” problems don’t require a cluster or Hadoop at all just to deal with the size of the data.  The smallest Hadoop cluster requires three nodes. Most of Cloudera’s customers have tens to a few hundred node Hadoop clusters at the high end.  Facebook and Yahoo! have clusters with thousands of nodes.

Joshua Klahr said that Yahoo! collects tens of terabytes per day of ad data and weblogs telling them what ads are successful and what content is relevant.  Since introducing social applications and features, they have seen a dramatic increase in data because they are collecting text generated by users rather than just clicks.

Joydeep Sen Sarma said Facebook has 400 million users and collects over 12 terabytes of compressed data per day.  Facebook has over two petabytes of data.  Joydeep noted that the fact that things can now be measured that could not be measured previously and the fact that the value of web companies data per byte is much lower than for earlier kinds of companies have driven the collection of larger amounts of data and the shift toward open source platforms and tools.

Relatively few companies have petabyte scale data sets.  But the issue is not so much the size of the data.  The real issue is complexity.  Things like Hadoop are important not just because they enable companies to work with “stupendous” data sets but also (and more importantly) because they enable companies to work with complex datasets including data that has not been organized into a RDBMS or schema, and including text and weblogs.

The consensus of the panel was that “big data” and associated technologies already are spreading and will continue to spread.  Silicon Valley companies may be early developers and early adapters of technology for analyzing “Big Data” but it is spreading from web companies to other sectors including banking, games, government, and telecommunications and from the West to the East Coast and overseas.

Owen invited the panelists to comment on the NoSQL movement. Perhaps surprisingly, several panelists identified with NoSQL came out against the term in one way or another.

Amr Awadalla (Cloudera) prefers “NOSQL” (Not Only SQL) instead of “NoSQL” since more than half of all analysts use SQL. NoSQL doesn’t make sense if it argues against having SQL.  The main issue for him is “agility”:  the ability to make changes quickly and to be flexible with regard to how things are done.  Amr breaks this down into two kinds of agility:  ”agility of data types” and “agility of language.”

Amr explained his concept of “agility of data types”:  When traditional RDBMs are used and rigid schemas are required, it is necessary to go thru a DBA whenever a change in the schema is required (for example, to add a new column) and this can take a long time.  Ditto for loading new data into the schema:  ETL is required to load the data and this can take too long.  NoSQL approaches have the benefit that they allow operating without a schema or they allow for changing schemata easily and quickly.

Amr’s concept of “agility of language” is:  Taking a purely SQL-based approach with traditional RDBMs’s is too inflexible.  approaches based on Hadoop can go beyond SQL and allow the use of programming languages more powerful than SQL that accomodate the preferences of your developers, (e.g., Java, Python, C, Perl).

James Phillips (Northscale) identified himself as a NoSQL advocate but said it is not about SQL:  it’s not about the query language.   The issues are really storage, scaling, and performance.  The ACID transaction guarantees provided by traditional RDBMSs come with performance and scalability costs and many applications don’t need the guarantees but rather need greater scalability and higher performance.

Joydeep said NoSQL is like a religion and he hates religion.  Although Hadoop is often considered to be part of NoSQL systems, his Hive project introduced a simplified version of SQL on top of Hadoop because many analysts prefer to work with SQL.  The real advance has not been to eliminate SQL but rather it is the breaking down or deconstruction of previously monolithic systems into separable components and layers. The components and layers include storage (e.g., the filesystem) and processing (e.g., indexing, mapreduce, query processing, and text processing). One can “rack and stack”and build systems out of the components according to ones needs.

Joshua said that as a Product Manager, he uses Excel extensively.  And he said that Yahoo finds it easier to find SQL coders than MapReduce programmers.  The consensus was that Excel and SQL are here to stay.

Owen asked the panelists how we can avoid having a “data priesthood” and how we can promote the “democratization of data.”  Several panelists referred to the panel on “Competing on Analytics at the Highest Level” because the practices of the analytics competitors on that panel addressed this issue.  In addition, several other ways to make data usable across the company were mentioned, for example providing tools that make it easier for people with various backgrounds and knowledge and skills to use data.  For example, Hadoop is written in Java but programmers more familiar with other languages like Python and SQL programmers can also use it (e.g., using Streaming or Hive).  Going forward, we will see more connections from Hadoop, Hive, and Pig to existing BI tools like Microstrategy (e.g., thru ODBC connectors under development by Cloudera and Facebook) and this should further “democratize the data.”

A recording of the panel discussion is available at dyyno.com.

The online controlled experiments advocated by Ronny including AB tests are like the trials used to test drugs and get at the causes of observed effects since all the experimental subjects are exposed to the same non-causal factors. Ronny ran the audience through a series of examples showing how difficult it is to make correct decisions about a series of alternative web page designs.  Since people are bad at evaluating proposals, especially in evaluating more novel innovations, it is important to test a lot of ideas, “fail fast” and try again quickly instead of doing elaborate planning and preparation in advance.

In Ronny’s years of experience, he has observed that people and organizations go through stages:  they go from hubris to getting insights through measurement followed by the “Semmelweis Reflex” and then fundamental understanding.  Initially, people are sure they know it all but then they realize it helps to do experiments and take measurements to get data.  The Semmelweis reflex is the reflex-like rejection of new knowledge because it contradicts entrenched beliefs.  It is named after Ignaz Semmelweis who proposed that doctors clean their hands to reduce the spread of infections but who was rejected in spite of the fact that he had data to support his claim.

Ronny’s “take home” points were:

• data trumps intuition – it’s hard to assess the value of ideas so do experiments;
• get your organization to agree on what to optimize and use data to drive decisions.

A video recording of Ronny’s presentation is available at dyyno.com.  More technical information including information on how to conduct experiments is available at http://www.exp-platform.com.

In many cases three steps need to be taken: choosing a representation language, encoding a model in that language, and then performing inference on the model. Peter summarized his recommended approach with the acronym DINO: Data In Non-parametric model Out. Google’s Seti system for using machine learning to acquire models by learning from massive data sets is described by Simon Tong in the Google research blog at “Lessons Learned Developing a Practical Large Scale Machine Learning System.”

Jaap Suermondt gave a counterexample later in the day in his closing keynote. In the example, unmanageable amounts of data needed to be processed to solve an optimization problem. It was a linear programming problem but if turned out to be a special case that had a more efficient solution. Even so, it turned out to be necessary to improve on that for the special problem at hand in order to get a solution in a reasonable time. In this case, they had tons of data but it was just clutter until an improved algorithm was found that made it possible to get what was wanted out of the data.

Peter’s response to this counterexample is that Google also invests time into improving their algorithms. They have a lot of nearest neighbor problems and need to avoid searching for nearest neighbors so they invest effort into locality sensitive hashing resulting in a simple algorithm. So they are not dogmatic. Even so, the point is that it is surprisingly often the case that data is more important than programs.

In trying to capture the gist of Peter’s presentation, I have skipped over a lot of great examples and interesting points. A complete video recording of Peter’s presentation provided by Dyyno is available at “Analytics Conference – Keynote – Peter Norvig.” “The Unreasonable Effectiveness of Data” also appears as an “expert opinion” article published in IEEE Intelligent Systems by Alon Halevy, Peter Norvig, and Fernando Pereira, pp. 8-12, March/April, 2009. Seeds of the notion that more data can be better or more important than trying for better algorithms on smaller datasets appeared in an earlier presentation Peter gave at PARC Forum in 2006 on “Web Search as a Product of and Catalyst for AI.”

HP also forecasts demand for labor so as to be able to avoid having to fire or hire people.  When people are overcommitted, attrition goes through the roof.  When people are idle, production costs are unnecessarily high and this is an issue in PC production because it is so competitive and the margins are low.

Another example involved maximizing the revenue from covered orders (RCO – Revenue Coverage Optimization). Trying to maximize revenue by individual products produces results worse than random.  It’s important to consider combinations because of dependencies between products (e.g., it may not be possible to assemble a highly profitable order for a customer because some relatively unprofitable item is unavailable). When HP ranked their products by importance to revenue coverage, they got a nice Pareto effect:  80% of the revenue with 25% of the products.  This work won the coveted INFORMS Edelman prize in 2009.

Interestingly, this example runs counter to the idea that a lot of data makes it a bad idea or unnecessary to improve one’s algorithm.  There was no shortage of data in this case and in fact there was so much data that the initial algorithm, an integer programming algorithm, was too slow to be useful. Reformulating the problem as a Lagrangian Relaxation problem shortened the solution time.  But the breakthrough in this case involved reformulating the problem as a bipartite graph flow problem and coming up with a new algorithm for solving this problem using the data in real time.

In addition to operations, Jaap’s lab covers collaborative filtering, customer segmentation, marketing analytics, personalization, recommendation, and so on. He views personalization as the killer app for consumer facing services. He argued that it is crucial to lead with the customer experience and make it a win-win so that companies don’t face angry customers and backlash against privacy violations.

Another non-operations example Jaap described involved analytics work with Stanford Children’s hospital that has saved over thirty children’s lives. They analyzed patient’s data to find indicators that children were in trouble and used them to trigger more rapid responses.  There is a huge number of preventable deaths (100,000!) in the United States each year so there is a lot of room for more work using analytics to help save lives.

The video for Jaap’s talk is available at dyyno.com.

Another example involved applications of analytics to the US Patents database. Search was made available on the database by IBM on the web earlier but now they are doing analytics on this database with some customers for example to find which strategic areas competitors are investing R&D into.

Jeffrey’s talk is available at dyyno.com.

Sanjay pointed to a recent issue of “The Economist” as evidence that business people and not just information technology professionals are becoming more aware of analytics and business intelligence.  The issue had a cover story about “The Data Deluge” that talked about how businesses have more data than ever before and they are just beginning to exploit it using Business Intelligence software, one of fastest growing categories of software.

One of the ways SAP is leading its competitors is by pushing Business Intelligence “on demand”, SAP’s SaaS (Software as a Service) offering.  You can learn more and try out a free personal edition here. SAP is also leading by leveraging and working on disruptive emerging technologies including cloud computing, in-memory computation, and mobile technologies.

Sanjay predicted that analytics would be used to close the loop between strategy and execution.  He said that people have tended to focus on strategy almost exclusively or execution but both are important.  He quoted Thomas Edison’s dictum “Vision without execution is hallucination.”  SAP wants to help companies go around the loop from strategizing to planning to executing to optimizing based on the results of execution.  They will do this by providing an analytics and BI platform at the center of the loop that supports their customers at each step.

An interesting example of an application of SAP’s analytics involved improving sustainability and reducing the negative environmental impacts of a corporation.  Several other examples involved specific verticals such as government and health care.  Going forward, SAP believes that much of the growth in the future will involve targeting specific verticals so SAP is partnering with a large number of smaller companies to build industry-specific solutions on SAP’s platforms, including BI on-demand.

Sanjay’s presentation is available at dyyno.com.