Everyone applies induction looking for spam in “bags of words” but researchers have favored Support Vector Machines, random forests, and ensemble methods.  Everyone then does cross validation and accuracies of 99% are common.

Industrial applications differ from academic research in part because spam is one of those areas where there is a continuing arms race as both spammers and filters become ever more sophisticated. Tom showed a number of examples of spam illustrating the fact that text models alone are insufficient.

In industry, multiple stages are used with more intensive text processing reserved for later stages after earlier simpler stages have already identified half or more of the spam.  Proofpoint uses reputation models on the IP addresses of the sender and on domains and URLs, even taking into account how long the owner has held a domain and so on.  They use Spam Assassin, a public domain regular expression and rule based content matching system.  But that is just a small step in their overall process.  It provides some of the information used but Proofpoint’s lexicon and content models provide most of the leverage.

Interestingly, while most researchers use a simple representation (bag of words) with complex learning methods such as Support Vector Machines (SVM), Proofpoint uses a complex representation including context markers, AND/OR rules as well as phrases and words. They use a very simple classifier and learning method:  they use logistic regression to assign positive and negative weights to terms.  SVM, random forests, ensembles and so on don’t offer much improvement.  R^2 is about .95 so there is no need for better accuracy.  This is because Proofpoint’s representation is sophisticated and already incorporates attribute interactions, conditional dependencies and so on.  There are 1.3 million features going into the logistic regression.

A major advantage of having a simple classifier and using logistic regression is that it is comprehensible and transparent.  A “white box” is better for this application than opaque “black box” approaches. When something is classified as spam, an analyst can run it through the classifier and show the terms that influenced the classification, their weights, and so on.

Proofpoint reparses their corpus every 24 hours and updates the classifier in about 1.5 hours.  They age things out of their corpus so that it can change over time as spam changes over time.  In order to be able to respond quickly to spam attacks, they also have a special “fast attack learning and response” system that adds new terms to the model every fifteen minutes.  Usually the representation is wrong rather than the model, so it is not necessary to rebuild the model from scratch and it suffices to improve the representation by adding terms.

Proofpoint does well in competitions against their competitors in part because they do text analysis whereas many of their competitors only do IP and domain filtering.  In an ongoing effort to stay ahead of spammers, link mining and network analysis are now being looked at as potential sources of information that can support spam identification.

Flurry made a big splash in the news when Steve Jobs got pissed off at them and called them out by name in an interview because they outed Apple’s iPad when it was still a closely guarded secret. (See a short video outtake of the interview at VentureBeat.)  Apple responded by changing legal agreements to exclude some third party analytics and some advertising.

Flurry changed how they collect device data.  The Department of Justice and the Federal Trade Commission began to investigate Apple around this time due to a collection of legal restrictions and potentially anti-competitive business practices which may have come to include the analytics proscription.  Apple later relaxed the analytics ban.

Flurry is inside about a fifth of all apps and they cover 150 million devices:  about 90% of all Android devices and 80% of all iPhones.

Flurry’s analytics works as follows:

1. Developers download Flurry’s SDK and wire it into their app (~ 5 minutes).
2. The instrumented app sends data via the internet to Flurry’s servers.
3. The data is processed using Hadoop on HBase and  stored in “giant cubes.”
4. Clients can then get metrics from a data warehouse thru a web service at Flurry.com.

Flurry provides a free service and then makes money with a product called AppCircle.  AppCircle combines an app recommender with rewards that use virtual currency.  AppCircle recommends another app to someone who already has a different app just as Amazon and others recommend a new purchase given that you have already bought or you are about to buy a given item.  Flurry makes money by splitting sales commissions with publishers roughly evenly if the app is installed after they recommend it.  One of the advantages they have is that they know which apps you have from the same publisher so they  will not recommend something you already have.

Flurry’s analytics service has accumulated 30TB of data since the company was launched.  They compress data on the client prior to sending it to their servers.  The core IP for this was developed earlier when the founders had to do compression to reduce data sizes on earlier handsets that were much less powerful than today’s handsets.  Flurry has one of the largest HBase databases.  Six servers receive reports from devices and store and forward them so they can be processed into their HBase db.

Perhaps the most interesting observation that was made with Flurry analytics is that the level of “engagement” (defined as frequency of use x the length of session) is far higher for iPhones than it is for other handsets.  iPhone apps are used far more often and for much longer periods of time (for example, people with iPhones use them to read far more than others do).

Perhaps the most important observation Flurry made is that there are 5k Android app developers and 100k apps in Android market versus 35k AppStore app developers and 300k apps in the AppStore. Apple is making money for developers in part by rewarding them with previously unheard of revenue sharing (70%).  For the Android marketplace to become viable, catch up, and surpass Apple it must attract app developers by making money for them.  Currently, the Android market is broken and it must be fixed.  Credit card scams occur in the Android store and they only go away when people vote them out (the customers are the only police) and there are a lot of other problems with the store that make it harder for app publishers to make money there.  One issue is that Android marketplace does not have a large paying customer base.  Apple has 150 million consumer credit cards (only Amazon and eBay have more).  Android is trying to work on this by working a deal with PayPal.

Currently, app developers and publishers “monetize” (make money) in several ways:

• customers pay up front;
• apps are free but they get you later:
  • advertising
  • micro payments
  • virtual currency
• app stores do not support subscriptions

Carrier’s stores failed in part because of the ways they tried to make money.  For example, typically nothing was free.

Flurry observed that Apple’s AppStore achieved the adaption levels achieved by the iTune store in 1/3 the time. Also, currently, the percentage of game apps is much higher in the AppStore as compared to Android market.  It may be that this is partly due to the fact that Apple regulates categories in the AppStore while Android allows developers or publishers to self-report the categories of their apps.  This is a data quality issue for analytics.

Apple provides basic analytics for example a top downloads list and monthly store statistics.  Android apps can only see how many downloads occur if they have Flurry inside.

Flurry pointed out that it is important to have users login with a userid and password so that you can tie their activity together (rather than trying to use the handset’s Unique Device Identification Number).  They also noted that “phones lie” about the current time:  10% had “bad clocks” reporting dates in the 1970’s or 2200AD.  Flurry also deals with duplicate data coming in and de-duplicates it.  And they use some form of machine learning to identify devices, carriers, and countries.

In addition to analytics and recommendation, Flurry does clustering (especially for Games).

Some interesting opinions about Google surfaced.  The Nexus One was a commercial flop because you had to pay $500 to buy it online and then you still had to sign up with a carrier (the carrier didn’t “come with it”).  Further, Google doesn’t know how to “cuddle” a consumer.  They are engineering-oriented to a fault.  Andy Rubin’s response to Steve Job’s comments about Android (about it not being “open”) was to tweet Unix code for downloading Android, a perfect example.

To show how Flurry can be used, examples were given of ad network performance and tracking cross selling.  In the case of cross selling multi-dimensional scaling was used to highlight differences and similarities between items (apps in a portfolio).

A member of the audience asked whether Jobs can or might do the same things Flurry is doing.  The response was that any platform provider can build something up and cut out earlier third parties.  So there is some risk of that happening.

Salesforce’s analytics strategy is to start simple and grow from there.  Donovan counted dashboards and reports as analytics (in contrast to others who define analytics as what you have left when you subtract ETL and reporting from BI). He noted that Salesforce users have built three quarters of a million dashboards and dashboards are viewed 700,000 times per day.  There are over 12 million reports and 2.5 million reports are run each day.  In addition to dashboards and reports, Donovan included list views and search in the collection of analytics tools currently available.  Future additions to analytics will build from this collection instead of adding relatively high end analytics.

Donovan showed a nice architecture slide contrasting the traditional datawarehouse based approach to the new cloud-based approach taken by Salesforce and claimed that the datawarehouse approach is out of date, too complicated, and too rigid.  Salesforce’s new approach takes advantage of their cloud-based force.com API, multitenant architecture and platform so it is easy to use, realtime, and flexible.

Donovan gave a brief review of Salesforce’s multi-tenant architecture. Craig Weissman, Salesforce’s CTO, gave a presentation on the multi-tenant architecture for SDForum’s Software Architecture SIG back when I helped organize that group. Click here for more on this topic.

Salesforce achieves realtime analytics and simplification by not having a datawarehouse.  But unlike many companies using NoSQL data stores Salesforce does use a traditional RDBMS: Oracle. Currently, there are around fifteen databases for production each containing around ten terabytes. Salesforce relies on ACID transactions because they are supporting companies doing business on their platform.  When transactions are committed, the changes are available immediately everywhere and this supports the goal of realtime analytics.

Interestingly, Salesforce does not cache results of queries and so on because things change so frequently.  Salesforce is able to get away without caching and does without a datawarehouse in part by making query execution efficient.  Donovan went into some depth on how queries are optimized.  Salesforce has its own optimizer and does its own indexing in part because the multi-tenant architecture of their data often makes it impossible to directly use Oracle’s indexing and optimization.

In closing, Donovan said his top priorities now are adding analytical capabilities, improving scalability and usability, and supporting collaboration.  You can try out analytics for free as it is available in Salesforce’s sandbox along with a new report builder.  Donovan’s slides are available here and at SDForum.org.

When hiring, consider what do you want the person you hire to accomplish? Create a scorecard comprised of mission, outcomes, and competencies:

• mission: the job’s core purpose; why the role exists (in 1-5 sentences)
• outcomes: what must get done for A performance (not what the person will be doing but rather 3-8 specific objectives)
• competencies: arbitrarily many role based competencies that describe behaviors that must be demonstrated to achieve outcomes plus 5-8 that describe the company culture

The scorecard ties the two legs of the “A Method” for hiring, “Source” and “Select” together.

Four layers of interviews are used to select people: the (phone) screening interview, the top-grading, focused and reference interviews.

The screening interview has four main questions:

• What are your career goals?
• What are you really good at professionally? (8-12 positives with examples)
• What are you not good at or not interested in doing professionally?
• Who were your last five bosses and how will they each rate your performance on a 1-10 scale when we talk with them?

The top-grading interview covers the last 15 years from the first company to the most recent one:

• What were you hired to do?
• What accomplishments are you most proud of?
• What were some low points during that job?
• Who were the people you worked with?
  • What was your boss’s name? What was it like working with him/her? What will he/she tell me were your biggest strengths and areas for improvement?
  • How would you rate the team you inherited on an A, B, C scale? What changes did you make? Did you hire anybody? Did you fire anybody? How would you rate the team on an A, B, C scale when you left?
• Why did you leave that job?

Check whether the accomplishments they were most proud of connected to the desired results, the mission and outcomes they were hired to do. Drill down into the low points, especially if they can’t think of any or claim there weren’t any: What went wrong? What was their biggest mistake? What part of the job did they not like? In what ways were their peers stronger? Asking for the names of people they worked with establishes the TORC (Threat of Reference Check) and improves veracity of responses. Watch for whether they are disrespectful of their previous bosses. Check for “push versus pull”: do not hire candidates pushed out of jobs more than 20% of the time.

Allow time for discussion of aspirations and career goals and for questions. To keep things on course, interrupt without shutting people up and use the “three P’s” asking for comparisons with previous, plans, and peers.

A focused interview focuses on the following statement and questions:

• “The purpose of this interview is to talk about ___ (outcome or competency.”
• What are your biggest accomplishments in this area in your career?
• What are your insights into your biggest mistakes and lessons learned in this area?

A reference interview checks references:

• In what context did you work with this person?
• What were the person’s biggest strengths?
• What were the person’s biggest areas for improvement back then?
• How would you rate the performance in that job on a 1-10 scale? What about his/her performance causes you to give that rating?
• The person mentioned that he/she struggled with ___ on that job. Can you tell me more about that?

After collecting information in interviews, one can then decide and act. The “Who” method looks for alignment or a good match (visualized in terms of concentric circles as in an archery target) between the candidate’s skillset and will and the scorecard. In this stage, avoid red flags like behavior warning signs (Marshall Goldsmith) including avoiding responsibility, blaming others, making excuses and making negative comments about previous colleagues.

A summary of the overall process for selecting “A” players is:

1. screening interview
2. top-grading interview
3. focused interview(s)
4. candidate discussion
5. reference interview(s)
6. final decision (skill will analysis = bulls eye?)

Once a candidate is selected, you must “sell” them on joining by addressing fit (between the company’s vision, needs, and culture and the candidate’s goals, strengths,and values), family, freedom, fortune, and fun.

The process described by this book and summarized here seems far better than the relatively irrational and random processes used by most companies to hire new employees. However, the book frequently makes assumptions that are open to question. For example, the book assumes that it is best to hire people who fit specific job requirements that a company has at a given time. This assumption is a basis for the use of scorecards. Google is currently employing a hiring strategy that appears to be successful that contradicts the specialist approach. Google’s strategy is to hire “smart generalists” who can work on a variety of different projects over time instead of hiring specialists.

Assumptions like these reflect core values of companies and their management and not just hypotheses about what will work best. For example, companies that minimize costs and headcount and that assume they can fire workers as soon as their skills are no longer needed and hire new workers with newly relevant skills have less interest in hiring generalists. But a company that wants to hire people and to avoid firing them for ethical or humanitarian reasons, just because corporate goals, projects and technologies change, may want to hire more broadly skilled individuals.

I would like to see analytics and a more scientific approach used to evaluate alternative hiring methods and their underlying assumptions. My guess is that different strategies work for different kinds of jobs in different companies and in different industries.

Agility was a key theme of the presentation and part of James’ approach is intended to support quick actions. When an event occurs, in some cases it is important to notice quickly and act almost immediately as the value of action based on the information about the event decreases rapidly as time passes. With other events, you have more time to consider options. But the key is to notice and act appropriately in a timely manner. It isn’t enough just to make information available or notice things quickly or even in real time. Sometimes noticing immediately isn’t even necessary or useful (e.g., when the required action would take a long time anyway).

James uses a three stage approach to improving operational decisions:

1. discover the important decisions
2. build (automate) decision services
3. analyze decisions and create a closed loop between analytics and decision services.

The slides for James’s presentation are available at the BI SIG’s web page at http://SDForum.org/BISIG and here.

Over 250 people attended. Some leading analytics gurus and data scientists gave presentations and participated in panel discussions.  This blog post links to summaries of the keynotes and panels.

There were four panel discussions at the conference:

• Competing on Analytics
• Analyzing Big Data
• New Frontiers
• The Investor Perspective

Keynote speakers representing an array of large companies with strong analytics capabilities gave presentations on a wide range of analytics efforts:

• Ronny Kohavi (Microsoft) “Online Controlled Experiments:  Listening to the Customers, Not to the HiPPO“
• Sanjay Poonen (SAP) “Leading the Analytics Revolution“
• Peter Norvig (Google) “The Unreasonable Effectiveness of Data“
• Jeff Kreulen (IBM) “Analytics:  An Applied Researcher’s Perspective“
• Jaap Suermondt (HP) “Research in Analytics for Operational Impact at HP“

You can click on the links above to get quick summaries.  The keynote presentations and panel discussions were recorded and the audio and video recordings are available individually from the pages above or at dynno.com or at SDForum here.  The conference was sponsored by Accel Partners, Dynno, IBM, Impetus, Microsoft and SAP and the organizing committee included Stacey Bishop (Scale Venture Partners), Jim Claussen (IBM), Lars Leckie (Hummer Winblad), Sonja London (Summit Software), Paul O’Rorke (Kraken Data), and Julia Queck (SAP).

David began by asking:  how do the panelists use analytics to get a competitive advantage?  What is an example of something they found that surprised them?

Ken Rudin and Zynga use analytics to achieve their goals of increasing revenue, improving user retention and increasing viral spread for their online social games including Farmville.  They collect 3-4 terabytes of data from their 40-50 million daily users.  Initially they were reactive:  producing reports in response to requests.  Now they use analytics as an integral part of game design:  they use AB testing and experiments in development just as QA is an integral part of development.  Analysts are part of the design team.  An example of something that surprised them occurred when they analyzed Mafia game players.  There are two groups of players, the “crime jobbers” and the “fighters.”  They discovered that fighters spend twice as much because they are trying to compete with their friends so they purchase weapons to arm their mafia members so they can fight and defeat their friend’s mafia gang.  Since they discovered this, they have changed the game to encourage players to fight more instead of just doing crime jobs.

Kevin Weil mentioned that a key accomplishment of the analytics team at Twitter has been to get everyone to consider data (and to make it possible for them to do so) in all important decisions.  He described an example of how analytics surprised them that involved the social network underlying Twitter.  Many people use Twitter largely as an information source and their network of “following” links says something about their interests.  But the bidirectional links can be ignored when trying to determine their interests.  The unidirectional links (e.g., the people they follow who don’t follow them) are the ones that carry the most important incoming information.

David Steier asked the panelists:  How do you organize people to achieve the company’s goals?  The panelists companies all have analytics teams and a team responsible for the company’s analytics platform and analysts who work with other teams but the way they work within the companies is different and several companies are adapting innovative approaches.

DJ Patil and LinkedIn started out by looking at other companies such as Google and Yahoo.  Yahoo had analytics in a separate research organization and it was difficult or impossible to get it into products.  Google is driven by technology and bolts products on top (see also http://techcrunch.com/2010/05/15/facebook-google/).  So to ensure that analytics is integrated into products at LinkedIn, the Analytics team is a substantial part (1/4th) of the product team and has its own designers and developers so it is easier to go straight to production or to integrate with existing products.

Neel Sundaresan (eBay) claimed that “everybody should be an analytics scientist.”  eBay has an analytics platform team that provides data to the rest of the company and “the data tells you what the product should be.”  With 200 million users and a billion searches per day, eBay gets tremendous amounts of data and product managers and developers and even some machine learning scientists need to learn to look at the data.

Ken Rudin (Zynga) argued that the whole idea of having an “analytics team” is flawed.  He asked:  ”Does Microsoft have an internet division?”  Like the internet, analytics is or should be fundamental to everything in the company.  So his goal is to work himself out of a job by putting analysts in development and product teams and by training almost everyone in the company in analytics starting with product managers, then engineers, and then quality assurance.

In summary, one of the key themes of the panel was that companies competing on analytics are finding innovative ways to integrate analytics throughout their companies.  One of the biggest problems companies face these days is that it is difficult to find good analytics people or data scientists.  Ken Rudin is looking for different kinds of people now as compared to five years ago.  Now he is looking for people with analytics and business abilities.  So for example, instead of just taking data that is given to you and looking for interesting patterns, you should be able to take a business goal like “increase player retention” and figure out how to do it, what data you need, and so on.

A recording of this panel is available at dyyno.com.

Amr Awadalla defined “big data” as 10 terabytes or more and noted that when Cloudera talks to prospective customers they often have “medium data” (less than 10 terabytes). Amr noted that individual nodes can have 4, 12, or soon even 24TB nodes.  So “medium data” problems don’t require a cluster or Hadoop at all just to deal with the size of the data.  The smallest Hadoop cluster requires three nodes. Most of Cloudera’s customers have tens to a few hundred node Hadoop clusters at the high end.  Facebook and Yahoo! have clusters with thousands of nodes.

Joshua Klahr said that Yahoo! collects tens of terabytes per day of ad data and weblogs telling them what ads are successful and what content is relevant.  Since introducing social applications and features, they have seen a dramatic increase in data because they are collecting text generated by users rather than just clicks.

Joydeep Sen Sarma said Facebook has 400 million users and collects over 12 terabytes of compressed data per day.  Facebook has over two petabytes of data.  Joydeep noted that the fact that things can now be measured that could not be measured previously and the fact that the value of web companies data per byte is much lower than for earlier kinds of companies have driven the collection of larger amounts of data and the shift toward open source platforms and tools.

Relatively few companies have petabyte scale data sets.  But the issue is not so much the size of the data.  The real issue is complexity.  Things like Hadoop are important not just because they enable companies to work with “stupendous” data sets but also (and more importantly) because they enable companies to work with complex datasets including data that has not been organized into a RDBMS or schema, and including text and weblogs.

The consensus of the panel was that “big data” and associated technologies already are spreading and will continue to spread.  Silicon Valley companies may be early developers and early adapters of technology for analyzing “Big Data” but it is spreading from web companies to other sectors including banking, games, government, and telecommunications and from the West to the East Coast and overseas.

Owen invited the panelists to comment on the NoSQL movement. Perhaps surprisingly, several panelists identified with NoSQL came out against the term in one way or another.

Amr Awadalla (Cloudera) prefers “NOSQL” (Not Only SQL) instead of “NoSQL” since more than half of all analysts use SQL. NoSQL doesn’t make sense if it argues against having SQL.  The main issue for him is “agility”:  the ability to make changes quickly and to be flexible with regard to how things are done.  Amr breaks this down into two kinds of agility:  ”agility of data types” and “agility of language.”

Amr explained his concept of “agility of data types”:  When traditional RDBMs are used and rigid schemas are required, it is necessary to go thru a DBA whenever a change in the schema is required (for example, to add a new column) and this can take a long time.  Ditto for loading new data into the schema:  ETL is required to load the data and this can take too long.  NoSQL approaches have the benefit that they allow operating without a schema or they allow for changing schemata easily and quickly.

Amr’s concept of “agility of language” is:  Taking a purely SQL-based approach with traditional RDBMs’s is too inflexible.  approaches based on Hadoop can go beyond SQL and allow the use of programming languages more powerful than SQL that accomodate the preferences of your developers, (e.g., Java, Python, C, Perl).

James Phillips (Northscale) identified himself as a NoSQL advocate but said it is not about SQL:  it’s not about the query language.   The issues are really storage, scaling, and performance.  The ACID transaction guarantees provided by traditional RDBMSs come with performance and scalability costs and many applications don’t need the guarantees but rather need greater scalability and higher performance.

Joydeep said NoSQL is like a religion and he hates religion.  Although Hadoop is often considered to be part of NoSQL systems, his Hive project introduced a simplified version of SQL on top of Hadoop because many analysts prefer to work with SQL.  The real advance has not been to eliminate SQL but rather it is the breaking down or deconstruction of previously monolithic systems into separable components and layers. The components and layers include storage (e.g., the filesystem) and processing (e.g., indexing, mapreduce, query processing, and text processing). One can “rack and stack”and build systems out of the components according to ones needs.

Joshua said that as a Product Manager, he uses Excel extensively.  And he said that Yahoo finds it easier to find SQL coders than MapReduce programmers.  The consensus was that Excel and SQL are here to stay.

Owen asked the panelists how we can avoid having a “data priesthood” and how we can promote the “democratization of data.”  Several panelists referred to the panel on “Competing on Analytics at the Highest Level” because the practices of the analytics competitors on that panel addressed this issue.  In addition, several other ways to make data usable across the company were mentioned, for example providing tools that make it easier for people with various backgrounds and knowledge and skills to use data.  For example, Hadoop is written in Java but programmers more familiar with other languages like Python and SQL programmers can also use it (e.g., using Streaming or Hive).  Going forward, we will see more connections from Hadoop, Hive, and Pig to existing BI tools like Microstrategy (e.g., thru ODBC connectors under development by Cloudera and Facebook) and this should further “democratize the data.”

A recording of the panel discussion is available at dyyno.com.

The online controlled experiments advocated by Ronny including AB tests are like the trials used to test drugs and get at the causes of observed effects since all the experimental subjects are exposed to the same non-causal factors. Ronny ran the audience through a series of examples showing how difficult it is to make correct decisions about a series of alternative web page designs.  Since people are bad at evaluating proposals, especially in evaluating more novel innovations, it is important to test a lot of ideas, “fail fast” and try again quickly instead of doing elaborate planning and preparation in advance.

In Ronny’s years of experience, he has observed that people and organizations go through stages:  they go from hubris to getting insights through measurement followed by the “Semmelweis Reflex” and then fundamental understanding.  Initially, people are sure they know it all but then they realize it helps to do experiments and take measurements to get data.  The Semmelweis reflex is the reflex-like rejection of new knowledge because it contradicts entrenched beliefs.  It is named after Ignaz Semmelweis who proposed that doctors clean their hands to reduce the spread of infections but who was rejected in spite of the fact that he had data to support his claim.

Ronny’s “take home” points were:

• data trumps intuition – it’s hard to assess the value of ideas so do experiments;
• get your organization to agree on what to optimize and use data to drive decisions.

A video recording of Ronny’s presentation is available at dyyno.com.  More technical information including information on how to conduct experiments is available at http://www.exp-platform.com.

In many cases three steps need to be taken: choosing a representation language, encoding a model in that language, and then performing inference on the model. Peter summarized his recommended approach with the acronym DINO: Data In Non-parametric model Out. Google’s Seti system for using machine learning to acquire models by learning from massive data sets is described by Simon Tong in the Google research blog at “Lessons Learned Developing a Practical Large Scale Machine Learning System.”

Jaap Suermondt gave a counterexample later in the day in his closing keynote. In the example, unmanageable amounts of data needed to be processed to solve an optimization problem. It was a linear programming problem but if turned out to be a special case that had a more efficient solution. Even so, it turned out to be necessary to improve on that for the special problem at hand in order to get a solution in a reasonable time. In this case, they had tons of data but it was just clutter until an improved algorithm was found that made it possible to get what was wanted out of the data.

Peter’s response to this counterexample is that Google also invests time into improving their algorithms. They have a lot of nearest neighbor problems and need to avoid searching for nearest neighbors so they invest effort into locality sensitive hashing resulting in a simple algorithm. So they are not dogmatic. Even so, the point is that it is surprisingly often the case that data is more important than programs.

In trying to capture the gist of Peter’s presentation, I have skipped over a lot of great examples and interesting points. A complete video recording of Peter’s presentation provided by Dyyno is available at “Analytics Conference – Keynote – Peter Norvig.” “The Unreasonable Effectiveness of Data” also appears as an “expert opinion” article published in IEEE Intelligent Systems by Alon Halevy, Peter Norvig, and Fernando Pereira, pp. 8-12, March/April, 2009. Seeds of the notion that more data can be better or more important than trying for better algorithms on smaller datasets appeared in an earlier presentation Peter gave at PARC Forum in 2006 on “Web Search as a Product of and Catalyst for AI.”