Kevin Foster talked about Stream Computing in general and at IBM at the monthly meeting of the SVForum Business Intelligence Special Interest Group on December 4th, 2012. The slides are available at the BI SIG’s home page on SVForum’s website.
Stream computing is part of a paradigm shift from on-line transaction processing and analytical processing (OLTP and OLAP) to RTAP: Real Time Analytical Processing. IBM’s RTAP offering is called Infosphere Streams. It provides a programming environment that enables users to set up dataflow graphs. It takes advantage of eclipse and is part of IBM’s larger Big Data platform which also includes text analytics.
Kevin asked two audience members to participate in a demonstration. He gave quarters to each person but asked one to put each quarter in a pocket and to draw them out in order to do an operation like calculating the total number of heads or tails. The other person did not have to store quarters in a pocket and repeatedly draw them out. Just as a person who has to “store” incoming quarters is much slower and finds it more difficult to deal with a stream of incoming quarters, so it is with computers when they have to store data on disk. If a huge stream of data is coming in very quickly, it is much better to have the computer do all operations in memory rather than attempting to store data on relatively slow disk drives.
Kevin gave a lot of outstanding examples of applications of stream computing in health care, environmental monitoring, telecommunications and other verticals. Telecom applications are currently the heaviest application area.
Streams has been benchmarked to handle over 10 million events per second. The development environment can automatically scale processing resources to multiple performance elements.
In responses to audience questions, Kevin mentioned that performance is not as good as Storm with respect to integrating Java but that is something that is being worked on.
IBM runs “Big Data Developer Hackathons” where they demonstrate Infosphere Streams and other related technologies. They give developers an opportunity to get their hands on these technologies and see how they work using simple examples. The IBM experts available at these sessions make it much easier to quickly learn to use the new technology.