2. August 2007 10:00
Earlier this week in episode 259 of .NET Rocks (DNR), Carl and Richard delved into the world of grid computing. Considering my natural love for the distributed environment, grid computing has always been a topic of great interest for me. Although DNR is already well known throughout the .NET world, I felt compelled to post a couple of comments about this particular show since it covered such a cool subject in the hopes that it might motivate some others to listen. For those of you that don’t know, grid computing is a specialized application of distributed computing that offers an alternative to the traditional supercomputer. Generally speaking, the idea is use a set of connected computers, called a grid, to harness its collective computing power for some form of parallel processing. Historically, this has been a niche area that was really only leveraged for problems that required extraordinarily computing capacity, which usually limited its use to the scientific community. However, it seems as though other industries are beginning to find opportunities to leverage grid computing as well.
On the show, Carl and Richard interviewed Dan Ciruli. He is the Product Manager of Digipede Network, which is a product offered by Digipede Technologies. Their product sounds incredibly cool and I am admittedly jealous of the developers that got to work on it. It provides a complete solution that handles all the plumbing of grid computing. Rather than having to roll your own solution, you can leverage theirs and spend your time focusing on the specific challenges of your domain. From what I understand, it is extremely flexible and you can literally start using the grid in about twenty lines of code. There is supposed to be an upcoming DNR TV episode that will demonstrate how easy it really is. You can also download a free developer editon to experiment with the product. I intend to give it a test drive myself.During the show, Dan was asked whether their product is capable of breaking up a task that is sent to the grid into a subset of parallel operations. Naturally, the answer was no. It would be extremely difficult, if not impossible, to programmatically determine how any given algorithm can be divided into subsets of logic that could be executed in parallel. The goal of their product seems to be more about making it easy to run embarrassingly parallel operations across the grid. Dan mentioned that probably close to zero of their customer base is actually attempting to redesign their algorithms into parallel subsets. It is just too complex and often too costly to undertake such an effort. Furthermore, many algorithms are simply too linear to break into parallel subsets. Rather than using parallel algorithms, Dan said most of their customers use the grid to process their data in parallel. In other words, it is easier to process 10,000 customers in parallel rather than divide the process itself into parallel components.
I completely agree with this point of view. In business applications, you are likely to get more bang for your buck by taking the approach of processing data in parallel than trying to break down your algorithms. Let me be clear. This isn’t a blanket statement that applies to all situations, but I suspect it applies more often than not. I can see the potential for several areas where this approach would be greatly beneficial in enterprise systems that process large volumes of data.
There is a great blog post written by Dan entitled It’s a Delightfully Parallel World in which he does a great job of elaborating this point. If the possibilities of parallel processing seem interesting to you, I would encourage you to read his post. Better yet, go listen to the DNR show!