Sunday, May 31, 2015

Calculate Input/Output Operations Per Second (IOPS) in a storage array

When it comes to measuring a storage system's overall performance, Input/Output Operations Per Second (IOPS) is still the most common metric in use. There are a number of factors that go into calculating the IOPS capability of an individual storage system.
In this article, I provide introductory information that goes into calculations that will help you figure out what your system can do. Specifically, I explain how individual storage components affect overall IOPS capability. I do not go into seriously convoluted mathematical formulas, but I do provide you with practical guidance and some formulas that might help you in your planning. Here are three notes to keep in mind when reading the article:
  • Published IOPS calculations aren't the end-all be-all of storage characteristics. Vendors often measure IOPS under only the best conditions, so it's up to you to verify the information and make sure the solution meets the needs of your environment.
  • IOPS calculations vary wildly based on the kind of workload being handled. In general, there are three performance categories related to IOPS: random performance, sequential performance, and a combination of the two, which is measured when you assess random and sequential performance at the same time.
  • The information presented here is intended to be very general and focuses primarily on random workloads.

IOPS calculations

Every disk in your storage system has a maximum theoretical IOPS value that is based on a formula. Disk performance -- and IOPS -- is based on three key factors:
  • Rotational speed (aka spindle speed). Measured in revolutions per minute (RPM), most disks you'll consider for enterprise storage rotate at speeds of 7,200, 10,000 or 15,000 RPM with the latter two being the most common. A higher rotational speed is associated with a higher performing disk. This value is not used directly in calculations, but it is highly important. The other three values depend heavily on the rotational speed, so I've included it for completeness.
  • Average latency. The time it takes for the sector of the disk being accessed to rotate into position under a read/write head.
  • Average seek time. The time (in ms) it takes for the hard drive's read/write head to position itself over the track being read or written. There are both read and write seek times; take the average of the two values.
To calculate the IOPS range, use this formula: Average IOPS: Divide 1 by the sum of the average latency in ms and the average seek time in ms (1 / (average latency in ms + average seek time in ms).

Sample drive:
  • Model: Western Digital VelociRaptor 2.5" SATA hard drive
  • Rotational speed: 10,000 RPM
  • Average latency: 3 ms (0.003 seconds)
  • Average seek time: 4.2 (r)/4.7 (w) = 4.45 ms (0.0045 seconds)
  • Calculated IOPS for this disk: 1/(0.003 + 0.0045) = about 133 IOPS
So, this sample drive can support about 133 IOPS. Compare this to the chart below, and you'll see that the value of 133 falls within the observed real-world performance exhibited by 10K RPM drives.
However, rather than working through a formula for your individual disks, there are a number of resources available that outline average observed IOPS values for a variety of different kinds of disks. For ease of calculation, use these values unless you think your own disks will vary greatly for some reason.
Below I list some of the values I've seen and used in my own environment for rough planning purposes. As you can see, the values for each kind of drive don't radically change from source to source.


 Note: The drive type doesn't enter into the equation at all. Sure, SAS disks will perform better than most SATA disks, but that's only because SAS disks are generally used for enterprise applications due to their often higher reliability as proven through their mean time between failure (MTBF) values. If a vendor decided to release a 15K RPM SATA disk with low latency and seek time values, it would have a high IOPS value, too.

Multidisk arrays

Enterprises don't install a single disk at a time, so the above calculations are pretty meaningless unless they can be translated to multidisk sets. Fortunately, it's easy to translate raw IOPS values from single disk to multiple disk implementations; it's a simple multiplication operation. For example, if you have ten 15K RPM disks, each with 175 IOPS capability, your disk system has 1,750 IOPS worth of performance capacity. But this is only if you opted for a RAID-0 or just a bunch of disks (JBOD) implementation. In the real world, RAID 0 is rarely used because the loss of a single disk in the array would result in the loss of all data in the array.
Let's explore what happens when you start looking at other RAID levels.

The IOPS RAID penalty

Perhaps the most important IOPS calculation component to understand lies in the realm of the write penalty associated with a number of RAID configurations. With the exception of RAID 0, which is simply an array of disks strung together to create a larger storage pool, RAID configurations rely on the fact that write operations actually result in multiple writes to the array. This characteristic is why different RAID configurations are suitable for different tasks.
For example, for each random write request, RAID 5 requires many disk operations, which has a significant impact on raw IOPS calculations. For general purposes, accept that RAID 5 writes require 4 IOPS per write operation. RAID 6's higher protection double fault tolerance is even worse in this regard, resulting in an "IO penalty" of 6 operations; in other words, plan on 6 IOPS for each random write operation. For read operations under RAID 5 and RAID 6, an IOPS is an IOPS; there is no negative performance or IOPS impact with read operations. Also, be aware that RAID 1 imposes a 2 to 1 IO penalty.
The chart below summarizes the read and write RAID penalties for the most common RAID levels.


Parity-based RAID systems also introduce other additional processing that result from the need to calculate parity information. The more parity protection you add to a system, the more processing overhead you incur. As you might expect, the overall imposed penalty is very dependent on the balance between read and write workloads.
A good starting point formula is below. This formula does not use the array IOPS value; it uses a workload IOPS value that you would derive on your own or by using some kind of calculation tool, such as the Exchange Server calculator.
(Total Workload IOPS * Percentage of workload that is read operations) + (Total Workload IOPS * Percentage of workload that is read operations * RAID IO Penalty)

As an example, let's assume the following:
  • Total IOPS need: 250 IOPS
  • Read workload: 50%
  • Write workload: 50%
  • RAID level: 6 (IO penalty of 6)
Result: You would need an array that could support 875 IOPS to support a 250 IOPS RAID 6-based workload that is 50% writes.
This could be an unpleasant surprise for some organizations, as it indicates that the number of disks might be more important than the size (i.e., you'd need twelve 7,200 RPM, seven 10K RPM, or five 15K RPM disks to support this IOPS need).

The transport choice

It's also important to understand what is not included in the raw numbers: the transport choice -- iSCSI or Fibre Channel. While the transport choice is an important consideration for many organizations, it doesn't directly impact the IOPS calculations. (None of the formulas consider the transport being used.)

The transport choice is an important one, but it's not the primary choice that many would make it out to be. For larger organizations that have significant transport needs (i.e., between the servers and the storage), Fibre Channel is a good choice, but this choice does not drive the IOPS wagon.

Summary

In order to intricately understand your IOPS needs, you need to know a whole lot, including specific disk technicalities, your workload breakdown as a function of read vs. write, and the RAID level you intend to use. Once you implement your solution, you can use tools that are tailor-made to IOPS analysis, such as Iometer, to get specific, real-time performance values. This assumes that you have a solution in place that you can measure.
If you're still in the planning stages or a deep level of analysis simply isn't necessary for your needs, the generalities presented in this article will help you figure out your needs.


Sources:

Saturday, May 23, 2015

NAT and PAT: A complete explanation

Network address translation (NAT) is the process of modifying IP address information in IP packet headers while in transit across a traffic routing device.
There are two different types of NAT:

NAT

    • Static NAT: The simplest type of NAT provides a one-to-one translation of IP addresses. It is often also referred to as one-to-one NAT. In this type of NAT only the IP addresses, IP header checksum and any higher level checksums that include the IP address need to be changed. The rest of the packet can be left untouched (at least for basic TCP/UDP functionality, some higher level protocols may need further translation). Basic NATs can be used when there is a requirement to interconnect two IP networks with incompatible addressing. With static NAT, translations exist in the NAT translation table as soon as you configure static NAT command(s), and they remain in the translation table until you delete the static NAT command(s).
    • Dynamic NAT: Dynamic NAT has some similarities and differences compared to static NAT. Like static NAT, the NAT router creates a one-to-one mapping between an inside local and inside global address and changes the IP addresses in packets as they exit and enter the inside network. However, the mapping of an inside local address to an inside global address happens dynamically. Dynamic NAT sets up a pool of possible inside global addresses and defines matching criteria to determine which inside local IP addresses should be translated with NAT. The dynamic entry stays in the table as long as traffic flows occasionally. With dynamic NAT, translations do not exist in the NAT table until the router receives traffic that requires translation. Dynamic translations have a timeout period after which they are purged from the translation table.
  •  

PAT

    • Static PAT: Static PAT translations allow a specific UDP or TCP port on a global address to be translated to a specific port on a local address. Static PAT is the same as static NAT, except that it enables you to specify the protocol (TCP or UDP) and port for the real and mapped addresses. Static PAT enables you to identify the same mapped address across many different static statements, provided that the port is different for each statement. You cannot use the same mapped address for multiple static NAT statements. With static PAT, translations exist in the NAT translation table as soon as you configure static PAT command(s), and they remain in the translation table until you delete the static PAT command(s).
    • NAT Overload or PAT: It is common to hide an entire IP address space, usually consisting of private IP addresses, behind a single IP address (or in some cases a small group of IP addresses) in another (usually public) address space. This type of NAT is called PAT in overload. The dynamic entry stays in the table as long as traffic flows occasionally. With PAT in overload, translations do not exist in the NAT table until the router receives traffic that requires translation. Translations have a timeout period after which they are purged from the translation table.

    Example #1: Static Source NAT

    How to translate the 172.16.0.5 IP address to the 10.16.0.5 ip address.



    Define the ip nat inside:

    Router(config)#interface fa0/0
    Router(config-if)#ip nat inside


     Define the ip nat outside:

    Router(config)#interface fa0/1
    Router(config-if)#ip nat outside

    Define the static NAT entry:

    ip nat inside source static 172.16.0.5 10.16.0.5
     
    With static NAT, translation exists in the NAT translation table as soon as you configure static NAT command, and it remains in the translation table until you delete the static NAT command:

    Router#sh ip nat translations
    Pro  Inside global      Inside local       Outside local      Outside global
    ---  10.16.0.5          172.16.0.5         ---                ---
    Router#

    If the client sends an ICMP packet or an HTTP request to the web server, the nat table will be:
     
    Router#sh ip nat translations
    Pro  Inside global      Inside local       Outside local      Outside global
    icmp 10.16.0.5:1        172.16.0.5:1       10.0.0.100:1       10.0.0.100:1
    tcp  10.16.0.5:56080    172.16.0.5:56080   10.0.0.100:80      10.0.0.100:80
    ---  10.16.0.5          172.16.0.5         ---                ---
    Router#

    Remember: Because the mapped address is the same for each consecutive connection with static NAT, and a persistent translation rule exists, static NAT allows hosts on the destination network to initiate traffic to a translated host (if an access list exists that allows it).

    Example #2: Dynamic Source NAT 

  •  How to translate the 172.16.0.0/28 network in the 10.16.0.0/29 network.

    Define the ip nat inside:

    Router(config)#interface fa0/0
    Router(config-if)#ip nat inside

    Define the ip nat outside:

    Router(config)#interface fa0/1
    Router(config-if)#ip nat outside
     
    Define the nat pool used in the NAT translation:

    Router(config)#ip nat pool dynamic-ip 10.0.16.1 10.0.16.6 prefix-length 29

    Define which network will be translated:

    Router(config)#ip access-list standard client-list
    Router(config-std-nacl)#permit 172.16.0.0 0.0.0.15

    Define the dynamic source NAT:
     
    Router(config)#ip nat inside source list client-list pool dynamic-ip

    With dynamic NAT, translations do not exist in the NAT table until the router receives traffic that requires translation.

    Router#sh ip nat translations
    Router#

    but when some packets match the ACL..
     
    Router#sh ip nat translations                              
    Pro Inside global      Inside local       Outside local      Outside global
    icmp 10.0.16.1:2       172.16.0.1:2       10.0.0.100:2       10.0.0.100:2
    tcp 10.0.16.2:35694    172.16.0.2:35694   10.0.0.100:80      10.0.0.100:80
    tcp 10.0.16.1:56185    172.16.0.1:56185   10.0.0.100:80      10.0.0.100:80
    --- 10.0.16.1          172.16.0.1         ---                ---
    --- 10.0.16.2          172.16.0.2         ---                ---
    Router#


     Note: If a new packet arrives from yet another inside host, and it needs a NAT entry, but all the pooled IP addresses are in use, the router simply discards the packet.
    This can be checked enabling the “debug ip nat”.

     May 12 19:26:09.895: NAT: translation failed (E), dropping packet s=172.16.0.5 d=10.0.0.100

    The user must try again until a NAT entry times out, at which point the NAT function works for the next host that sends a packet. Essentially, the inside global pool of addresses needs to be as large as the maximum number of concurrent hosts that need to use the Internet at the same time—unless we use PAT.

    Remember: The main difference between dynamic NAT and a range of addresses for static NAT is that static NAT allows a remote host to initiate a connection to a translated host (if an access list exists that allows it), while dynamic NAT does not. You also need an equal number of mapped addresses as real addresses with static NAT.

    Example #3: Static PAT

    How to expose two different services on Internet:
  • The Web server (172.16.0.5) is listening on tcp port 80; this server responds on public address 88.88.88.88:80 from the Internet (outside).
  • The SSH server (172.16.0.6) is listening on tcp port 22; this server responds on public address 88.88.88.88:666 from the Internet (outside)






  • Define the ip nat inside:

    Router(config)#interface fa0/0
    Router(config-if)#ip nat inside

    Define the ip nat outside:

    Router(config)#interface fa0/1
    Router(config-if)#ip nat outside

    Define the static PAT:
     The web server responds on tcp port 80 on the ‘outside’ interface.
     
    ip nat inside source static tcp 172.17.0.5 80 88.88.88.88 80

    The SSH server responds on tcp port 666 on the ‘outside’ interface ; in this case, the real port (22 tcp) is translated to the 666 tcp port when a request comes from Internet.
     
    ip nat inside source static tcp 172.17.0.6 22 88.88.88.88 666

    Like static NAT, static PAT translation exists in the NAT translation table as soon as you configure static PAT command, and it remains in the translation table until you delete the static PAT command.
     
    Router#sh ip nat translations 
    Pro Inside global      Inside local       Outside local      Outside global
    tcp 88.88.88.88:80     172.16.0.5:80      ---                ---
    tcp 88.88.88.88:666    172.16.0.6:22      ---                ---
    Router#

    If an Internet client sends an HTTP request or an SSH Connection (on tcp port 666), the nat table will be:
     
    Router#sh ip nat translations 
    Pro Inside global      Inside local       Outside local      Outside global
    tcp 88.88.88.88:80     172.16.0.5:80      56.56.56.56:54686  56.56.56.56:54686
    tcp 88.88.88.88:80     172.16.0.5:80      ---                ---
    tcp 88.88.88.88:666    172.16.0.6:22      56.56.56.56:33704  56.56.56.56:33704
    tcp 88.88.88.88:666    172.16.0.6:22      ---                ---
    Router#

     Example #4: PAT – NAT Overload

    How to share an Internet connection.



    Define the ip nat inside:

    Router(config)#interface fa0/0
    Router(config-if)#ip nat inside

    Define the ip nat outside:

    Router(config)#interface fa0/1
    Router(config-if)#ip nat outside

    Define which network will be translated:

    Router(config)#ip access-list standard client-list
    Router(config-std-nacl)#permit 172.16.0.0 0.0.0.255

    Define the NAT Overload:
     
    Router(config)#ip nat inside source list client-list interface fastethernet0/1 overload

    Like dynamic NAT, translations do not exist in the NAT table until the router receives traffic that requires translation:
     
    Router#sh ip nat translations
    Router#

    but when some packets match the ACL..
     
    Router#show ip nat translations
    Pro Inside global      Inside local       Outside local      Outside global
    tcp 88.88.88.88:7921   172.16.0.2:7921    95.100.96.233:443  95.100.96.233:443
    tcp 88.88.88.88:8651   172.16.0.5:8651    173.194.44.18:80   173.194.44.18:80
    tcp 88.88.88.88:8652   172.16.0.111:8652  173.194.44.18:443  173.194.44.18:443
    tcp 88.88.88.88:8653   172.16.0.223:8653  173.194.70.84:443  173.194.70.84:443
    udp 88.88.88.88:64116  172.16.0.222:64116 8.8.8.8:53         8.8.8.8:53
    udp 88.88.88.88:64756  172.16.0.223:64756 8.8.4.4:53         8.8.4.4:53
    Router#

     Other types of NAT/PAT


     One type of NAT/PAT widely used is the ip nat outside source; this command permits to translate the source address of a packet that enter in the ‘outside’ interface and leave the ‘inside’ interface.
    In simple terms, if you see the first example #1



    The command:

    ip nat outside source static 10.0.0.100 192.168.0.100

    translate the 10.0.0.100 to the 192.168.0.100 , so the client must call the 192.168.0.100 ip address to contact the server web and not the 10.0.0.100.
    Another particolar type of nat is the ip nat inside destination used when multiple inside devices are identical servers with mirrored content, which from the outside appear to be a single server load balancing.

    You define a pool of addresses containing the real hosts’ addresses ending with “type rotary” making the servers available in round-robin fashion. The access list now permits the IP address of the virtual host, i.e. what the outside world thinks is the host address. So the virtual host is 123.123.123.132, with the real hosts being 172.16.0.2 through 10.

    Partial configuration
     
    interface FastEthernet0/0
    ip address 172.16.0.0 255.255.255.0
    ip nat inside
    !
    interface FastEthernet0/1
    ip address 88.88.88.88 255.255.255.252
    ip nat outside
    !
    ip nat pool real-ip-server 172.16.0.2 172.16.0.10 prefix-length 24 type rotary
    ip nat inside destination list 1 pool real-ip-server
    !
    ip route 0.0.0.0 0.0.0.0 FastEthernet0/1
    !
    access-list 1 permit 123.123.123.123

    This translation is not bi-directional in nature. You will have to use a one to one static NAT to accomplish it. A “ip nat inside source static” kind of functionality can be achieved with the above configuration using a single address in the NAT pool, but that would only work for outside to inside traffic.


    Where apply nat inside/outside?


    Typically “ip nat inside” is configured on the interfaces in your local environment which cannot be routed to the Internet (typically private range of IP Addresses) and  and “ip nat outside” on the interface which is connected to the Internet.

     When does the router perform NAT? 

     

      Inside to Outside:

  • If IPSec then check input access list
  • decryption – for CET (Cisco Encryption Technology) or IPSec
  • check input access list
  • check input rate limits
  • input accounting
  • redirect to web cache
  • policy routing
  • routing
  • NAT inside to outside (local to global translation)
  • crypto (check map and mark for encryption)
  • check output access list
  • inspect (Context-based Access Control (CBAC))
  • TCP intercept
  • encryption
  • Queueing

Outside to Inside:

  1. If IPSec then check input access list
  2. decryption – for CET or IPSec
  3. check input access list
  4. check input rate limits
  5. input accounting
  6. redirect to web cache
  7. NAT outside to inside (global to local translation)
  8. policy routing
  9. routing
  10. crypto (check map and mark for encryption)
  11. check output access list
  12. inspect CBAC
  13. TCP intercept
  14. encryption
  15. Queueing

Some Useful Commands

  • To see some statistics about NAT: show ip nat statistics
  • To see a complete list of the static/dynamic NAT/PAT entries: show ip nat translations 
  • To clear dynamic nat entry: clear ip na translation *
  • To debug NAT: debug ip nat                                                   

Sunday, May 3, 2015

Big Data - Impact on Enterprise Integration

In this article I tries to explains the new challenges in enterprise integration caused by the advent of Big Data, and presents some approaches to overcome them.



Big Data

 Big data is the latest buzz-word in the industry. At the basic level, big data is just that: large amounts of data that cannot be handled by conventional systems. With increases in hardware capacity, our definition of what constitutes a conventional system changes, and so does the threshold of big data, which is not something new but has always been around. It’s just that the threshold of what constitutes big data has changed. Today, the threshold for big data may be terabytes (1012 bytes). Soon, it will be petabytes (1015 bytes). Twenty years ago, there were very few systems that could process gigabytes (109 bytes) of data in an acceptable time-frame. So gigabytes would have been the lower threshold of big data at that time.

Integration

Integration is needed when we have to connect two or more software systems together. Large enterprises will have tens or hundreds of systems to connect together. Since the number of systems involved in integration could be large, the amount of data that flows across these connections will also be large.

How does big data impact enterprise integration?

With the advent of big data processors, more and more organizations recognize the value of big data analytic, and the need to process big data within enterprises is increasing. This will lead to large amounts of data (which is a subset of the big data captured by the organization) moving across integration middle-ware (a software that connects two or more software systems). Such large amounts of data would overload existing integration middle-ware systems, since they were designed to handle lower volumes of data. This is depicted in Figure 1.

Figure 1: Big data and integration

Let analyze Figure 1 in detail. Big data is not what is really interesting. Who needs a mountain of data anyway? What we want are the results of processing this data. Depending on the processing algorithm used to process the data, we get different results. This is where it gets interesting.

Let us suppose that an enterprise implements a big data solution. The big data processing (refer to the big data processor in the figure) solution could be a shared service across the enterprise. Due to the technical complexity and cost involved in building a big data processor, it is not possible for each division in the organization to have its own. But due to the business value accruing from big data analysis, sooner or later, different divisions in the enterprise will want their own processing on the big data set. This can be done by moving a subset of the big data, relevant to that division, over to its systems. This is the point at which the integration systems in the enterprise will feel the impact. To enable different divisions in the organization to “do their own stuff” with data, a subset of the big data will start moving across the integration middle-ware.

Overcoming the big data challenge

So how can we solve the new challenge caused by subsets of big data moving across the integration middle-ware? Let us look at three options:
  1. Buy more hardware: This is technically called vertical scaling or horizontal scaling. This approach may work with smaller challenges, but not with big data, since the amount of new hardware required to support the load will make the idea financially enviable.
  2. Buy specialized big data solutions: These solutions can be purchased and given to each division that needs a big data processor. There are very few big data processors in the market now, but we can expect more soon. This will be cheaper than buying more hardware, and can be considered, if the organization has enough funds. Note that this approach works by avoiding the problem altogether: we avoid moving any data across the integration middle-ware. On the flip side, this approach will result in multiple copies of big data in the enterprise.
  3. The third option involves extending our middle-ware’s capabilities by using data grids or distributed caching platforms. This attempts to overcome the big data challenge by increasing the integration middle-ware’s available memory, and introducing an asynchronous link in the integration middle-ware’s data-persistence mechanism.
How do these techniques help us overcome the challenge? Answering that question requires a deeper understanding of the root cause of the underlying problem. A middle-ware solution fails at high loads due to the following issues:
  • Memory overload, caused by data, threads or sockets
  • Lack of system resources like threads, sockets and swap space
A distributed cache helps with the first issue, by increasing the available memory. For example, if you have ten servers with 10 GB each, a distributed cache can help you add up all that RAM and use it like local RAM, effectively giving you 100 GB. It helps with the second issue by avoiding the need for a large number of threads, or a large swap space. This is accomplished by intelligent persistence mechanisms, like write-behind-cache.
In a write-behind-cache, data that needs to be persisted is written in an asynchronous manner: the write request is accepted, and the write function returns immediately. The persistence mechanism then writes into the file or DB; this frees up the persistence threads of the middle-ware, increasing the scalability of the overall solution.

 

A telecom use-case

Let us look at how the third option from the previous section, of using data grids, can be implemented. The use-case here is from a telecom scenario, and is depicted in Figure 2.


Figure 2: Big data and integration: A telecom use case

Figure 2 has conceptual similarity with Figure 1. The Network Switch is the data source here. The big data processor maps to the Mediation solution here. The Analytic Application is similar to the Data Warehouse. The Event Processor system is similar to the Fraud Management application. Let us understand the data flow in this figure.

This is the data flow for a cell phone services provider — a telecom company, or telco, as per industry parlance. Whenever a phone call is made, records called Call Detail Records are generated by the telco’s hardware, and the records get collated at the Network Switch. The Mediation system then processes these records. It performs validation, filtering, etc., and gives the records to the three systems it connects to: Fraud Management, Billing and Data Warehouse. The Billing system needs to connect to many other systems: CRM, Inventory, Fraud Management. For some of the data flows, like CRM to Billing, the volume is so high that we have to provide a direct connection from Billing to CRM, for a few use-cases (around 5 per cent result in such high volumes).
Normal middle-ware, even with clustering and load balancing, cannot handle this. This is where the need for Middle-ware Infrastructure comes in, in the architecture (see Figure 2). The Middle-ware Infrastructure component is a separate product that provides features like local and distributed caching, load balancing, fail-over and recovery, with much higher scalability than that provided by standard middle-ware products. Some of the middle-ware infrastructure products come in the form of data grids, which support scaling to hundreds of nodes. Examples of such products are Oracle Coherence, JBoss Infinispan, Websphere Extremescale and Terracotta Big Memory.

The future of Big data and integration

Any solution in the technology space starts out in a niche area, and as it becomes mainstream, it gets more and more commoditized. We can expect big data solutions to follow this path in the near future with:
  1. The arrival of big data processing appliances
  2. Support for big data in cloud platforms
  3. Cloud-based integration platforms that are per-packaged with middle-ware infrastructure.
Hopefully, this article gives you a good overview of the challenges posed to integration solutions by the advent of big data in enterprises. The example discussed, which is a use-case from the telecom domain, is generic enough to be applicable to other domains. The key value that open source brings to such solutions is that we can scale out our solution with much lower financial implications, compared to commercial solutions.