The strongest way to ensure portability is a deterministic, frozen, non-extensible execution model. Every host runs exactly the same computation on the same image and input, for all time. When you move that image, the only thing it notices is that its IP address has changed.

The ZeroMQ binding can be combined with Ring Buffer Channels and the LMAX Disruptor to create low-latency processing engines. Consider the following topology where a consumer receives events from one channel, processes them and subsequently invokes another service that in turn publishes output to a second channel:

A message is read from a ZeroMQ socket on thread 1 and placed in a ring buffer (Disruptor) slot. The business logic runs on a second thread, which receives the message from the ring buffer (Disruptor). When processing is complete, a business logic component publishes the output event to the second channel which results it being placed in an outgoing ring buffer slot. A third thread responsible for dispatching messages over a ZeroMQ socket receives the output event and publishes it.

This channel architecture can be applied to a number of use cases which require fast (low-latency) and predictable (limited GC activity) performance.

+ Table of Contents
Introduction

The financial sector lives off messaging technology. On "Wall Street" (the global stock trading business), capacity and latency are everything. Current infrastructure, highly tuned to get million-message per second throughputs, and sub-millisecond latencies, still fails when trading gets frantic. Huge amounts of money depend on being the first to get data, and the first to trade.

The stock trading business is evolving dramatically. Penny pricing generates more data. New US and EU regulations increase the number of parties involved in financial markets. New algorithmic trading technologies increase the demand for up-to-date stock data and icrease number of orders. While the existing infrastructure can double in capacity or speed per 18 months, traffic is expected to grow by 20 times over the next three years1.

In the same time, prices for messaging technology are steadily rising. Messaging middleware - software that connects applications or pieces of applications in a generalised plug-and-play fashion - is one of the last big-ticket items still not turned into a commodity by the Internet age of cheap software.

Mainframes got much of their power from clever messaging, transaction processing systems like IBM CICS. But today even 1980's-standard middleware - unlike databases, operating systems, compilers, editors, GUIs, and so on - is still not widely available to ordinary developers. The software industry is producing various business applications and pieces of applications, and the tools to make these, in ever greater quantities, and ever lower prices, but the messaging bit is still missing. The lack of a way to connect these applications has become not just an unconquered terrain, but also a serious bottleneck to growth, especially for new start-ups that could in theory compete aggressively with larger, older firms, if they were able to cheaply combine existing blocks of software.

This frustration is visible in many markets and has lead to the growth of messaging-over-HTTP (SOAP), and other compromises. Architectures like SOAP do work, but they don't solve the two main issues of a enterprise-level messaging, namely routing and queuing. Thus businesses who use such technologies cannot scale, and cannot compete in really large markets, unless they write their own messaging software, or buy a commercial product. Various other standardisation attempts were made to commoditise the market: CORBA, JMS and lately AMQP, CORBA being unsuccessful because of RPC metaphor that doesn't suit the needs of financial markets, JMS succeeding in Java world, but unable to expand any further and AMQP still being a big unknown.

The increasing demand, and lack of real competition shows in the financial statements of high-end messaging vendors like Tibco Software Inc: "Total revenue in the first quarter of fiscal year 2007 compared to the same quarter last year increased by $11.0 million or 10%. The increase was comprised of a $7.0 million or 11% increase in service and maintenance revenue and by a $4.0 million or 8% increase in license revenue."2 Tibco customers report that license fees are increasing, year on year.

The market

The global stock trading market is primary focus of ØMQ because that's where most emphasis is placed on messaging, most resources are accumulated and most edge-cutting technologies are used.

The main characteristic of the market is hunger for fast delivery. Every millisecond the stock quote or the trade order is faster than the competing one translates into direct financial profit, so the firms involved are naturally eager for any advantage they can get.

Currently, in the stock-trading business traffic load is so high and latency so critical, that the middleware has to be highly optimised. Latencies are given in microseconds and throughputs in millions of messages per second… In spite of that, trading often experiences problems when message load peaks. Latency can suddenly drop to seconds (or even tens of seconds) and huge amounts of money can be lost as trades are delayed or fail.3

The situation is getting worse for several reasons:

In 2001, the NYSE and NASDAQ switched from pricing their stocks in 1/16th dollar units to single cent units. This so-called "penny pricing" means stock markets produce more data and this data must be shifted across networks.
Both in the US and EU, regulators are forcing financial markets to compete more openly and aggressively, in the interests of consumers. For example US SEC regulatory changes allow new firms to act as intermediaries in the stock trading sector while the EU's Markets in Financial Instruments Directive (MiFID)4 is expected to increase stock-trading traffic rates in EU to match the volumes seen in US after Reg NMS5.
Many new and aggressive firms are entering the market, especially building or using 'algorithmic trading' platforms.
Algorithmic trading executes big amount of low-volume orders as opposed to small amount of high-volume orders executed by traditional human traders.
So we have increased data flows, to more participants, who are pushing to develop new business models which depend on getting that data rapidly, detecting temporary market anomalies, and responding to it (with trades) before their competitors. A more flexible regulatory environment is opening previously protected markets to new competition. Overall, we see an arms race for bandwidth and latency in which better technology translates directly into more profits.6

Message traffic is expected to grow significantly in the near term - we have heard different figures of up to 30 times over the next three years - and existing systems can only double capacity every 18 months.

There are many attempts to solve this emerging issue. The most dramatic improvements in performance come from replacing the classic central broker with a peer-to-peer architecture in which messages can flow directly across the network with no extra hops. Not all messaging systems can adapt their architecture in this way.

Apart from architecture, the obvious place to optimise messaging is in the "stack", i.e. the layers that separate the application program from the physical network. The software itself is already heavily optimised in most cases, so vendors are shifting to other options, such as:

Optimising network architecture by connectivity providers to get better latencies, including moving message consumers close (in network terms) to the message producers;7,8.
Clients moving from consolidated stock quote feeds to direct connectivity to the exchanges;9
Optimising formats in which data are passed (FIX/FAST10);
Providing full-blown hardware solutions (Tervela, Exegy, etc.);
Replacing the physical transport layer (Infiniband11, 10GB Ethernet);
Optimising existing networking hardware. TCP offload engines, Intel's I/OAT technology;12, etc.;
Modifying the OS to handle the messages in real-time. Various real-time OS's, like Novell's SLERT13;
Modifying the OS to use more efficient messaging stack: Asynchronous I/O, SDP, various zero-copy techniques etc.;
Using multicast to distribute stock quotes on the client's LAN;
As well as these optimisations, which focus on individual aspects of the messaging stack or architecture, we also see attempts that look at the problem as a whole:

Intel's Low Latency Lab14
Securities Technology Analysis Center (STAC)15
Various measurement & monitoring solutions (Endace etc.)
Highly optimised products with extensive hardware support become very expensive. Only the largest trading firms can afford the full range of products and even for these firms, costs remain a persistent concern. For the smaller firms, many of the solutions are simply not an option.

Opportunities

In this section we look at the opportunities for new high-performance messaging products such as those we are building.

High-performance take-out

The first and most obvious target is any firm using high-end commercial middleware for stock trading, where we can provide a cheaper equivalent. This market is cost-sensitive and in our experience it is willing to absorb change and risk in order to get a compelling price and/or performance advantage over their competitors.

Further, there are many firms who cannot afford these products, but would use them if the cost was set lower. Zipf's Law (usually used for language but also applicable to business sizes) suggests that the number of firms and their size follows an inverse power ratio, so offering a product at 20% the price of the high-cost market leaders should open a market five times as large. (In fact it's probably not this large, because smaller firms will buy or rent trading platforms rather than try to build their own.)

Trading platforms

Trading platforms are software applications that trading firms can buy ready-made, rather than build themselves using messaging middleware. Given the demand for cheaper, faster trading, there is a large market for these platforms. Obviously a firm that builds a trading platform is sensitive to the cost of the messaging it uses and these firms provide a market for our planned products.

Investment banks

Investment banks build their own trading systems and (from our limited experience) like to have control over the technology they use. Standards-based systems are highly attractive here. The calculation is that a standard technology is easier to control, and is served by a larger market of cheaper experts. Any AMQP solution has immediate attraction. Cost is always a driver as well but for firms that do significant development around the messaging, reduction of secondary costs (such as the number and cost of in-house consultants) is an important aspect.

It becomes clear why JPMorganChase was motivated to push and invest in the AMQP process, even taking considerable risks at the time: AMQP enables very large savings on IT expenditure, for messaging licenses, custom development, operational control, and so on. We can deliver a much lower-risk proposal to other investment banks, but with the same kinds of benefits.

Data consolidators

The stock trading world connects many exchanges (NASDAQ, NYSE, etc.) to many clients. Large clients make separate connections to each exchange, but most work via data consolidators, firms like Reuters who provide unified streams from many sources.

Today's consolidators run highly-tuned custom messaging software, it is not standards-based, and has little scope for getting cheaper and faster. It can get faster, but only at high cost, which punishes those firms that stick with custom messaging, and gives an advantage to those firms using standards-based messaging, which spreads the costs and leverages far more work on performance.

There is a definite opportunity for opening this market, and allowing new firms to compete as data consolidators, using our high-performance products to carry quotes to clients. New US regulations are opening this market to real competition.

Exchanges

The exchanges (stock exchanges, currency exchanges, commodities, etc.) are heavily impacted by the growth in demand for their services. It seems inevitable that standards at the edges will slowly force their way into the center, and we should be able to follow with product offerings.

Also, new types of trading venues are emerging (ATS's, MTF's and dark pools16) that gradually take still greater share of the market from the traditional exchanges. Given that this trend is quite new and still gaining momentum, we expect to see increasing demand for high-end messaging systems on this market.

Moving the value to different markets

One of the goals of ØMQ is to use money, resources and experience accumulated during low-latency arms race in stock-trading business to deliver free high-end general-purpose messaging solution to the rest of IT sector.

Some of the areas where ØMQ may prove useful follow.

Business and institutional messaging

Sending payments, doing business-to-business communication, passing documents within governmental organisations etc. is the primary market to focus on apart of stock trading. The reason is that this is the field where messaging is used traditionally, with lot of experienced IT personel aware of messaging and using it for a long time.

It should be also taken into account that even applications that don't use messaging proper may be still sending 'messages' by different means. Consider an application located at place A writing a record to remote database server and another one at place B reading the record. In fact, there was a message sent from A to B, even though the programmer might not be aware of it. Even inter-process and inter-thread communication can be considered messaging. Synchronising different applications by copying files to remote destinations once a day can be considered messaging as well (although it is a spectacularly low-latency one).

Basically any application made for financial or institutional sector needs some kind of messaging and the cost of the implementation varies between 10 and 30 per cent of the total project cost, so using existing standards-based middleware implementation seems to be a rather good investment.

Although low latency is not a key requirement in this sphere, we expect that growing transaction rates (consider regulations like EU's SEPA17 and standardisation efforts like TWIST18) will slowly force financial institutions to adopt high-performance messaging solutions, thus causing the current small slice of the messaging market addressed by high-performance solutions will steadily grow, until it ultimately reaches 100%.

Embedded systems

Embedded systems often have real-time requirements similar to those seen in stock-trading business. Consider, for example, an equipment measuring some critical value in a technological process. The data have to be delivered to the unit controlling the process within 1 ms, otherwise the whole process will be spoiled.

Embedded systems don't usually need the throughput provided by stock-trading stacks, however, if the latency, reliability and deterministic delivery times are guaranteed, they can take advantage of it, even though it doesn't use all the bandwidth capacity available.

Multimedia

Same remark about real-time requirements applies to multimedia (streaming audio and video, teleconferencing, etc.). As opposed to embedded systems, latency is not that critical, the paramount being deterministic delivery time and high throughput.

In the future we may find out that lot-of-small-messages model of stock-trading apps is incompatible with stream-based multimedia approach. However, we don't believe this is the case. To test the hypothesis, we've built proof-of-concept teleconferencing application over AMQP and we've seen it perform smoothly.

Grid computing

Having almost the same requirements as stock trading, grid systems are natural area to employ ØMQ stack.

Grids are icreasingly being used in financials19 and - not surprisingly - in stock trading itself, providing a solution for computationally expensive problems like risk management and algorithmic trading20.

The low-latency bubble

The market for low-latency solutions is very lively and expanding these days. However, some have a feeling that the value of the market is overestimated and that low-latency arms races going on will result in the burst of the bubble, similar to dot-com crash of early 2000's.

Let's examine possible causes of market breakdown:

There are law's of physics that place lower bound on the latency. Specifically, speed of light cannot be exceeded and once the messaging hits this limit, there won't be much space for competition and low-latency arms race will come to its end.
The costs for fast messaging are constantly growing. Once we hit the point where improving the latency will require investments exceeding the profits it can possibly yield, the flow of money into the market will end.
Unreasonable spending on low-latency solutions can result in hysteria, once the still growing low-latency market starts shrinking. Hysteria can make the market plummet even below it's real value.
Our view of the problems above is following:

Speed of light is certainly an ultimate barrier, however, as can be seen with microprocessors, barriers seen as ultimate are quite prone to be crossed over and over again. In messaging business for example, we see emerging proximity solutions (handling speed of light problem by placing interdependent applications physically close one to another) or the trend to optimise software part of the messaging stack thus removing endpoint latency rather than on-the-wire latency. In fact, we don't believe there are any real unpenetrable barriers to stop low-latency arms race at least in the next several years.
Although costliness of the low-latency messaging grows steadily, it should be taken into account that technology price - both hardware and software - is steadily decreasing at the same time. What cost $100 last year, costs $50 today. So, even in stable, non-expanding market, where spending on IT keeps constant, there will be a demand for new solutions to keep pace with new technologies.
Hysteria can happen at any time and there's no way to prevent it completely. However, as stock-trading messaging is in a way a world for itself, we expect hysteria to be restricted to this turbulent little market leaving the rest of messaging market intact. Thus the main victims will be the firms that provide specialised stock-trading solutions rather than general-purpose messaging. Specifically, ØMQ project, by taking advantage of the resources accumulated in stock-trading-focused IT market to develop general-purpose messaging solution can survive market breakdown by relying on its presence in different sectors of messaging market.
Conclusion

The primary focus of ØMQ starts with stock trading because this market has a well-defined and growing demand for high-end solutions, and the options for collaborations and return on investment are plentiful. However, the construction of a cost-efficient, standards-based messaging system that can compete head-on with the best in the world opens doors into many other domains as well.

Counterparty in a "holding pattern" implementing Smart Contracts because Ethereum tech "not ready for prime-time."