Multiple Data Feeds, Multiple Broker Connections, Universal Symbology

[Bruce DeVault]

This is a recap of several broader ideas and discussions in one thread: in the next version of NeoTicker, it would be great if multiple simultaneous broker connections and multiple simultaneous data feeds could be supported. To help them work better together, we need a universal symbol list, such as that it's possible to define that "ES" (or some simple variable, such as is used for local continuous symbols) is a tradable symbol that's called "ESD_F" when backfill server data is needed, called "ES #F" when eSignal historical or real-time data is needed, and called "ES X10" etc. when orders are placed to a specific broker. There should only be one price multiple, minimum tick size, etc. for the entire universal symbol, but there should be unique order placement symbols and margin requirements for each broker connection. The universal symbol should define the order of precedence for where to get historical data e.g. it might specify to get tick data for ES as ES #F from eSignal as far back as it is available, then IQFeed for as far back as that goes using @ES#, and then fall over to ESD_F from backfill server after that. It is understood that only one real-time data source could be feeding a given symbol such as "ES" at any given time, at least in this next implementation, but it's important that we should be able to receive real-time data from multiple sources at the same time for different symbols - for instance, to get some symbols from eSignal and some from Interactive Brokers if connected to that order server.

It is understood that TickQuest can't take on the burden of figuring out every broker's symbology for every symbol, or dealing with it when a broker changes a symbol (this happens actually all the time, despite that individual users may not be aware of it because they're focused on specific symbols). That would have to be maintained by the user - TickQuest would simply provide a few reasonable examples such as perhaps one in each instrument class e.g. an ES example set up for eSignal, IQFeed, IB, MB Trading, OpenECry, Backfill Server, and NinjaTrader order server; and, an EUR/USD example for eSignal, IQFeed, IB, MB Trading, FXCM, Backfill Server and NinjaTrader order server.

Separately, though, if a data feed does provide symbol info, such as price multiples or minimum tick sizes, there should be an option to allow this information to be populated automatically in symbol info manager if it's currently blank.

It should also be possible to connect or disconnect data feeds without restarting the NeoTicker platform - having to completely restart the platform because of a need to switch for instance to offline mode to pull backfill server data instead of being connected to eSignal is something that would be great to not have to do. Connecting or disconnecting a data feed should be a non-event and not disrupt anything, with the exception of the optional recalculation.

[pippi]

That would be really useful. I find myself often switching between 2 datafeeds.

[gooni]

This would be absolutely amazing to have and would completely remove the need for me personally to use other platforms (such as Ninja) for certain tasks.

[han]

Visionary. But a practical vision that would address real life decisions I've had to make about what broker(s) to use.

[deep]

Is this really possible ? It would be just amazing if it could be practically achieved. It would give immense freedom to the end user for his/her data needs. Right now I feel that Neoticker is very limiting in this particular segment, whereas it is in its own league in most of other segments of trading platform.

Thanks Bruce for the brilliant idea ! As han says it, it is truly Visionary

Hopefully we would see some positive remarks from Laurance on this suggestion.

[Bruce DeVault]

It's certainly possible. There are some (myself included) who've had to "rig" multiple connections already - it just needs to be supported at the platform level out of the box, as it is in some other platforms for a few years now. Right now, some key limitations are the lack of a universal symbol list to tie it all together, so that the symbol being analyzed is no longer a symbol for some specific feed, and the logic of how the platform restarts when you change data feeds - once those have been addressed, the rest is as they say a simple matter of programming. I'm confident this can be done and should be done, and I have no doubt TickQuest can do it right.

[Clidona]

This is a dream feature.

A few questions:
1) Although real-time data for charting etc will be coming from one data feed for each symbol, is it feasible to still collect data from all data feeds? Even minute data is slightly different between data feeds and tick data is bound to show even more discrepancies. What would be ideal is to extract and save data for all data feeds and base the priority for building a longer data series based on whatever data has been extracted
2) What is the suggested way to handle naming for symbols like MSFT, which have the same meaning across multiple data feeds?
3) My understanding is one of the main reasons to handle a few data feeds simultaneously (as opposed to switching between data feeds without restart) is to have "fall-back", when a data feed that "breaks" can be automatically replaced with a working one. Is this explicitely excluded from the suggestion?
4) Does this suggestion include being able to send orders simultaneously to different brokers?

[pippi]

Originally Posted by Clidona

Although real-time data for charting etc will be coming from one data feed for each symbol, is it feasible to still collect data from all data feeds? Even minute data is slightly different between data feeds and tick data is bound to show even more discrepancies. What would be ideal is to extract and save data for all data feeds and base the priority for building a longer data series based on whatever data has been extracted.

Yes, even more so in otc markets where discrepancy in price is expected to some extent, and just this discrepancy might be a point of interest (i.e. to compare pricing from different dealers). In this case a mechanism to store data for the same security from different sources seperately would be useful too.

[Bruce DeVault]

Under this proposed scenario, you could receive data from more than one feed for the same symbol at once by creating a different universal symbol for each feed. For instance, if you think there's an arb opportunity between eSignal and IQFeed for MSFT, define a universal symbol eSignal:MSFT that only pulls MSFT from eSignal, and a universal symbol IQFeed:MSFT that only pulls MSFT from IQFeed. Then, spreads could be constructed, bollinger bands around the spread, etc. For those interested in spread analysis or short-term arb opportunities, this is exactly how you would do it.

Failover is a good thing, and doesn't violate the principle that only one feed be receiving real-time data into a given universal symbol at a time. In this case (as distinctly separate from the arb setup described above), the universal symbol would be MSFT and it would define MSFT from eSignal as first priority, then from IQFeed as second priority etc. This is separate from capturing a separate real-time stream from two data feeds at the same time - that would require two universal symbols in this proposal. Please understand the whole point of doing this suggestion in this manner was to structure it in a way that was achievable - if you must have streaming both real-time feeds into the SAME universal symbol at the same time, it becomes more difficult to short-term achieve and requires different structures than we have now, whereas, this is actually something that could be done without a lot of structural change.

Yes, you should be able to send orders to multiple brokers simultaneously. All that's needed a new hook on the Trade functions to allow the account routing to be specified by order or if necessary, a short-term convention such as an account number and colon prefix on the comment.

[mikesch]

Maybe you could have multiple Data Caches, each cache would be captive to its own data vendor. Time charts and other windows could read individual data items from multiple data caches simultaneously, mixing symbols as required for display.

Symbology could be the individual vendors native symbology while getting the data captured off the wire and into the vendor's Data Cache. The centralized symbol translation would be employed when the data was requested to be pulled out of cache and placed into a chart or window.

I have no idea if this is architecturally feasible in NT, just my 2 cents.

Cheers
MS

[Bruce DeVault]

While I don't want to prejudge what TickQuest should do, I think personally that there's a lot of benefit to be had from having a universal symbol and a single central disk cache. In particular, this makes it possible to do things like pull real-time data from one vendor but historical data from another, or to use one vendor's historical data for the first X days and another after that, depending on which lookbacks each has available.

[Bruce DeVault]

There's a downpayment on this functionality in the new 4.2 build 62. Looking forward to seeing this fleshed out in the upcoming 4.3 release. Thank you TickQuest! These features when fully implemented will really take NeoTicker to the next level.

[Clidona]

I may have missed this while reeading the proposal...

If the data source to be used with a particular symbol is not defined, will the symbol data be extracted from:
1) Datafeed 1
2) From all data feeds in a pre-defind order till teh data is found
3) In some other way

Also, is there a chance, when this functionality is implemented, that Neoticker will be able to connect to more than 2 data feeds simultaneously?

[Bruce DeVault]

Eventually, it should be possible to connect to more than one data feed simultaneously, but as a first step, it may be that it's something more like a data feed + order server e.g. you can get eSignal symbols + IB symbols + backfill for realtime + historical vs. now where you have to choose one of the above. This is still a work in progress as TickQuest has articulated - we'll know more when 4.3 is released and the intended architecture is better visible.

[Clidona]

I guess will have to wait and see what TickQuest come out with. Any progress in this direction is highly welcome.