|
Laughable Web Services
I'm currently writing up the technical specifications for some simulation software that needs access to real-time stock-market pricing information (20 minute delayed won't cut it), from all of the major US markets (NYSE, NASDAQ, and AMEX).
I understand that such a service may be expensive. And I'm willing to pay for it.
(I should mention that I'm not just writing the specs. I'll also be writing the code, and managing all aspects of the project. If it becomes successful, I may hire someone to help out later. But for now, it's just me.)
Anyhow, I start looking around for a datasource. I'm pretty familiar with the Apache SOAP toolkit, so it makes sense to see if I can use SOAP web services as a real-time datasource.
There are a handful of websites that list publicly available web services (including free & fee services). Checking out these lists, I was really disturbed by the plethora of useless web services and the dearth of genuine, useful services. Check these descriptions out:
* Returns a random word in English and first corresponding image from Google image search.
* Morse Code Translator WebService
* Returns a random quote from George W. Bush
* Displays legal and illegal Icelandic names, first, second and middle names.
* Is that famous person dead or alive?
* Generate globally unique ID with timestamp tracking
* This Web service takes a string of numbers (separated by a space) as input and allows the user to perform various statistical functions using this string of numbers.
These web services are so laughable, it just kills me.
The category of web services that I think is the silliest are the ones like the GUID generator and the statistics package. Why would anyone who knew how to use the SOAP framework bother with a web service for statistics? Why not just install a statistics library?
(And somebody should warn Aðalbjörn Hámundur. He may not know it yet, but he might have an illegal Icelandic name.)
Anyhow, I'm not satisfied with the results I've found. The supply of real-time-financial-market-data is very very thin. There are about six dozen different web services that can return 20-minute delayed data (they just scrape the data from http://finance.yahoo.com and return it in a SOAP message; it's pathetic), but I have not yet found any services that provide comprehensive real-time data, direct from the market floor.
The company I work for uses SOAP messaging to communicate between the client/server in our own software, but none of our software actually consumes external web services.
Have any of you actually written software that consumes an external service?
And if so, can you tell me: is Liza Minelli dead or alive?
Benji Smith
Wednesday, February 25, 2004
You haven't found any real-time stock quote feeds because the SEC doesn't allow it.
Ron
Wednesday, February 25, 2004
Ron, there are several web sites trying to sell such a service that disagree with you.
They might be lying, I don't know.
A quick Google Search turned up this one along with a few others.
http://www.dtniq.com/template.cfm?navgroup=aboutlist&urlcode=110&view=1
Steve Barbour
Wednesday, February 25, 2004
Most of those "services" are probably just people playing around with web services for the first time, or experimenting with a toolkit, and they just left them live. They're demos, not intended to be actually useful for anything.
Chris Tavares
Wednesday, February 25, 2004
You might want to try Service Objects:
http://www.serviceobjects.com/products/dots_fastquote.asp
I used their tax service and it was a real timesaver.
Sassy
Wednesday, February 25, 2004
> All Equities and Futures data is delayed according to exchange rules. NYSE and AMEX at least 20 minutes. Nasdaq at least 15 minutes. CME and CBOT at least 10 minutes. NYBOT, NYMEX and COMEX at least 30 minutes.
Is this as realtime as it gets?
--
ee
eclectic_echidna
Wednesday, February 25, 2004
most definately not. many brokers offer real-time data, though probably only for interactive users.
you could subscribe to a service like this:
http://www.tradestation.com/fees/subscription_op.shtm
they may have a web service.
mb
Wednesday, February 25, 2004
You're looking in the wrong place...
The "15 minute delayed" quotes, which are delayed 20 minutes, are generally free.
Live quotes are available from many sources, but you pay through the teeth.
Try S&P Comstock (http://www.comstock-interactivedata.com/index.shtml) or Reuters. Or look in the back of this magazine:
http://www.wallstreetandtech.com/
The data comes in way too fast an furious for SOAP. Anyway, the data is push, not pull, so SOAP doesn't really cut it. Most of the vendors on Windows give you a DDE server. Since it uses Windows messages for IPC at the lowest level, it's screamingly fast, which it needs to be to keep up with today's ticker.
Joel Spolsky
Fog Creek Software
Wednesday, February 25, 2004
As Joel said, check with the finacial houses theses things exist, but are spendy.
Mike
Wednesday, February 25, 2004
" it's screamingly fast, which it needs to be to keep up with today's ticker."
True. Also interesting was the huge amount of bandwidth gobbled by changing to 1/100 of a dollar movements.
Mike
Wednesday, February 25, 2004
Consider Comsfock, Reuters or Bloomberg; They all provide streaming data without delays straight from the trading floor. It will cost you, be prepared.
DDE is fast only when compared to Web Services; It's dog slow compared to the native interfaces provided by those services (Reuters' SSL (no relation to secure socket layer), and Bloomberg's COM objects and low level API) - DDE does buy you relative standardisation, though.
If you're looking for a database/processing platform that already interfaces with these services and can also do with more, you might consider contacting Kx systems (who'll probably refer you to First Derivatives - I'm not sure who talks to customers), or TimesTen. Be prepared to spend a lot of cash. [This is not a recommendation, I haven't used either - but both can probably offer you most of what you're looking for, and -- who knows -- the price may be right for you]
Ori Berger
Wednesday, February 25, 2004
Unfortunately, I'm not looking for every bid/ask that funnels through the market. I'd just like a summary for each minute in the trading day, including the open/high/low/close/volume for each minute.
And I don't need it for every security traded on every market.
I'm building a paper-trading simulation service, so I only need data for securities which my customers will have an open order.
So, if I had 500 customers, and each of them had two orders outstanding, there would be 1000 orders. And, since many of the users would probably be placing orders on the same securities, I would probably only need quotes on 400 securities.
I would want summary data for each minute, but I don't need to _retreive_ it every minute. I would be happy to fetch the data every 3 minutes.
I already have a co-loaded server running with an ample supply of bandwidth to PULL these types of requests. But there's no way I could handle a pipeline of data being PUSHED at me directly from the market floor. I really need someone in the middle to act as an arbiter of the data, sending me only what I need, based on the dynamic requests of my clients.
Still looking.........
Benji Smith
Wednesday, February 25, 2004
Are you sure the expense of real-time data is worth it for a paper-trading simulation? There are professional and non-professional fees you must pay to each exchange (AMEX, NYSE, NASDAQ) for real-time quotes. Since you are running a business, you will need to pay the professional fees, which total about $180/month. This does not include the fees charged by the service provider.
If you still want real-time data, DTN is probably the cheapest.
http://www.iqfeed.net/
The more expensive data services generally have cleaner data. You get what you pay for (usually).
magoo
Wednesday, February 25, 2004
Ori, I'm interested in your claim that DDE is slower than "native" COM --
The COM calls are likely to cross a process boundary, which means they will need to be synchronized, which uses Windows messages at the lowest level for the interapp synchronization. The bottom line is that you have to wait until you get back to the message loop to hear about the trade, and DDE is just Windows messages.
Of course, you could use a free-threaded COM object and accept calls at any time. But the only good way to write a free-threaded server is to receive the call and then throw yourself a message so that you can process the call at some intelligent time in the future. Now we're back to Windows messages again.
There seems to be a mistaken belief that interapp communication using DDE is "slow" while COM is "fast" but the bottom line is that in almost every reasonable Windows scenario, you have no choice but to do some interprocess synchronization, which is best done with Windows messages, which is all DDE is in the first place.
Joel Spolsky
Fog Creek Software
Thursday, February 26, 2004
I think people think that DDE is slow because they watched it used with live data in Excel spreadsheets and they somehow imagined that the Excel recalc slowness is coming from the DDE. While they think that COM is fast because at the source code level, it LOOKS like a function call. What could be faster than a function call?
I also should add that I haven't benchmarked this since the mid 1990s, and for all I know the vendors' DDE implementations are awful while their COM implementations are better and so even though DDE should be faster, in practice, you can't tell until you benchmark.
Benji, you need real market data from a real market data vendor. There are dozens of market data vendors and they're very competitive and they'll give you exactly what you want in exactly the format you want. With lox and cream cheese, no extra charge. But you need to talk to the vendors, not us.
Joel Spolsky
Fog Creek Software
Thursday, February 26, 2004
Bloomberg's COM API is, if I recall correctly, a single threaded, in process COM object that interfaces to their native C library, able to have zero call overhead. Don't remember how it calls back - I don't think it uses connection points, but I'll check.
Bloomberg's DDE server uses the same native C library, and adapts it to communicate through DDE.
I have a program that does essentially the same thing, and is capable of using either interface at the flick of a command line switch, and using DDE the computer's overall CPU usage (DDE server + DDE client) is ~10%. Using the native C api or the COM api, it is not easily measurable (something like 0.05% or so - I don't have a reliable tool to measure this).
It's probably DDE to blame, rather than Bloomberg or my code - profiling my code indicates that most of the time is spent in DDE; Perhaps I can tweak that a little (e.g., not use DDE string compare, and instead copy the string out and use hashing), but using "idiomatic" DDE, it is, comparatively, very slow.
It seems to me that many DDE (or at least DDEML) calls are synchronous. This can explain a lot if it's true. (Is it?)
Ori Berger
Thursday, February 26, 2004
DDE is by far the lightest and fastest way of moving data from one application to another, how fast either end processes that data is something else.
I have DDE servers that provide business type objects from an accounting system; most often they're used for bulk processing of data, such as repricing the entire stock system or producing debtor's letters in a high volume sales organisation.
Simon Lucy
Thursday, February 26, 2004
>Unfortunately, I'm not looking for every bid/ask that
>funnels through the market. I'd >just like a summary for
>each minute in the trading day, including the >open/high/low/close/volume for each minute.
>And I don't need it for every security traded on every market.
I did this in 1987.
It was for a box that the company leased to dealing rooms in half a dozen cities (matching the markets). Back then, it was Z80 and RS232 serial and parallel (Reuters). The system had a matrix that mapped customers to services and created a feed for each. Another box took the feeds and aggregated them to provide the summaries.
This was for Futures, Options and Equities from live feeds from Reuters Dealing & Banking, Telerate et al.
Justin
Thursday, February 26, 2004
How many securities or contracts do you want to monitor? If it's only a few there are numerous, what I'd call "second tier" data providers that are much cheaper than Reuters or Bloomberg. eSignal is the first that comes to mind. They can typically provide realtime quotes on <500 instruments. But in my experience, if you want to keep up with several thousand, you're just going to have to pay Mike Bloomberg a couple thousand a month. (Just picking on Mayor Mike...Reuters is good too.)
I have an app that grabs the last price for >3000 securities from the Bloomberg API. It usually takes about 60 seconds to get them all.
And Liza Minelli is very much alive. She was just in the news for beating up her ex-husband or something like that...
Rob VH
Thursday, February 26, 2004
Benji -
"Anyhow, I'm not satisfied with the results I've found. The supply of real-time-financial-market-data is very very thin. "
You're not looking hard enough. There are a numerous of real-time data sources out there.
http://www.quotetracker.com/qsources.shtml
As you'll find out, most of the data providers/brokerages provide connectivity not by means of webservices, but mostly through their proprietary API's.
To generalize your programming effort, I would take a look at using QuoteTracker ( http://www.quotetracker.com ) and it's HTTP interface. QuoteTracker has the capability to connect to a ton of providers and brokerages, and all of the quote data is exposed through their HTTP server.
http://www.quotetracker.com/qtserver.htm
You can add/remove symbols for tracking, request sales data, etc. all via a series of HTTP requests.
So no matter what data source you utilize within QT, you will always be able to access the data in the same manner. If you decide to switch providers, there won't be any redevelopment needed on your end.
But before you start developing your papertrading service, take your programming hat off and put your business hat on. The SEC strictly prohibits the redistribution of real-time quotes. When you sign up for any data service provider or brokerage, you will have to agree to this. This isn't something specific to the vendor, these are agreements handed down by the SEC, NASDAQ, and NYSE. You will sign them not matter what vendor you go with.
Real-time distributatble quotes are NOT cheap. You will likely pay a flat montly rate along with a request fee per quote. There is a reason why you don't see any services that provide real-time stock data to the public; it's too costly.
Thinking Hard.
Thursday, February 26, 2004
Hehe,... as an Icelander I'm proud that one of our services made your list :-D
Gauti (legal Icelandic name, believe it or not)
Thursday, February 26, 2004
http://www.xignite.com/
pb
Thursday, February 26, 2004
Joel said:
"Benji, you need real market data from a real market data vendor. There are dozens of market data vendors and they're very competitive and they'll give you exactly what you want in exactly the format you want. With lox and cream cheese, no extra charge. But you need to talk to the vendors, not us."
Actually, the whole point of this message wasn't originally to ask for help finding web services. I know about many of the providers of financial data (like reuters, comstock). I had already sent information requests to them, but I thought I'd look around in the web-services world to see if I could find anything there.
The central point of my post was just to point out how many weird useless web services are floating around out there.
Benji Smith
Thursday, February 26, 2004
The truly ironic thing is that the vast majority of the mildly useful web services are just wrappers around a dumb screen-scraping perl/VBScript script.
All the fragility of screen-scraping that Web Services were meant to avoid compounded with all the overhead of SOAP.
Richard P
Friday, February 27, 2004
---"The central point of my post was just to point out how many weird useless web services are floating around out there. "----
'Weird' and 'useless' are not synonyms. What you should be saying is
---"The central point of my post was just to point out how many web services are floating around out there, that I don't know enough to understand the purpose of."-----
Stephen Jones
Friday, February 27, 2004
Someone added a link to our web site (www.xignite.com). While we do provide the usual laughable-soap-wrapped-around-scrapper free quote service (www.xignite.com/xquotes.amsx), we also provide a realtime web service based on a direct link to Island (www.xignite.com/xrealtime.asmx).
Good luck with your search
Stephane DUbois
Friday, February 27, 2004
Stephanie said:
"we also provide a realtime web service based on a direct link to Island (www.xignite.com/xrealtime.asmx)."
Thanks Stephane. I found your service a few days ago. It was one of the first things I found in my search, and it's near the top of my list.
Benji Smith
Friday, February 27, 2004
www.freerealtime.com
bella
Friday, February 27, 2004
As the author of two of the Web services you mention ( random Google image search at http://www.ghettodriveby.com and random Bushism at http://greg.froh.ca/fun/random_bushism/ ), I admit that they are indeed laugable and useless.
That was the point.
When I was first researching Web services, I found the majority of tutorials and sample code were needlessly complex.
Sometimes laughable and useless does a better job of illustrating concepts than obtuse and contrived.
Greg Froh
Monday, April 12, 2004
 Recent Topics
Fog Creek Home
| 
