Sunday, February 20, 2005

Web: My trials with Amazon Web Services 

I have been playing with Amazon Web Services for the last 1 week. It is a very very powerful application available with vast amount of data which you can use for variety of purposes. I am using it for data-mining and find relationship between products. It is an experiment and for the fun sake only.

Now for the problem. Since, I am running random queries with page numbers I get error in certain resultsets. I am using C# and .Net for the development and have added the AWS using Add Web Reference method with .Net handling Serialization and related issues. In some of the result sets, there is a formatting error with certain results being not recognized as boolean. Now, Amazon database is very dynamic. Even if I send the same query with same page number I will probably not get the same error. I tried tracing the requests using Ethereal, which is a fine fine tool, and was able to get the exact packets causing the problem. But each Amazon response is in multiple HTTP packets and I was not able to combine them together to make a proper SOAP/XML file and analyze.

Next I installed SoapScope and tried to play around with it. Another fine piece of software, but was using upto 200 Meg of memory on my machine. Since I have 1 GB of RAM (I am running the machine without virtual memory :)) my machine was able to handle it. It uses a web front-end with some nice work and worked very well on my Firefox, which was a nice surprise. It raises the status of company behind it, MindReef, in my mind. Most of the companies will just make IE friendly and not care. But it is a bit complex piece of software and I was looking for just something simple to see the packets.

Then I fell back upon the only expert I knew, Softwaremaker. He pointed me to PocketSoap, which I could not try and a tool he has developed to send manually crafted SOAP messages, available here. I tried and it worked like charm. Simple and to the point. Though it can do with some UI enhancements. Typical programmer developed tool, you can say. But as I said before, the Amazon data is dynamic I could not get back any results with faulty response.

In the end, did something which should have done the first time. I found the message board on Amazon and found already reported by someone that IsAutopraphed field is sending back 'N' which is probably not getting recognized as boolean. Ah, will have to wait for them to solve this. But it was a very good time. playing with a few tools and learning a little bit more about SOAP, Sniffing tools and WSDL.

This page is powered by Blogger. Isn't yours?