Manish Pandit's Blog

Beware of Tomcat Logging

2011-03-26T16:21:00.001-07:00

Here are IGN we have built a Social API, which is built on top of OpenSocial specs, deployed in Tomcat (6.0.20 to be precise) on Centos5. Lately we changed our API so that a method that was returning a POJO was refactored to throw an Exception for a specific condition - a PersonNotFoundException instead of returning a null on getPersonById().

So far so good, but we started to see the GC getting more and more inefficient, and at some point, all the 4 VMs where we've the API running used to run out of heap space. The GC attempts got bigger, at times I'd see 3-4000 collections during the process. To add to my surprises, all VMs started crashing within minutes of each other once every 3-4 days. We use a super awesome product called NewRelic to monitor our services farm, and the behavior I noticed was that of a somewhat massive memory leak. Not only that, the GC CPU Utilization hit as much as 70% right before the VM crashes. We did an internal code review, not once, not twice but many times over and over. We used yourkit and tried to simulate this on local and stage, still no help.

This was going on for a good few weeks till we launched a new product, the IGN Stock Exchange. The API volume increased, as did the number of PersonNotFoundExceptions given the various touchpoints across ign.com. Now instead of the servers blowing up every 3-4 days, the blowups started happening every 2 hours. I tried changing application log levels to FATAL from WARN in a desperate attempt to save the servers but no help. The logs kept rolling, and so did the memory on the VMs.

We run a completely stateless (although not share-nothing) architecture for the API so there was no question of any leaks from the code - which is what baffled me a lot. At this point I dumped the heap from one of the VMs using jmap, compressed it, and scp'd it to my local machine. Then I used Eclipse MAT to open it up and there it was:

One instance of "java.util.logging.ConsoleHandler" loaded by "system class loader" occupies 423,661,904 (87.89%) bytes. The instance is referenced by org.apache.juli.ClassLoaderLogManager$RootLogger @ 0x2aaac3945b38 , loaded by "sun.misc.Launcher$AppClassLoader @ 0x2aaac39010e8". The memory is accumulated in one instance of "byte[]" loaded by "system class loader".

Keywords

java.util.logging.ConsoleHandler

sun.misc.Launcher$AppClassLoader @ 0x2aaac39010e8

byte[]

First off, we do not use juli. The app uses Log4j - thats when I noticed that the logs where most of the stuff was getting written were not the application logs, but Tomcat logs, like catalina.out and catalina.YYYY-MM-DD.log. This gave me a hint to look at Tomcat's logging configuration and here is what I found in CATALINA_HOME/conf/logging.properties:

1catalina.org.apache.juli.FileHandler.level = FINE

1catalina.org.apache.juli.FileHandler.directory = ${catalina.base}/logs

1catalina.org.apache.juli.FileHandler.prefix = catalina.

2localhost.org.apache.juli.FileHandler.level = FINE

2localhost.org.apache.juli.FileHandler.directory = ${catalina.base}/logs

2localhost.org.apache.juli.FileHandler.prefix = localhost.

3manager.org.apache.juli.FileHandler.level = FINE

3manager.org.apache.juli.FileHandler.directory = ${catalina.base}/logs

3manager.org.apache.juli.FileHandler.prefix = manager.

4host-manager.org.apache.juli.FileHandler.level = FINE

4host-manager.org.apache.juli.FileHandler.directory = ${catalina.base}/logs

4host-manager.org.apache.juli.FileHandler.prefix = host-manager.

java.util.logging.ConsoleHandler.level = FINE

java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter

These are the default settings for Tomcat logging, and too brutal for Production given the FINE level. Although Tomcat does support customizing the logging provider, I did not have much time to experiment while the boxes kept blowing up.

I immediately changed FINE to SEVERE (Since Tomcat uses juli and juli does not have FATAL, which is the Log4j equivalent) and bounced the server. Surprise! No more logging to console.

Earlier, the exceptions were logged at the throw new PersonNotFoundException() line and the traces were dumped in Tomcat's console. I am very curious to find out what could be causing the leak in Tomcat's ConsoleAppender (and quite possibly, juli) but for now I am glad that all VMs are behaving normally, the sawtooth GC pattern is back to normal. I'll change the provider to Log4j and see if that makes any difference, and report back my findings.

At the end of the day, NewRelic was instrumental in helping us isolate the problem, and the offline heap profiling with jmap and Eclipse/MAT saved the day.

So if you're deploying under Tomcat in production, you may want to look at logging.properties, more so if you have a public API that can lead to lot of exceptions being thrown by the app due to validation/bad data.

Using MongoDB for IGN's Social Platform

2011-02-15T22:37:00.001-08:00

My Presentation at SF Bay Area MongoDB Meetup on 02/15/2011.

Using MongoDB for IGN's Social Platform

View more presentations from Manish Pandit.

Social Platforms : What goes on under the hood

2010-10-11T17:34:00.001-07:00

My Presentation at Silicon Valley Code Camp on 10/09/2010 at 11:15 AM, Room 5001.

Social Platforms : What goes on under the hood

View more presentations from Manish Pandit.

Eclipse fails to start on Ubuntu

2010-09-19T12:34:00.001-07:00

Often times I've found Eclipse (Galileo) failing to start on Ubuntu. The progress bar on the splash shows up almost half way and the process hangs. Retries do not help. I've the usual plugins (Maven2, subclipse) and nothing fancy.

The only solution I've found to this problem is getting rid of the metadata folder. The downside to this is that I've to reimport the projects in my workspace but at least Eclipse comes back to life.

Since I use multiple workspaces it sucks even more as I've to make sure which one I was using last which is the exact one preventing Eclipse to start.

mpandit@mpandit-ubuntu:~$ find . -name '.metadata' -print

./.local/share/Trash/files/.metadata

./Manish_Laptop_Backup/refactor/.metadata

./Manish_Laptop_Backup/workspace/temp/.metadata

./Manish_Laptop_Backup/workspace/.metadata

./Documents/workspace-sts-2.3.2.RELEASE/.metadata

./Documents/mongoworkspace/.metadata

./workspace/.metadata

./newworkspace/.metadata

Since I was working last in .newworkspace, I'll go and verify the timestamp of the files in it.

mpandit@mpandit-ubuntu:~$ cd newworkspace/

mpandit@mpandit-ubuntu:~/newworkspace$ cd .metadata/

mpandit@mpandit-ubuntu:~/newworkspace/.metadata$ ls -al

total 1876

drwxr-xr-x 4 mpandit mpandit 4096 2010-09-13 23:05 .

drwxr-xr-x 4 mpandit mpandit 4096 2010-07-03 23:03 ..

-rw-r--r-- 1 mpandit mpandit 1025091 2010-09-13 23:05 .bak_0.log

-rw-r--r-- 1 mpandit mpandit 0 2010-06-02 11:03 .lock

-rw-r--r-- 1 mpandit mpandit 860893 2010-09-18 13:21 .log

drwxr-xr-x 3 mpandit mpandit 4096 2010-09-06 22:51 .mylyn

drwxr-xr-x 31 mpandit mpandit 4096 2010-09-12 14:56 .plugins

-rw-r--r-- 1 mpandit mpandit 26 2010-09-18 13:21 version.ini

version.ini's timestamp confirmed that this is the workspace I was using.

Next, we delete the .metadata folder.

mpandit@mpandit-ubuntu:~/newworkspace/.metadata$ cd ..

mpandit@mpandit-ubuntu:~/newworkspace$ rm -rf .metadata/

mpandit@mpandit-ubuntu:~/newworkspace$

Restart Eclipse and it works!

Ruby Memcached Gem on Centos and Ubuntu

2010-05-20T21:24:00.000-07:00

Tried installing memcached ruby gem on Ubuntu 10.04 (lucid) and ran into an error.

mpandit@vaio-ubuntu:~/work$ sudo gem install memcached
[sudo] password for mpandit:
Building native extensions. This could take a while...
ERROR: Error installing memcached:
ERROR: Failed to build gem native extension.

/usr/bin/ruby1.8 extconf.rb
Libmemcached already built; run 'rake clean' first if you need to rebuild.
creating Makefile

make
gcc -I. -I/usr/lib/ruby/1.8/i486-linux -I/usr/lib/ruby/1.8/i486-linux -I. -fPIC -I/var/lib/gems/1.8/gems/memcached-0.19.3/ext/include -L/var/lib/gems/1.8/gems/memcached-0.19.3/ext/lib -fno-strict-aliasing -g -g -O2 -fPIC -fno-strict-aliasing -g -g -O2 -fPIC -c rlibmemcached_wrap.c
rlibmemcached_wrap.c: In function ‘_wrap_MemcachedSt_sasl_callbacks_set’:
rlibmemcached_wrap.c:6669: error: ‘sasl_callback_t’ undeclared (first use in this function)
rlibmemcached_wrap.c:6669: error: (Each undeclared identifier is reported only once
rlibmemcached_wrap.c:6669: error: for each function it appears in.)
rlibmemcached_wrap.c:6669: error: ‘arg2’ undeclared (first use in this function)
rlibmemcached_wrap.c:6669: error: expected expression before ‘)’ token
rlibmemcached_wrap.c:6687: error: expected expression before ‘)’ token
rlibmemcached_wrap.c:6688: error: ‘struct memcached_st’ has no member named ‘sasl_callbacks’
rlibmemcached_wrap.c:6688: error: expected ‘)’ before ‘const’
rlibmemcached_wrap.c:6688: error: expected ‘;’ before ‘arg2’
rlibmemcached_wrap.c: In function ‘_wrap_MemcachedSt_sasl_callbacks_get’:
rlibmemcached_wrap.c:6700: error: ‘sasl_callback_t’ undeclared (first use in this function)
rlibmemcached_wrap.c:6700: error: ‘result’ undeclared (first use in this function)
rlibmemcached_wrap.c:6711: error: expected expression before ‘)’ token
rlibmemcached_wrap.c:6711: error: ‘struct memcached_st’ has no member named ‘sasl_callbacks’
rlibmemcached_wrap.c: In function ‘_wrap_memcached_set_sasl_callbacks’:
rlibmemcached_wrap.c:12021: error: ‘sasl_callback_t’ undeclared (first use in this function)
rlibmemcached_wrap.c:12021: error: ‘arg2’ undeclared (first use in this function)
rlibmemcached_wrap.c:12021: error: expected expression before ‘)’ token
rlibmemcached_wrap.c:12039: error: expected expression before ‘)’ token
rlibmemcached_wrap.c:12040: error: expected ‘)’ before ‘const’
rlibmemcached_wrap.c:12040: error: expected ‘)’ before ‘arg2’
rlibmemcached_wrap.c: In function ‘_wrap_memcached_get_sasl_callbacks’:
rlibmemcached_wrap.c:12122: error: ‘sasl_callback_t’ undeclared (first use in this function)
rlibmemcached_wrap.c:12122: error: ‘result’ undeclared (first use in this function)
rlibmemcached_wrap.c:12133: error: expected expression before ‘)’ token
rlibmemcached_wrap.c: In function ‘Init_rlibmemcached’:
rlibmemcached_wrap.c:13285: error: ‘SASL_OK’ undeclared (first use in this function)
make: *** [rlibmemcached_wrap.o] Error 1

Gem files will remain installed in /var/lib/gems/1.8/gems/memcached-0.19.3 for inspection.
Results logged to /var/lib/gems/1.8/gems/memcached-0.19.3/ext/gem_make.out

I googled around, turns out I need libsasl2 and libsasl2-devel installed. Thats easy.

mpandit@vaio-ubuntu:~/work$ sudo apt-get install libsasl2 libsasl2-dev

After these got installed, I ran into the same error as above. After wasting 15 minutes, I realized that I should try deleting the gem crap it has already put in the folder. This did the trick:

mpandit@vaio-ubuntu:~/work$ sudo rm -rf /var/lib/gems/1.8/gems/memcache*
mpandit@vaio-ubuntu:~/work$ sudo gem install memcached
Building native extensions. This could take a while...
Successfully installed memcached-0.19.3
1 gem installed
Installing ri documentation for memcached-0.19.3...

Now it was turn to do this on my work servers. Unfortunately they run Centos 5.4, and I have a bias against non-Ubuntu distributions, simply because they are a few years/months behind Ubuntu when it comes to packages.

So, lesson learnt from Ubuntu, I tried this:

[mpandit@dev-services-vm-01:~] $ sudo yum install libsasl2
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* addons: distro
* base: distro
* epel: mirrors.kernel.org
* extras: distro
* updates: distro
addons | 951 B 00:00
base | 1.1 kB 00:00
citrix | 951 B 00:00
elff | 1.9 kB 00:00
epel | 3.4 kB 00:00
extras | 2.1 kB 00:00
updates | 1.9 kB 00:00
Excluding Packages in global exclude list
Finished
Excluding Packages from CentOS-5 - Base
Finished
Setting up Install Process
No package libsasl2 available.
Nothing to do

Googled around again, turns out the sasl2 library is called cyrus-sasl2 (dont ask me why). After searching around rpmfind and pbone, I was able to grab the sasl2 RPM and sasl2-dev RPM for Centos 5.4. Low and behold, I had sasl2 already installed, and dev was missing.

[mpandit@dev-services-vm-01:/tmp] $ wget ftp://ftp.pbone.net/mirror/ftp.centos.org/5.4/os/x86_64/CentOS/cyrus-sasl-2.1.22-5.el5.x86_64.rpm
[mpandit@dev-services-vm-01:/tmp] $ sudo rpm -vi cyrus-sasl-2.1.22-5.el5.x86_64.rpm
Preparing packages for installation...
package cyrus-sasl-2.1.22-5.el5.x86_64 is already installed

[mpandit@dev-services-vm-01:/tmp] $ wget ftp://ftp.pbone.net/mirror/ftp.centos.org/5.4/os/x86_64/CentOS/cyrus-sasl-devel-2.1.22-5.el5.x86_64.rpm
[mpandit@dev-services-vm-01:/tmp] $ sudo rpm -iv cyrus-sasl-devel-2.1.22-5.el5.x86_64.rpm
Preparing packages for installation...
cyrus-sasl-devel-2.1.22-5.el5

Now that this was installed, I ran sudo gem install memcached, and saw the same ugly 1000 line error. Again, deleted the directory and it worked like a charm.

Success at the end!

[mpandit@dev-services-vm-01:/tmp] $ sudo rm -rf /usr/lib/ruby/gems/1.8/gems/mem*
[mpandit@dev-services-vm-01:/tmp] $ sudo gem install memcachedBuilding native extensions. This could take a while...
Successfully installed memcached-0.19.3
1 gem installed
Installing ri documentation for memcached-0.19.3...

Not sure if its a bug or a feature, but I really wish gem install would check to see if half-ass installation from a failed attempt already exists in the LIB_DIR. If so, delete it, or at least move it to a .prev folder or something.

Thats why whenever someone tells me to just "apt-get install" or "yum install" anything, I always take it with a grain of salt.

Falling in love with Ruby!

2010-02-08T22:17:00.001-08:00

So my newfound interest is Ruby. Starting as a n00b but picking it up quite a bit. However I doubt I'd be as comfortable with or accustomed to Ruby as I am with Java, given the 12 years of hardcore programming experience with the latter. Anyway, so the task at hand was a small piece of code that I could use to generate timestamps for OAuth requests (oauth_timestamp). I did not realize how verbose Java programming can get till I tried to compare apples to apples.

Here is the Java way to do it:

public class TimestampUtil{
public static void main(String ... args) throws Throwable{
System.out.println(System.currentTimeMillis()/1000);
}
}

mpandit@vaio-ubuntu:~$ javac TimestampUtil
mpandit@vaio-ubuntu:~$ java TimestampUtil
1265696612
mpandit@vaio-ubuntu:~$

Here is the Ruby way:

vi time.rb

puts Time.now.to_i

mpandit@vaio-ubuntu:~$ ruby time.rb
1265696797
mpandit@vaio-ubuntu:~$

Apache commons-lang StringUtils and Numbers

2010-02-07T18:10:00.000-08:00

I had my unit tests fail because I was using the StringUtils.isNumeric( ) on a string containing a negative number. The javadoc on this API is excellent, including the sample request/response but it does not talk about negative numbers. The definition of this API seems debatable from a negative number standpoint : "Checks if the String contains only unicode digits. A decimal point is not a unicode digit and returns false."

I'd call the method something else, like isWholeNumber( ) if the method is going to say that decimals (and negatives) are going to be ignored.

Anyway, here is a simple way I used to figure out if the string contains a number (negative or with decimal, doesnt matter).

String string = "-12.49";

boolean isNumeric = string!=null && string.length() > 0 && string.toUpperCase().equals(string.toLowerCase());

Shindig and boolean properties in the POJOs

2010-02-07T15:09:00.000-08:00

If you ever developed with Apache Shindig, you might have noticed the JSON/XML/Atom marshaller/unmarshaller that takes care of the requests coming in, and responses going out. However, if you use Eclipse (not sure about other IDEs, but it could be applicable to them as well) you may also have used the IDE's generate getters and setters feature to populate your POJO with the mundane getters/setters.

I thought of writing this post after I spent a good 30 minutes trying to figure out why Shindig was not populating the booleans in JSON response, while the XML had them just fine. Turns out, there is different logic to pull "fields" for JSON vs. XML/Atom. The JSONSerializer uses getters (they have to start with a 'get' per the JavaBeans API) to get the values. However, if the boolean property is isAsyncAllowed, Eclipse will generate the methods as : isAsyncAllowed( ) and setAsyncAllowed( ). Since the JSON Serializer looks for getXXX( ), the isXXX( ) never shows up - and hence you never see the booleans/Booleans ever make it to the resulting JSON. This is not an issue with XML, but is sure with JSON. I was able to correct this by renaming the isXXX( ) to setXXX( ).

I will try to do a fix and contribute it back the Shindig source once I have more than 30 minutes of free time though.

Back in action

2009-09-04T18:42:00.000-07:00

Been away for almost 4 weeks - started a new job. Loving it, even though the commute is horrible. As my career grows, I get hit by a worse commute with every single step-up. Back in the days, I used to rent an apartment 1-2 miles from my work. Then I started working in the peninsula (32 miles) and now IGN Entertainment (45 miles). Now I cannot move as I am going to live the Great American Debt for 30 years :)

Anyway - quite a few things. Sprint is coming up with Android phone (the HTC Hero) on Oct 11th. As you noticed from my earlier posts, I jumped on the Pre the day it launched, but had to return it in 2 weeks as the keyboard froze on me for good. So, hopefully HTC Hero will not disappoint me. The Treo 600's speaker magnet is already clotting iron near my heart for 7 years now.

What else..oh yes. If you are following the financial markets (fancy term for stock market), I am sure you are noticing bank failures every Friday. One site that I love - marketwatch.com has a very vibrant community, who have coined a term "FDIC Friday". It is fun to watch community members greet 'Happy FDIC Friday' to each other, and recently the community has even started a group where members put bets on how many banks will fail, and in which states. Interestingly, FL and GA are not allowed as per the community - "Banks fail all the time there". So the trick is to get the numbers right, and the states too. So far they're a very good indicator and I am surprised to see how many members are able to guess the correct number and most of the states right. I say more power to the community - that is what the internet is all about now. The shift from producer-consumer model of the internet to peer-to-peer model is simply amazing and engaging.

Kenmore HE2T 4757 Flooding : Fixed

2009-08-09T23:35:00.000-07:00

It is a great feeling to be a first time homebuyer, have the worst nightmare come true, and to be able to fix it. My Kenmore HE2T 4757 started leaking as soon as I turned it on today. The worst part is that the clogging also pushed water back, with water flowing on the cars parked in my garage. This seemed to be a giant clusterf*k, with the only fortunate part that I was right there witnessing the event. I turned the water off immediately. Initially I thought it could be the cheapo Costco detergent that may have clogged the pipe. The leak was coming from the soap dispenser area, and the water managed to flow all the way through the loading door. I pulled out the dispenser, and the whole area was flooded under there. I siphoned the water out using 3rd grade physics, and noticed a hole right under the dispensing area. I knew the clog is somewhere between the detergent dispenser and the tub. Anyway, I decided to give it another try, this time trying to ensure I capture every detail. Same thing, the water immediately flooded the dispensing area, and onto the loading door, and on the carpet. Clearly it is not the cheapo detergent. I recalled a banging sound during the heavy spin cycle last time that I ignored. This could be related - I googled around and found suggestions to look for coin, etc. in the drain hose. My problem was different - it is not the drain hose, it is the intake hose. Anyway, before going to bed (it is a special day tomorrow - 1st day at my new job) this disaster kept me up and I decided to give a final shot - try to reach that area from inside the machine. I squeezed my fingers to the point where the dispensing hose drains out in the tub and there you have it - I felt a piece of clothing. This was a wave of encouragement, and I immediately felt like I've nailed this problem. I got a pair of pliers from my toolbox, and tried to pull that piece of clothing with no help. This time I rolled the drum and managed to get that thing out fairly easily - it was my wife's sock. It was drenched with detergent, and was full of holes. I believe it got stuck during the spin cycle last time, and got lodged in there to make a mess today. Anyway, it was a good chance to wake her up at 11:30 and show her my accomplishment! We turned on the machine to verify the fix - and it worked. No leaks this time. I am not sure if we have more crap stuck in there, but now at least I know where to find one!

Proud to have saved 75 dollars for Sears to show up, and to know that crappy hardware design is far worse than crappy software design.

gazelle.com review

2009-08-06T19:24:00.000-07:00

Being a huge Costco fan, I closely pay attention to their site for new announcements. Lately they put an ad for a new service (at least new to me!) called Gazelle.com. The idea is that of a "re-commerce", where they take electronic items that are old/outdated and offer you a price. That price is dependent on the time, market demand as well as the reported condition of the item. The search is pretty good, and the autocomplete feature is extremely useful when entering a product name. I tried putting in my old Kodak cameras, which the site promptly reported as being "un-offerable" and offered to recycle them instead - thats pretty neat. Next was time for something serious, getting rid of my Sony DCR TRV 330 camcorder, which I proudly bought back in 2001 for $900 or so. I entered the info I had, and they offered me quite a decent price - which beat my plan of throwing away the camera by a long shot. I could definitely use the extra moola, and I decided to pawn it out. The site offers free shipping, and allows users to print a packing slip. They also have an option to send a box to mail the stuff in. I picked this option, and promptly received the email from them once they shipped it. I got the folded-flat box in a cute green envelope 2 days later. To my surprise, what they sent me was a flat-rate USPS box which could at the most hold the charger for the camcorder. Apparently they send the same size box regardless of the product being sent. Anyway, the envelope did have a postage prepaid mailing slip that I put on a box big enough to fit the camcorder, and dropped it off at the post office. 2 days later, I received an email alert from them - they had received it and the camcorder was in line for an inspection. Next day, I got another email saying the product checks out, and the information I gave them is correct. I liked this notification feature, and keeping the user/customer in loop with the progress. They'd be mailing me a Coscto Cash Card in the next 10 business days. I logged into the site to check out the status, and the site being very simple is very navigable and user-friendly. It took me no more than 5 clicks to check out the progress, and get details on their inspection checkpoints.

All in all, this is a service that I recommend and encourage everyone uses, even if you do not have a product with a residual value. The re-commerce concept is pretty neat, and it also helps the environment. The service (and the speed of it) beats the services where you pay to get something - with reversal of roles (they pay you!), I was surprised to see the quality of updates and user-friendliness of the site.

I am sure I'd be using Gazelle.com quite often - regardless of the moola. I feel proud to be a part of the green-earth/recycle/re-commerce initiative that this company is all about - and above all, they partner with Costco which in and of itself says a lot about their service.

I am on a roll..

2009-07-16T09:59:00.000-07:00

I am on a roll running into these well-known sites with messed up grammar!

LinkedIn needs a visit from the Grammar Police..

2009-07-15T19:40:00.000-07:00

Ran into this today (image capture blurred with GIMP).

Clearly, LinkedIn needs a visit from the Grammar Police (or the Grammar Fairy, based on what you're into).

Unit testing Exception scenarios

2009-07-15T11:15:00.000-07:00

As a huge fan of Test Driven Development, I try to make sure I get good coverage (~80%). However, most often this is not the case as the code throwing checked exceptions usually does not get executed. The goal is to write test cases to cause the code to trigger those code blocks. When triggered, the code will fail (as it is supposed to!) but the JUnit must pass. There have been plenty of discussions around having a clean way to unit-test this in the past, and I suggested this on the CLJP newsgroup a few years ago as well. Wanted to use a code snippet here to illustrate the solution in a dev-friendly way instead of plain english.

Let us say this is the piece of code that needs to be JUnit'd for exception cases. The code below returns a String value stored in a collection that is local to the class for an index. For negative index it throws an exception.


public String getValueAtIndex(int index) throws NegativeIndexException{
    if(index < 0) throw new NegativeIndexException(index);
    else ...
}

In order to test this code, here is the JUnit method as I use it.


public void testNegativeIndexException(){
    boolean failure = true;
    try{
    //we already instantiated the object/singleton via setup() method
        String value = someInstance.getValueAtIndex(-1);
        failure = false;
    } catch(NegativeIndexException ex){
        logger.error(e,e);
    }finally{
        //this is where we put the assertion
        assertTrue(failure);
    }
}

This will ensure the code is tested/covered for Exceptions.

Improvements for Palm Pre : My Rant

2009-06-07T22:05:00.001-07:00

Here is a list of items that I'd think I either do not know how to figure out, or Palm Pre does not have them, but I'd like to see them. Please note that this is coming from someone who have never used an iPhone, so let us not get started on what/who is better. That is not the point of this post, but to reflect on what is missing, or hard to find on the Pre. Hopefully a good number of these can be handled via documentation or alternative software (like a third party browser). I will keep adding more items to this list (and removing the existing ones) as I get more familiar with the Pre. For now from the battery drainage it seems my phone will be headed back to the Sprint Store for a refund in a week.

Ability to change the Home Page on the Browser instead of the (hardcoded?) favorites page.
The current home page (favorites) view needs to be customized as a list vs. preview icons. I'd prefer a list so it can fit more than 4 on a screen.
On the browser, make the -> icon clickable so that I can have a drop down of recently visited sites just like the regular PC browser we're all used to for the last 10 years.
"Add to quick launch" option across all menus if not done via some gesture, which also means ability for the quick launch bar to hold more than 5 apps.
A "menu" shortcut key on the keyboard so that I do not have to click on the tiny little app title with my nailtip to open the menu (which can of course be operated with a finger tip instead of a nail tip)
When the phone sleeps while the keyboard is out, the center button as reaching out to the back of the phone with keyboard sticking out is hard. For now I am just pressing the orange button to wake it up.
Macromedia Flash plugin for the Browser, I thought it was already there but guess not.
Is there a place to review the apps, instead of typing the review on the Pre itself via the App Catalog app?
Some kind of an hourglass or splash screen or other gesture to show that the application is being loaded. Some apps are slow and it just confuses me as to whether I need to click the icon again or wait till it launches. I know the little nailtip menu changes the title but thats too hard to notice given the size of the text.
A visual indication on the screen when the volume is muted using the cheap plastic toggle switch at the top.
Release the freakin SDK already!

Here are the bugs I found in the Pre:

Light Leakage at the bottom of the screen, this could be related to the heat at the bottom of the phone. This is a no-no. Here are the pix that I uploaded on picasa.
Adding Gmail account to synergy pumped my contact list with 800 emails and no other information. Ended up removing that account from contacts.
Heating - while charging, or using the Pre for more than 10 minutes, the bottom part of the phone heats up real bad.

Ok Palm Pre..show me you can hold better than my Treo 600

2009-06-07T20:21:00.000-07:00

So here I am a full 36 hours after I got the Pre in my hands. Used it as my alarm clock earlier in the morning, and surfed a few sites while lying in my bed last night. I did not use it as a cheese knife but I am getting tempted to :)
Drove to Monterey Bay Aquarium in the morning with a full charge. By the time I reached there (about 1:45 hours), the battery had already drained down to 85%. Then, at Monterey, I noticed that I had zero reception and of course no internet. Magically, the letter "R" appeared near the bars, and all the bars glowed. I have no clue what this "R" was, but I am guessing it meant Roaming. Still no internet. I then restarted the phone (turned it off, and booted it back up) and was impressed to see it connect in a snap, and was able to dial out (I usually dial *4 or *2 to verify the call quality, as I have learnt not to trust the bars any more). So far so good.
While driving back at 4:30 PM (about 8 hours from the time the phone was off-charger), I had about 66% juice left. Now the shittiest piece of electronic gadget I ever owned (Garmin C330) failed to receive satellite signal like it always does. I reverted to Sprint Navigation on the Pre, which worked like a charm. My wife held the phone while I was driving, and we saw the battery go down like a lead baloon. In half an hour of drive (Gilroy/San Jose border), the phone had about 33% battery left, and had gotten hot enough to fry an egg. I turned the nav off off as we went to Shalimar in Fremont for a well deserved dinner after the long drive. That is where I wanted to check my emails, and oops - we are at 28%. I turned it off completely, and now I am home putting it back on the charger. Going through the threads at PreCentral, it seems to be the default settings taking up all the juice, and I will try to tweak mine and post back. I still have 29 days to return the phone and I really really do not want to do so, unless this seems to be a universal problem.
I would recommend everyone buy a microUSB car charger no matter what with this phone - I know I will.
It is like a laptop with a Fisher Price toy battery - at least that is my feeling before changing any of the default values. I do have google set up to push emails so I doubt the phone is reaching out every time. I have location services turned on, which could be a culprit too. I am off to PreCentral to check out what others are feeling about this phone - stay tuned for more :)
Meanwhile my Treo 600 battery is still showing a 92% after being off the charger for 2 full days.

Took the plunge : Bought me a Palm Pre!

2009-06-06T16:33:00.000-07:00

Got to the Pleasanton Sprint Store at 8:45 AM, was out with a Pre at 11:30 AM. I had to change my plan though - earlier I was with Family plan but got it changed to 2 individual plans, as my wife does not need the Data/Texting stuff.
Anyway, out the box the Pre was about 40% charged, so first thing was to charge it to a decent level. The only drawbacks comparing it to my ancient Treo 600 is that the buttons feel rubbery (vs. metallic with Treo), and the speaker is a little weak. Also I am a little afraid that someday the banana design is going to split while I am sliding the keyboard.
So far I've played around with the following applications, and I must say it is living up to my expectations.

Sprint Navigation (GPS), which does not work in landscape mode btw
Youtube
Downloaded the LinkedIn app, which mostly displays the connections (not sure if it displays status alerts, etc.)
The browser (it rocks!)
Synergy to sync with Google and Facebook (Seamless, painless, braindead easy)
Camera (very nice)

The weekend is still young, and I'll be spending quite a few hours playing around with it, and will post my thoughts here....stay tuned!

Fixing InnoDB table corruption (MySQL Error 2013)

2009-06-06T01:51:00.000-07:00

Here I am, trying to get some time off for the weekend, when a good friend of mine called me up with a strange problem on his site - a 404. Does not sound too bad, does it? But wait, there is more. Turns out Tomcat was up, Apache httpd was up, and MySQL was up - but then why a 404? A peek in catalina.out took me by surprise. The JSP failed to render due to the fact that the servlet was trying to pull something from the database to set as a request attribute. Instead of a 500 which we'd normally expect in this situation, the error was a 404 though. Since the error codes at this point were not that important, I started to look into the logs and configurations..

Here is what the DBCP messages said:

2009-06-04 03:10:40,635 [TP-Processor739] FATAL com.xxxx.yyy.SimpleGenericDriver (SimpleGenericDriver.java:54) - Failure getting connection from datasource
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool exhausted
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:103)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)

As if this wasn't bad enough, here are more gory details on this situation:

2009-06-04 03:23:19,041 [TP-Processor836] FATAL com.xxx.yyyy.SimpleGenericDriver (SimpleGenericDriver.java:54) - Failure getting connection from datasource
com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:

** BEGIN NESTED EXCEPTION **

java.io.EOFException

STACKTRACE:

java.io.EOFException
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1865)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2256)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2756)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:777)
at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:3216)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1147)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:2595)
at com.mysql.jdbc.Connection.(Connection.java:1509)

I logged into MySQL, tried to list the tables in the database, worked just fine. Restarted the database (/sbin/service mysqld restart). This did not fix the problem at all. So I decided to back up the database (mysqldump) and then restore it after cleaning it up. This is where I found a more serious problem. While dumping a larger table (about 15k rows), I got this:

Error 2013: Lost connection to MySQL server

Googled around this, and was glad to see I was not alone. However, the MySQL documentation lists a good number of causes for this error. The list of discussions over this particular error and how it pertains to my situation lef me to write this blog post. This error seemed to be a catch-all, and I wanted to share my experiences with symptoms that were specific to the situation I was trying to help my buddy with.

For me, I had a few clues - the disk was not full, not all tables were failing, I was able to run mysqlcheck on all tables except one. Then I did what I should have done first - take a look at mysqld log. And there it was, in clear text:

InnoDB: Page lsn 0 212096651, low 4 bytes of lsn at page end 212096651
InnoDB: Page number (if stored to page already) 1422,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 0 851
InnoDB: (index PRIMARY of table systemdb/member)
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 1422.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/mysql/en/Forcing_recovery.html
InnoDB: about forcing recovery.

I went to the recovery documentation, and added innodb_force_recovery = 1 in the [mysqld] section to /etc/my.cnf. After this I restarted MySQL, and tried mysqldump. Surprise!! I was able to get past the table which was causing the problem. Reading the documentation closely I realized that I need to remove the inndb_force_recovery = 1 before proceeding with a restore, as a non-zero value disables inserts. So I commented out this line, restarted MySQL and dropped the table that was creating all this problem. The drop worked just fine (however, when I tried to do a select count(*) on that table I still got the dreadful 2013 error). Then I sourced the restored table to this schema and we were good to go!

So, if you have the similar symptoms for 2013, give this exercise a shot. Here is the summary:

Symptoms:

1. If using JDBC, the logs will have fatal errors with MySQL connection failures (java.io.EOFException)

2. The disk is not full

3. The tables in question are InnoDB

4. When you try to run mysqlcheck on the schema, the connection drops suddently after reading a certain table -

mysqlcheck: Got error: 2013: Lost connection to MySQL server during query when executing 'CHECK TABLE ... '

5. Any queries on that table via mysql command line client fail

mysql> select count(*) from sometable;

ERROR 2013 (HY000): Lost connection to MySQL server during query

6. The mysqld log mentions page corruption.

If all of these symptoms match your situation, give the innodb recovery a shot, and most likely you'd be able to restore the table(s).

Treo 600/650 on SNL..

2009-05-09T23:49:00.000-07:00

Thought of writing this post right after I saw my beloved Treo 600 (it could have been a 650 though) on SNL skit on 5/9/09. Justin Timberlake pulled out a tricked out Treo from his purse during the famous Target skit starring Kirsten Wigg. Found it pretty interesting because frankly I have not seen anyone with a Treo 6xx except myself ever since the Jesus Phone hit the shelves. So here I am holding the fort till Pre comes out - hopefully during June, but definitely glad to see that people do carry them around, even if it is in 5 minute SNL skits :)

http4e - What took them so long!

2009-04-17T00:06:00.000-07:00

I've been heavily involved with designing and coding something that needs me to hit a bunch of Servlets after setting HTTP headers, and verify the HTTP raw response. With my strict practice of Test Driven Development, it started to become a constant pain - the tools in my box (Poster, soapUI and JMeter) wore me out as I wanted something easy enough to set up, embedded in Eclipse for very frequent use - just like soapUI for SOAP. I was getting desperate enough to roll my own little SWT app but then I ran into this great Eclipse Plugin - http4e. It fit my needs 100% - even generates one-click Java Code that you can copy/paste and modify if the test code has to be run outside of the IDE, or on any other machine. One annoying thing is the lack of saving the view - so if you exit Eclipse, you will lose the URLs and anything that you had in there. I end up sticking the URLs and parameters/headers in a text editor and save it for easy access to copy/paste.

I'd recommend http4e to anyone dealing with similar situations - like validating HTTP headers, or testing RESTful services, etc.

ML 320 Machine Gun Locks

2009-01-30T20:24:00.001-08:00

It is only funny when it happens to others. Today it happened to me - the phenomenon I've been reading about on the ML 320 Forums since 2002. My '99 ML 320 was locking, but the door locks were machine-gunning 3 times each. I was scared as I knew people were paying through the nose to get this fixed. I hoped the alarm would engage, even though I do not mind if the truck gets stolen at this point.

I was ready to shell out a grand, and call the dealership in the AM when I thought about the actual behavior of the locks. The only lock that was not firing like a gun was the cargo lock. I immediately went to the garage and made sure the car is locked. It did the machine gun sound so the problem persisted. I pulled the cargo door latch and what do you know - the alarm went off. I turned it off with the frob, and inspected the rear latch. I could notice the metal clamp which I without a clue moved to the side and slammed the door to close. This time, no machine gun and the locking was normal - I could see the turn signals come on to confirm. Just saved a grand or two, feels great in this economic depression.

Other painful stories about Mercedes Benz M-Class here. I wanted to write my story, but it'd take me at least a week to go over the phonebook of repair records on the ML 320.

If you landed on this page searching for the machine gun door locks on your ML 320, give this a shot, maybe you can save a grand or two too :)

Apache CXF with Maven Archetype Plugin and Eclipse

2009-01-22T21:27:00.002-08:00

Been using Apache CXF for quite sometime, never had a chance to create the project with Maven Archetype Plugin - I usually started with a Maven Web Project, and added CXF dependencies, configuration, etc. along the way in Eclipse.

However, cxf archetype makes things a lot easier. However, there are a few gotchs (and hence this post).

To start with, I am assuming that you've Eclipse (Ganymede preferred) with Sonatype Maven Plugin.

Configure this plugin via the settings that can be found under Window->Preferences->Maven. Personally, I do not use the embedded version, but 2.0.9 instead.
To start a very basic, simple CXF service via the archetype, create a new Maven Project.
File->New -> Other -> Maven Project

In the Wizard's filter section, type in cxf and wait for the combobox to update.
In the list, you should see org.apache.cxf as groupId and cxf-http-basic artifactId. This is what we'd need. If you do not see it, check for Proxy settings in Maven Configuration and try to use external Maven installation instead of the one that comes embedded.

Once selected, click next. In the next dialog, enter the groupId (com.myproject) and artifactId (cxfservice) and version. Keep the Package same as groupId. Click Finish.

You should see the project called cxfservice in the Eclipse's package explorer. Notice the red-marks on src/main/java. This is where the problem is.

The archetype plugin creates a sample HelloWorld service and impl, and ignores the groupId and package name - it uses org.apache.cxf as package name instead.

To fix this, change the packages in the 2 files to com.myproject.cxfservice. You can use the Eclipse QuickFix for this too.

You'd also need to change the package name in the annotation in the impl :

@WebService(endpointInterface = "org.apache.cxf.HelloWorld")
change it to
@WebService(endpointInterface = "com.myproject.cxfservice.HelloWorld")

In my case, I also had a problem with runtime, as I had JDK 1.5 intead of 1.6 in there. Since 1.5 is not aware of JAX-WS annotations, the redness did not go away by fixing packages. To fix this, I right clicked on JRE System Library, selected Properties and picked JDK 1.6u10. This took care of the @Webservice annotation's compilation.

Next is to fix the beans.xml file, where CXF looks for the endpoints and implementations. This also has org.apache.cxf package, which we need to change to com.myproject.cxfservice.

The beans.xml file is under src->main->webapp->WEB-INF. Edit the file to fix the package.

<jaxws:endpoint id="helloWorld" implementor="com.myproject.cxfservice.HelloWorldImpl" address="/HelloWorld">
</jaxws:endpoint>

Also, change the finalName in the pom.xml to a more friendly name, preferably project name. In the <build> section add this:

<finalname>cxfservice</finalname>

Now right click the project -> Run As -> Maven clean. Once done, do the same and select Maven install.

You should see this in the console:

[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building Simple CXF project using spring configuration
[INFO] task-segment: [install]
[INFO] ------------------------------------------------------------------------
[INFO] [resources:resources]
[INFO] Using default encoding to copy filtered resources.
[INFO] [compiler:compile]
[INFO] Compiling 2 source files to /home/mpandit/workspace/cxfservice/target/classes
[INFO] [resources:testResources]
[INFO] Using default encoding to copy filtered resources.
[INFO] [compiler:testCompile]
[INFO] No sources to compile
[INFO] [surefire:test]
[INFO] No tests to run.
[INFO] [war:war]
[INFO] Packaging webapp
[INFO] Assembling webapp[cxfservice] in [/home/mpandit/workspace/cxfservice/target/cxfservice]
[INFO] Processing war project
[INFO] Webapp assembled in[207 msecs]
[INFO] Building war: /home/mpandit/workspace/cxfservice/target/cxfservice.war
[INFO] [install:install]
[INFO] Installing /home/mpandit/workspace/cxfservice/target/cxfservice.war to /home/mpandit/.m2/repository/com/myproject/cxfservice/0.0.1-SNAPSHOT/cxfservice-0.0.1-SNAPSHOT.war
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8 seconds
[INFO] Finished at: Thu Jan 22 22:26:21 PST 2009
[INFO] Final Memory: 16M/81M
[INFO] ------------------------------------------------------------------------

To run this service, we'd need to deploy this war in a server. Create a new server in eclipse (I use Tomcat 6) and associate this project with the server. Start the server, and you should be all set.

Open a browser, and go to http://localhost:8080/cxfservice/

If everything went well, the page should display a list of available services under that context, which in our case is HelloWorld. To get the WSDL for this service, point the browser to http://localhost:8080/cxfservice/HelloWorld?wsdl

I suggest playing around with this service and make it RESTful if you're new to JAX-WS.

Palm Pre - Cannot wait!

2009-01-08T19:07:00.000-08:00

Today Palm announced the Palm Pre, probably the most feature rich phone after the Treo series. I do not want to get into the iPhone vs. Pre debate, but this is definitely something I can see myself switching to. I am stuck with Treos because my Treo 600 is a die-hard phone, and has survived almost 6 years of very heavy usage, including falling on the hard floor a bunch of times. It still runs great, holds the battery charge and has maintained the touch screen without many scratches or bumps. Secondly, Sprint. I love their data plan, even though a majority of users complain about their voice service. I am not much of a talker and rely on voicemails for better communication myself. Treo 600 is a great internet device, unforunately its too ancient for HTML browsers like Opera Mini 4 and not too many portals these days are into WAP. Thankfully iGoogle works great, which helps me organize my stock watchlist, emails and news on the Treo.

Anyway, I have seen a few images and videos of Pre, and I am sold. The rich featureset combined with Sprint's EVDO RevA will do wonders. Finally Sprint has a phone I want to buy (secretly I've been waiting for Android, but hey! Palm is all the better!). I cannot wait to check out the new WebOS, get familiar with the SDK (if there is one) and write a few apps on this baby. I've signed up for an availability notification, and a Sprint Store is about 4 blocks from where I live.

The only thing I did not like about this phone is its name . Treo sounds much cooler. They could have named it Treo Touch or something similar :)

I am very happy that Palm did come up with something new other than modifying the form factor of the original Treo 600 and releasing it every year. Hopefully we'll get more details on pricing and availability in the coming days. I also hope Sprint will not screw the customers by inventing a new, expensive data plan to get this Phone.

Microsoft Security Warning on Ubuntu - Love these smartasses..

2009-01-01T01:26:00.001-08:00

Got this while clicking around trying to find San Francisco New Years Fireworks Photos...

Talk about a New Years Surprise - I am speechless. Love their creativity too - check out the names of the "dangerous spyware" that their sophisticated system detected on my box. I really feel for people who fall for this.

Here is the link that got me to this page - I searched for blogs with the search text "sf fireworks 2009" within the last 12 hours. Note that the link shows up on the 1st page itself.

Oh, BTW, Happy New Year 2009 everyone!

VLC celebrates Christmas

2008-12-19T17:57:00.000-08:00

Last night around 1:00 AM, I was surprised to see a different VLC icon in the dock bar. I was happy to take a screenshot to post it here, and will see how long before they push the next update to turn the icon back to normal ;-)

Merry Christmas and Happy Holidays everyone!

Digital Photos with no timestamp

2008-12-18T19:18:00.000-08:00

I recently bought a HD Camcorder which works great - however, the digital still images do not have a time-stamp on them. I struggled through the thick manual but could not find a way to stamp the date time on the images. I knew that the information has to be there (as the camera was able to display the date + time in preview mode) - but if I could just stamp it. So there you have it - I've set the stage for the little nugget I am working on with Drew Noakes' raw extraction library.

The library provided me the core of this effort - getting the timestamp from the EXIF header. Rest was done with Java's ImageIO API.

I hope to have the code complete and pushed to code.google.com as an open source artifact real soon. So far the testing has been great - I am mostly bulletproofing the configuration and making it more flexible as we speak. Stay tuned!

Setting up jUDDI with Tomcat and MySQL

2008-12-09T18:05:00.000-08:00

Thought of writing this post to help folks who are interested in setting up jUDDI with Tomcat and MySQL without jumping a lot of hoops and investing hours googling around like I did. There are a couple of posts here and there like:

I must say the links above give an excellent walkthrough, but unfortunately the release version of jUDDI does not quite fit with those instructions. Interestingly, a Google search with installing juddi gives the top results where users are complaining about installation problems.
NOTE: This will work with jUDDI Release Candidate #5 for Version 2.0.(previously version 0.9) only. I have not tried any other release, and I seriously doubt that this will work on any other release either.
Here is what we'd need - I am putting in version numbers as well.

You'd also need JDK 1.6u10, however I'd think 1.5 should work just as fine.
First step would be to set up Tomcat, verify it is up and running. Once verified, shut it down.
mpandit@mubuntu-laptop:~/apache-tomcat-6.0.16/bin$ ./catalina.sh run
Using CATALINA_BASE: /home/mpandit/apache-tomcat-6.0.16
Using CATALINA_HOME: /home/mpandit/apache-tomcat-6.0.16
Using CATALINA_TMPDIR: /home/mpandit/apache-tomcat-6.0.16/temp
Using JRE_HOME: /usr/lib/jvm/java-6-sun
..
..
INFO: Server startup in 10498 ms
Hit ^C to kill the server. This verifies that Tomcat is all set.
I also assume that you've MySQL installed. We will need to create a user as well as an empty schema for jUDDI.
Log in to MySQL as root.
mpandit@mubuntu-laptop:~$ mysql -u root -p
Create the juddi schema
mysql> create database juddi;
Query OK, 1 row affected (0.01 sec)
We'll use scripts packaged with jUDDI to populate this schema.
Next, we create the database user for this schema.
mysql> GRANT ALL PRIVILEGES ON juddi.* TO 'juddi'@'localhost' IDENTIFIED BY 'juddi' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)
Verify that we're able to log in with the new user. Exit from MySQL client, and re-login as juddi.
mpandit@mubuntu-laptop:~$ mysql -u juddi -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 46
Server version: 5.0.67-0ubuntu6 (Ubuntu)
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> use juddi;
Database changed
mysql> show tables;
Empty set (0.00 sec)
mysql>
Now we've done the groundwork, so its time to download jUDDI.
Download juddi-2.0rc5.jar and juddi-web-2.0rc5.war files from the jUDDI download site. Explode the .jar on a folder on your machine.
Copy the .war to $TOMCAT_HOME/webapps folder. Start Tomcat, making sure there are no errors.
This should deploy the jUDDI registry webapp after exploding the .war in the webapps folder.
mpandit@mubuntu-laptop:~/apache-tomcat-6.0.16/webapps$ ls
docs host-manager juddi-web-2.0rc5.war ROOT
examples juddi-web-2.0rc5 manager
Stop Tomcat. Remove the .war while keeping the exploded folder intact.
mpandit@mubuntu-laptop:~/apache-tomcat-6.0.16/webapps$ rm -rf juddi-web-2.0rc5.war
Next, if you hate long folder names as much as I do, we'll create a symlink.
mpandit@mubuntu-laptop:~/apache-tomcat-6.0.16/webapps$ ln -s juddi-web-2.0rc5/ juddi
Start Tomcat. We'll check the happyjuddi page now.
Point the browser to http://localhost:8080/juddi and click the 'Validate jUDDI' link. This will bring us to why I wrote this post - the only red section will be the datasource configuration.

jUDDI DataSource Validation

+ Got a JNDI Context!
+ Got a JDBC DataSource (dsname=java:comp/env/jdbc/juddiDB)
- DB connection was not acquired. (Cannot create JDBC driver of
class '' for connect URL 'null')
- SELECT COUNT(*) FROM PUBLISHER failed (null)

This means jUDDI has no clue on how to talk to the MySQL database we just set up. Let us fix that.
Stop Tomcat. Download MySQL Connector/J 5.1 JDBC driver, and explode the archive in a temp folder. Copy mysql-connector-java-5.1.7-bin.jar from the extracted folder to $TOMCAT_HOME/lib.
mpandit@mubuntu-laptop:/tmp/mysql-connector-java-5.1.7$ cp mysql-connector-java-5.1.7-bin.jar /home/mpandit/apache-tomcat-6.0.16/lib/
Back to Tomcat. We'd create a context.xml and configure datasource for juddi context. Change directory to $TOMCAT_HOME/webapps/juddi/META-INF folder, and create a context.xml file
mpandit@mubuntu-laptop:~/apache-tomcat-6.0.16/webapps/juddi/META-INF$ vi context.xml
Paste this and change accordingly (if you are using a different database/user/password..)

<context path="juddi" docbase="juddi" debug="5" reloadable="true" crosscontext="true">
<resource name="jdbc/juddiDB" auth="Container" type="javax.sql.DataSource" username="juddi" password="juddi" driverclassname="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost:3306/juddi?autoReconnect=true" validationquery="select count(*) from PUBLISHER">
</resource>
</context>

Save and exit. (Re)start Tomcat, and revisit the happyjuddi page at http://localhost:8080/juddi/happyjuddi.jsp. Now you should notice a different error.

jUDDI DataSource Validation

+ Got a JNDI Context!
+ Got a JDBC DataSource (dsname=java:comp/env/jdbc/juddiDB)
- DB connection was not acquired. (Cannot create PoolableConnectionFactory
(Table 'juddi.PUBLISHER' doesn't exist))
- SELECT COUNT(*) FROM PUBLISHER failed (null)

Great! We need to populate the schema. This is where the jUDDI jar file will come into picture. I hope you followed the instructions and exploded it somewhere on your disk.
mpandit@mubuntu-laptop:/tmp/juddi-jar$ ls
juddi-2.0rc5.jar
mpandit@mubuntu-laptop:/tmp/juddi-jar$ jar xvf juddi-2.0rc5.jar
Notice the juddi-sql folder. We're almost done - but we need to find/replace ${prefix} from 2 SQL files. Locate insert_publishers.sql file, and remove the ${prefix} from the file. Also update the information like email address, name.
INSERT INTO PUBLISHER (PUBLISHER_ID,PUBLISHER_NAME,EMAIL_ADDRESS,IS_ENABLED,
IS_ADMIN, MAX_BUSINESSES, MAX_SERVICES_PER_BUSINESS,MAX_BINDINGS_PER_SERVICE,MAX_TMODELS)
VALUES ('juddi','Manish Pandit','my_email_address','true','true',25,20,10,100);
Next, locate mysql folder, and edit create_database.sql to remove ${prefix}. I had 155 instances of it.

Thats it! Now we run these SQLs in MySQL. Log in as juddi user and change database to juddi.
mysql> use juddi
Database changed
mysql> source /tmp/juddi-jar/juddi-sql/mysql/create_database.sql
..
mysql> source /tmp/juddi-jar/juddi-sql/insert_publishers.sql
Query OK, 1 row affected (0.00 sec)
This completes the installation. Refresh the happyjuddi page - you should see no errors.

jUDDI DataSource Validation

+ Got a JNDI Context!
+ Got a JDBC DataSource (dsname=java:comp/env/jdbc/juddiDB)
+ Got a JDBC Connection!
+ SELECT COUNT(*) FROM PUBLISHER = 1

So, the registry is up and running. Use any UDDI registry browser (login as juddi/juddi). Open up the web.xml for the publish, discovery and inquiry URLs.
Do comment on this post if you run into any issues, or have any suggestions. Hope this helps!

Create MP3s out of flash videos with Firefox and ffmpeg

2008-12-08T21:12:00.000-08:00

Disclaimer : Please be aware of the copyright rules as with any media. For this demo, I'd go ahead and use Creative Commons licensed flash movie on Youtube..but the same is true for videos on AOL, etc.

Here is what you'd need:

1. Firefox (I got 3.0.4)
mpandit@mubuntu-laptop:~$ firefox -v
Mozilla Firefox 3.0.4, Copyright (c) 1998 - 2008 mozilla.org

2. Most likely a *nix based system
mpandit@mubuntu-laptop:~$ uname -a
Linux mubuntu-laptop 2.6.27-9-generic #1 SMP Thu Nov 20 21:57:00 UTC 2008 i686 GNU/Linux

3. ffmpeg
mpandit@mubuntu-laptop:~$ ffmpeg -version
FFmpeg version r11872+debian_3:0.svn20080206-12ubuntu3, Copyright (c) 2000-2008 Fabrice Bellard, et al.
configuration: --enable-gpl --enable-pp --enable-swscaler --enable-x11grab --prefix=/usr --enable-libgsm --enable-libtheora --enable-libvorbis --enable-pthreads --disable-strip --enable-libfaad --enable-libfaadbin --enable-liba52 --enable-liba52bin --enable-libdc1394 --disable-armv5te --disable-armv6 --disable-altivec --disable-vis --enable-shared --disable-static
libavutil version: 49.6.0
libavcodec version: 51.50.0
libavformat version: 52.7.0
libavdevice version: 52.0.0
built on Oct 3 2008 22:40:31, gcc: 4.3.2
FFmpeg r11872+debian_3:0.svn20080206-12ubuntu3
libavutil 3212800
libavcodec 3355136
libavformat 3409664
libavdevice 3407872

Do not have ffmpeg? Follow the instructions on the site, or if you're on Ubuntu, use this:

mpandit@mubuntu-laptop:~$ sudo apt-get install ffmpeg

Last but not the least, install Download Helper extension for Firefox.

Next, open up page with the video that you want to be converted to MP3.

Let us use Creative Commons video for this example.

Cool. Notice the rotating balls on the DownloadHelper Icon. Next to it is a little arrow, click that. From the list of videos, click the one with 'Youtube' logo on it. This could vary on other sites, but you get the idea about how to locate the .flv that needs to be mp3-ised.

Save the flv on a folder on your disk.

mpandit@mubuntu-laptop:~/tempfolder$ ls
Creative_Commons_-_Get_Creative.flv

Use VLC Player to obtain audio information of the file. If not, you can stick to 22050 Hz sampling rate, and 16bits per sample.

To get info from VLC Player, open the video in VLC Player. Then click Tools -> Media Information -> Codec Details -> Stream 1 (audio). Note down the Sample Rate and Bits per Sample.

Next, we use the magic of ffmpeg to get ourselves a nice MP3!

mpandit@mubuntu-laptop:~/tempfolder$ ffmpeg -i Creative_Commons_-_Get_Creative.flv -ar 22050 -ab 64 -f mp3 mymp3.mp3
FFmpeg version r11872+debian_3:0.svn20080206-12ubuntu3, Copyright (c) 2000-2008 Fabrice Bellard, et al.
configuration: --enable-gpl --enable-pp --enable-swscaler --enable-x11grab --prefix=/usr --enable-libgsm --enable-libtheora --enable-libvorbis --enable-pthreads --disable-strip --enable-libfaad --enable-libfaadbin --enable-liba52 --enable-liba52bin --enable-libdc1394 --disable-armv5te --disable-armv6 --disable-altivec --disable-vis --enable-shared --disable-static
libavutil version: 49.6.0
libavcodec version: 51.50.0
libavformat version: 52.7.0
libavdevice version: 52.0.0
built on Oct 3 2008 22:40:31, gcc: 4.3.2

Seems stream 0 codec frame rate differs from container frame rate: 1000.00 (1000/1) -> 12.00 (12/1)
Input #0, flv, from 'Creative_Commons_-_Get_Creative.flv':
Duration: 00:06:36.9, start: 0.000000, bitrate: 64 kb/s
Stream #0.0: Video: flv, yuv420p, 320x240 [PAR 0:1 DAR 0:1], 12.00 tb(r)
Stream #0.1: Audio: mp3, 22050 Hz, mono, 64 kb/s
WARNING: The bitrate parameter is set too low. It takes bits/s as argument, not kbits/s
Output #0, mp3, to 'mymp3.mp3':
Stream #0.0: Audio: libmp3lame, 22050 Hz, mono, 0 kb/s
Stream mapping:
Stream #0.1 -> #0.0
Press [q] to stop encoding
size= 1550kB time=396.9 bitrate= 32.0kbits/s
video:0kB audio:1550kB global headers:0kB muxing overhead 0.001953%
mpandit@mubuntu-laptop:~/tempfolder$

In other words, the command would be :

ffmpeg -i {input file name} -ar {audio sample rate} -ab {bits per sample} -f {force format to mp3 in our case} {output file name}

Have fun :)

Windows Media Player Streams on Ubuntu..

2008-11-17T19:33:00.000-08:00

I've always been a fan of VLC and ffmpeg. Both of these have taken Open Source Software to the point where average, non-tech-savvy folks can see the power of Open Source and how it matches neck to neck with commercially (or bundled) media players without all the fluff.

Earlier I had Firefox 2.0 on Ubuntu Hardy to use gstreamer to render Windows Media Player streams which used to freeze up the system, or did not play video. However, with Intrepid, I was able to get vlc plugin for Firefox and boy am I glad I did not use gstreamer. Now all I got is VLC Player to play almost anything and everything, and the in-browser VLC plugin to render non-flash streams.

Finally I can watch live Indian News channels without having to resort to Windows, or see my system crash with gstreamer :)

Frozen Eggs, Carrots, Milk and Tazo Chai..renter's delight

2008-11-15T09:51:00.000-08:00

Anyone ever had these? My apartment complex replaced my old 20 year old HotPoint fridge with a new 20 year old Hotpoint fridge as the old one's compressor gave up. However, the new 20 year old fridge beats the one by far - its freezing anything and everything, regardless of the shelf. So this morning I woke up to frozen eggs (yes, all 18 of them), frozen baby carrots and get this - 2 gallons of milk and my favorite Tazo Chai Syrup. The dial is at its low but the fridge is running full throttle. Thats a good start to my Saturday.

Sometimes all this makes me feel I should throw in the towel, borrow a million or two and buy a 3 bedroom SFH in the bay area with a zero down interest only loan. At least I'd have a decent fridge :)

I voted..

2008-11-04T18:54:00.000-08:00

Just got back after voting. This is my first time voting, as I became a Naturalized Citizen in July of this year. Went to the polling station around 6 PM (Polls to close around 8 PM) - not a lot of rush. I waited in line for about 5 minutes and was done with the ballot in another 10 minutes. I've been hearing about long lines all day but lucky for me voting afterhours worked out :)

Its going to be a CNN all-nighter tonight.

God bless the USA!

Got Flash Audio Going!

2008-11-03T19:42:00.001-08:00

Finally I installed Mac4Lin as well as figured out the Flash Audio Problem. I had to install the nonfree-extrasound package. Once done, I was able to get the sound going on the upgrade.

Other than that, everything seems excellent, and I could not be happier after this upgrade.

The 64b installation is doing wonders too - the only thing that does not work on there is Citrix ICA Client. Since there is no Flash available (AFAIK) for 64b from Adobe, I got the flash working on that box through gnash.

Missing VLC on Ubuntu 8.10!

2008-11-01T03:14:00.001-07:00

Has anyone else noticed that the files that were associated with VLC (.avi, .wmv, .flv....) no longer open when you double click them after the 8.10 upgrade? When you right click to open with, do you see 2 VLCs in there, and selecting the 1st one does not do a thing, while the second one opens up VLC? Here is what I found out.

Apparently the upgrade gets rid of wxvlc which was being used to open those files. However, vlc exists, which happens to be the second 'VLC Player' you see in the context menu.

Solution? Simple. Symlinks.

mpandit@mubuntu-laptop:/usr/bin$ which wxvlc
mpandit@mubuntu-laptop:/usr/bin$ which vlc
/usr/bin/vlc
mpandit@mubuntu-laptop:/usr/bin$ sudo ln -s /usr/bin/vlc /usr/bin/wxvlc
mpandit@mubuntu-laptop:/usr/bin$ which wxvlc
/usr/bin/wxvlc

Ubuntu Upgrade to 8.10 Intrepid

2008-11-01T02:34:00.000-07:00

Today I had a chance to install Ubuntu 8.10 AMD64 fresh install, as well as upgrading my existing i386/32b Hardy Heron. I did not play around much with the AMD64 box (Holloween Friday!), but here I am at 2:40 AM trying to figure out what exactly was upgraded from the 8.04 I had on my Dell E1505. After downloading packages for about 3 hours, the setup took an extra 20 minutes. The installation was smooth, and was able to handle the wireless-connection fade for a couple minutes too. Here is what I found different when the laptop booted off with the 8.10 release.

1. User Switcher Applet is much more intuitive, and has the user's status (Online, Away, Busy...) tied to the Pidgin messenger.

2. There is a "Guest" session, which, if switched to, will lock the existing user's session (not logout), while the other user can use the box as guest. I really liked this feature. The Guest session is in the User Switcher Applet.

3. The wifi status light on the laptop started to blink. It used to do so with 7.10 but with 8.04 the light just went off. Glad to see it come back!

4. USB Startup Disk creation, pretty much like UNetbootin (http://unetbootin.sourceforge.net/) built right into the distribution.

Now some oddballs:

1. The shutdown/logout/lock screen/hibernate, etc. dialog that used to show up on clicking the shutdown button is gone. Instead I get a dialog to logout or switch user. I figured that there is a shutdown button available to add to the panel, which brings back the old dialog (and its beautiful too!).

2. Here is a bummer - hibernate *still* does not work

3. Another bummer - Audio on Flash does not work. I was able to open media files with Rythmbox and VLC - the audio was loud and clear but no sound on any flash movie. I plan on spending rest of the night/weekend to figure this one out. I've given up on hibernate anyway.

4. Cannot change the location name in the Time Applet. I can select Livermore, CA as the closest location to me (Pleasanton, CA). In 8.04 I was able to edit the location name to Pleasanton keeping the same coordinates. I guess I can figure it out in a while.

Overall, things seem to be pretty nice - not much change in the user experience, but I am yet to explore the new goodies that 8.10 introduces. Wish me luck with the Flash Audio issue - it could be a showstopper, specially since I convinced my wife to use Ubuntu on her machine.

Ok, so now 'puters are calling me!?!

2008-04-11T17:46:00.001-07:00

Five calls in a few days, and I decided to blog and moan about it. Apparently, the telemarketers are losing out to techonology. I've been getting calls that have a sales pitch followed by typically 4 options:

To repeat this message, press 1
To get more information about this service from one of our reps, press 2
To receive this call at another time or number, press 3
To remove your number from our calling list, press 4

I think this could catch on pretty well, as I could think of quite a few advantages over traditional call from telemarketers:

1. One has to listen through the whole pitch (anywhere from 1 to 5 minutes) to get to the options, which could be randomized.
2. Software does not need smoke, lunch, dinner or bio-breaks.
3. Customers, who are normally already pissed when they get such calls, cannot yell back. They can however hang up, but the software can call them right back.
4. ROI is pretty good compared to an actual human being - like spamming 5 million customers and getting for 5 to respond. It is not-humanely-possible to call 5 million customers. The software can dial out to multiple customers in parallel, being more efficient for a 5-in-a-5-million target.
5. The business has to invest one-time in this service and there are no recurring costs to deal with, unlike salaries and benefits.
6. Software can telemarket different products across different segments and timezones. No longer a human being will need to be trained to make calls for long distance services, and for magazine subscriptions separately. A perfect mix and match opportunity.
7. It could be a hosted solution, where the businesses will not even need to buy outbound phone lines. They can provide the pitch as an audio, an inbound number to transfer for more info (from a real human being) and the target leads. The human will be happy to speak to someone who is really bought into the pitch instead of convincing them upfront.
8. The pitch will not have any accent, hence the folks wont get ticked off, even though the product being sold or supported is made in China.

I can think of a few more, but I'd end here.

Determining HTTP Server Info from HTTP Response Headers

2008-04-09T20:40:00.000-07:00

Just for the heck of it, I tried sending HTTP HEAD requests to a bunch of hosts to check out the variances among different type of servers, and to see what additional information I can get (like PHP version) within the Server header. I kind of think that the Server header can speak a lot about the web server, just like 'User Agent' can for a client.

Here are some samples ...

Resin/3.1.5
Apache
Apache/2.2.8 (Unix)
Apache/2.0.55 (Ubuntu) DAV/2 SVN/1.3.1 mod_jk/1.2.14 PHP/5.1.2 mod_ssl/2.0.55 OpenSSL/0.9.8a
lighttpd/1.4.19
Microsoft-IIS/6.0
BIG-IP
Apache/2.0.54 (Fedora)
Apache-Coyote/1.1
IBM_HTTP_Server/2.0.47.1-PK44274 Apache/2.0.47 (Unix) DAV/2
Sun-Java-System-Web-Server/6.1
Netscape-Enterprise/4.0
Apache/2.0.52 (CentOS)
Apache/2.0.55 (Unix) PHP/4.3.10
Netscape-Enterprise/6.0
Sun-ONE-Web-Server/6.1
AkamaiGHost

There is also the X-Powered-By header which can provide additional info about the components, like:

Servlet 2.4; JBoss-4.0.4.GA (build: CVSTag=JBoss_4_0_4_GA date=200605151000)/Tomcat-5.5
Servlet/2.4,JSP/2.0
ASP.NET
PHP/5.1.2
PHP/4.3.9

Pretty interesting stuff I'd think.

Hello, world!

2008-03-12T03:41:00.000-07:00

My family and friends say that I am quite funny and insightful at the same time, and I should start a blog. So here I am - more pearls of contrarian wisdom on day-to-day life, technology and anything in between coming soon!