ActionScript 3.0 :: Remove Stage Event Listeners?

I create a stage keyboard event listener in an externally loaded swf such as:

Code:
stage.addEventListener(KeyboardEvent.KEY_DOWN, onDown);

My question is how and where I should remove this event listener? I have added an unload event handler to the external swf like this:

Code:
this.loaderInfo.addEventListener(Event.UNLOAD, unloadHandler);

The unloadHandler code looks like this:

Code:
private function unloadHandler(event:Event):void {
stage.removeEventListener(KeyboardEvent.KEY_DOWN, onDown);
}

When the container swf unloads the external swf, unloadHandler() gets called, but with the following error:

TypeError: Error #1009: Cannot access a property or method of a null object reference.

Where should the keyboard event be removed?