ActionScript 3.0 :: App Keeps Getting Slower And Becomes Useless

I have a simple application where you can sort isometric boxes on a grid. It works just fine but the more often you apply changes (i.e. click on a box and drag and drop it) the slower the app gets. After about 10 changes it has something like a 5 second delay in response and becomes completely useless.

As long as I stay on the first MOUSE_DOWN it works as fast as it should, but speed decreases with every click. Strangely CPU-levels will always return to near zero when I stop doing things in the app and always go up to the same level when working, so I do not think it that there is some procedure I call while dragging and forget to close again (I also checked this about a hundred times in the code).

Basically I think it must something with the most basic event listeners that look like this:
Code:
container.addEventListener(MouseEvent.MOUSE_UP, dropElement);
container.addEventListener(MouseEvent.MOUSE_DOWN, dragElement);
function dragElement(e:Event):void{
currentElement = currentDropTarget = previousDropTarget = Kiste(e.target);
currentElement.alpha = 0.5;
stage.addEventListener(Event.ENTER_FRAME, whileDrag);
}function dropElement(e:Event):void{
currentElement.alpha = 1;
stage.removeEventListener(Event.ENTER_FRAME,whileDrag);
}
I also checked if my "whileDrag"-function keeps running but it doesn't.