While automating a process involving a browser based application a popup window from the browser...

Set the HKEY CURRENT USER\Software\Microsoft\Internet Explorer\Main\TabProcGrowth Registry key to 2

Change the scope of the process so that the popup window does not need to be automated

Create a new separate object that attaches to the running instance of the popup and attempt to spy elements with this object instead.

Use Surface Automation techniques to interface with the popup window