Two Great Tastes: IDA + WinDbg

Setting up remote debugging with IDA+WinDbg can be difficult when it doesn't work right off the bat, because the errors don't jog the right thought process for you to fix the setup and get it working. This causes some people to walk away from the whole thing, which is unfortunate. This setup is SOOOOO useful that it's worth slogging through the pain to get it working. The value of having graph view and IDA's annotation capabilities on-hand while debugging cannot be overstated.

Here, I'll emphasize one thing that could stand to be better emphasized in Hex-Rays' own documentation: you have to be using the same version of WinDbg on each side. And I'll indicate some ways to isolate end-to-end (E2E) issues. Note that the system with IDA Pro on it is referred to here as the analysis system (it's where you do your analysis of the code), and the system where you run malware is referred to as the target system.

Pointers

• Resolve any end-to-end (E2E) issues first (firewalls, networking, etc.)
• Lock IDA Pro into using the same version of WinDbg as is on your target system
• Use WinDbg itself to verify that there are no E2E issues

Algorithm

This is exactly how to set up a remote debugging setup with IDA Pro and WinDbg. Here are the steps:

1. Both systems: Ensure your analysis and target machines can access each other over the network
1. If they are VMs, you may need to adjust them to ensure they are both host-only
2. You might need to mess with firewall settings
3. If you are using FakeNet-NG, you might need to add an exception for dbgsrv.exe
2. Target system: Locate (install, if necessary) WinDbg on your target system.
3. Target -> Analysis system: If you haven't installed the same version of WinDbg to both systems, then simply copy the entire x86 directory where you located WinDbg on the target system, onto your analysis system. It doesn't matter where you place this.
4. Analysis system: Edit ida.cfg to set DBGTOOLS to point to the x86 directory
1. Use double backslashes, e.g. DBGTOOLS = "C:\\Program Files (x86)\\Windows Kits\\10\\Debuggers\\x86\\";
5. Target system: Start the WinDbg debug server
1. "C:\path\to\dbgsrv.exe" -t tcp:port=9999
6. Analysis system: Test by trying to connect remotely with WinDbg itself - if this doesn't work, then you've got end-to-end issues to resolve before IDA will work
7. Analysis system: configure your IDB to use WinDbg:
1. Debugger -> Switch debugger... (select Windbg debugger and click OK)
2. Debugger -> Options...
1. Application: path\on\your\target\system\to\binary.exe
2. Input file: path\on\your\target\system\to\binary.exe
3. Directory: path\on\your\target\system\to
4. Parameters: command-line arguments you want passed to the malware (if any)
5. Connection string: tcp:server=TARGETSYSTEMNAME,port=9999
6. Click OK 
8. Analysis system: Click on an instruction and hit F4 to "run to" that instruction, or set a breakpoint and strike F9
9. Disregard warnings as applicable ;-)

Troubleshooting

You may want to audit your user and system PATH environment variables to ensure that they don't include the x86 directory of a conflicting version of WinDbg, or the x64 directory for that matter.

If you get "Could not initialize WinDbg engine 0x7f / The specified procedure could not be found... You might try adding the path to that x86 directory to your system path and closing/reopening IDA. I also find that certain Python scripts seem to cause IDA Pro to emit this error, so you might also try closing/reopening IDA, initiating your debug session, and only THEN loading any ancillary IDAPython scripts you were using.

Miscellany

As of 2011, Hex-Rays indicated that this would not work with the x64 tools.