問題
Q1
Why by Nvidia motherboard's DVI output cannot use with D-Sub monitor?

This is chipset hardware limitation for the motherboard.

So please connect DVI header with only DVI monitor, then the display will work properly.