Well it would appear that switching out the D-Link NIC for the 3COM card fixed the problem. For the first time, I successfully transcoded a movie, which definately makes me think that NIC was conflicting with one of the other devices it was sharing an IRQ with (probably the Nvidia board). Plus network access with a lot of bandwidth moving through it doesn't crash Myth like it did before.

Many thanks to everyone! I'll post an update if it does end up crashing, but hopefully I'm now ready to start running this next to my Tivo!