Ventrilo Latency issue.

[KillSlim]

It doesn't matter what server I connect to, latency is a major issue.
Previously this has never happened to me, but it seems more and more people are getting the same issue.
Here's the issue: Whenever someone in the same channel talks, my ping rockets to several thousand and will stay that way for minutes at a time.

I've confirmed that there's no viruses or spyware on my computer, so I don't know what on earth is happening.
Help would be greatly appreciated.

Edit-
Internet is fine on all other applications except Ventrilo.
This has only started happening to me tonight around an hour ago.