c# - Windows kernel queuing outbound network connections -

we have application (a meta-search engine) must make 50 - 250 outbound http connections in response user action, frequently.

the way creating bunch of httpwebrequests , running them asynchronously using action.begininvoke. uses threadpool launch web requests, run synchronously on own thread. note way .net 2.0 app , there no tpl speak of.

using etw (our event sources combined .net framework , kernal ones) , netmon while thread pool can start 200 threads running our code in 300ms (so, no threadpool exhaustion issues here), takes variable amount of time, 10 - 15 seconds windows kernel make tcp connections have been queued up.

this obvious in netmon - see around 60 - 100 tcp connections open (syn) (the number varies, it's never more around 120), rest trickle in on period of time. it's if connections being queued somewhere, don't know , don't know how tune can perform more concurrent outgoing connections. perfmon outbound connection queue stays @ 0 in connections established counter can see initial spike of connections gradual increase rest filter through.

it appear latency endpoints connecting play part, running code close endpoints connects doesn't show problem significantly.

i've taken comprehensive etw traces there no decent documentation on many of microsoft providers, i'm sure.

any advice work around or advice on tuning windows large amount of outgoing connections great. platform win7 (dev) , win2k8r2 (prod).

it looks slow dns queries culprit here. looking @ etw provider "microsoft-windows-networking-correlation", can trace network call inception connection , note many connections taking > 1 second @ dns resolver (microsoft-windows-rpc).

it appears our local dns server slow/can't handle load throwing @ , isn't caching aggressively. production wasn't showing severe symptoms prod dns servers right.