DNS slowly falling apart
to start, give short description of our network setup (from way understand it). have 2 stores. we'll call them cp, , hq. hq domain controller, , have local domain called billsgs.net. each store operates on own. each have firewall, , own server running windows server 2008 r2. time interact through replication. have specified replicated directories, user profiles, , our database files. backup part.
now onto problem... few weeks ago (early june) noticed replication service on hq server hogging ton of memory, , ton, mean of available memory hands on. have 13gbs , within 10 minutes of running dfs 98% memory usage. stopped it. havent been bothered this, if crashes, pretty screwed on backups. have ran hot fixes nothing has worked. of right now, dfs not running. now, couple of weeks ago firewalls operating system corrupted, have no idea how, wasn't there when happened. @ hq store. have broken firewall , dfs isn't working properly. have reinstalled operating system on firewall, pfsense. seemed working fine.. except start noticing dns problems. @ point don't know if related dns/ad/dfs issues or if related firewall issues. have firewall open, have decided that not problem, @ least doesn't seem it. here few debugging things have done...
here dcdiag output...
c:\users\administrator>dcdiag directory server diagnosis performing initial setup: trying find home server... home server = bgs-hq-vrdsvr01 * identified ad forest. done gathering initial info. doing initial required tests testing server: bgs-hq\bgs-hq-vrdsvr01 starting test: connectivity ......................... bgs-hq-vrdsvr01 passed test connectivity doing primary tests testing server: bgs-hq\bgs-hq-vrdsvr01 starting test: advertising ......................... bgs-hq-vrdsvr01 passed test advertising starting test: frsevent there warning or error events within last 24 hours after sysvol has been shared. failing sysvol replication problems may cause group policy problems. ......................... bgs-hq-vrdsvr01 passed test frsevent starting test: dfsrevent ......................... bgs-hq-vrdsvr01 passed test dfsrevent starting test: sysvolcheck ......................... bgs-hq-vrdsvr01 passed test sysvolcheck starting test: kccevent warning event occurred. eventid: 0x8000082c time generated: 08/05/2011 15:04:12 event string: warning event occurred. eventid: 0x8000082c time generated: 08/05/2011 15:05:12 event string: ......................... bgs-hq-vrdsvr01 passed test kccevent starting test: knowsofroleholders ......................... bgs-hq-vrdsvr01 passed test knowsofroleholders starting test: machineaccount ......................... bgs-hq-vrdsvr01 passed test machineaccount starting test: ncsecdesc ......................... bgs-hq-vrdsvr01 passed test ncsecdesc starting test: netlogons ......................... bgs-hq-vrdsvr01 passed test netlogons starting test: objectsreplicated ......................... bgs-hq-vrdsvr01 passed test objectsreplicated starting test: replications [replications check,bgs-hq-vrdsvr01] recent replication attempt failed: bgs-cp-vrdsvr01 bgs-hq-vrdsvr01 naming context: dc=forestdnszones,dc=billsgs,dc=net replication generated error (1908): not find domain controller domain. failure occurred @ 2011-08-05 14:34:49. last success occurred @ 2011-08-05 13:51:35. 1 failures have occurred since last success. kerberos error. kdc not found authenticate call. check sufficient domain controllers available. [replications check,bgs-hq-vrdsvr01] recent replication attempt failed: bgs-cp-vrdsvr01 bgs-hq-vrdsvr01 naming context: dc=domaindnszones,dc=billsgs,dc=net replication generated error (1908): not find domain controller domain. failure occurred @ 2011-08-05 14:34:48. last success occurred @ 2011-08-05 13:51:35. 1 failures have occurred since last success. kerberos error. kdc not found authenticate call. check sufficient domain controllers available. [replications check,bgs-hq-vrdsvr01] recent replication attempt failed: bgs-cp-vrdsvr01 bgs-hq-vrdsvr01 naming context: cn=schema,cn=configuration,dc=billsgs,dc=net replication generated error (1908): not find domain controller domain. failure occurred @ 2011-08-05 14:34:47. last success occurred @ 2011-08-05 13:51:34. 1 failures have occurred since last success. kerberos error. kdc not found authenticate call. check sufficient domain controllers available. [replications check,bgs-hq-vrdsvr01] recent replication attempt failed: bgs-cp-vrdsvr01 bgs-hq-vrdsvr01 naming context: cn=configuration,dc=billsgs,dc=net replication generated error (1908): not find domain controller domain. failure occurred @ 2011-08-05 14:34:46. last success occurred @ 2011-08-05 13:51:34. 1 failures have occurred since last success. kerberos error. kdc not found authenticate call. check sufficient domain controllers available. [replications check,bgs-hq-vrdsvr01] recent replication attempt failed: bgs-cp-vrdsvr01 bgs-hq-vrdsvr01 naming context: dc=billsgs,dc=net replication generated error (1908): not find domain controller domain. failure occurred @ 2011-08-05 14:34:46. last success occurred @ 2011-08-05 13:51:34. 1 failures have occurred since last success. kerberos error. kdc not found authenticate call. check sufficient domain controllers available. ......................... bgs-hq-vrdsvr01 failed test replications starting test: ridmanager ......................... bgs-hq-vrdsvr01 passed test ridmanager starting test: services invalid service startup type: dfsr on bgs-hq-vrdsvr01, current value disabled, expected value auto_start dfsr service stopped on [bgs-hq-vrdsvr01] ......................... bgs-hq-vrdsvr01 failed test services starting test: systemlog warning event occurred. eventid: 0x00000458 time generated: 08/05/2011 14:08:10 event string: group policy client side extension folder redirection unable apply 1 or more settings because changes must processed before system startup or u ser logon. system wait group policy processing finish before next startup or logon user, , may result in slow startup , boot p erformance. error event occurred. eventid: 0x00000456 time generated: 08/05/2011 14:23:08 event string: processing of group policy failed. windows not determine if user , computer accounts in same forest. ensure user domain name matches th e name of trusted domain resides in same forest computer account. error event occurred. eventid: 0xc0001b78 time generated: 08/05/2011 14:28:16 event string: service control manager tried take corrective action (restart service) after unexpected termination of dfs replication service, actio n failed following error: error event occurred. eventid: 0xc000271a time generated: 08/05/2011 14:31:28 event string: server {995c996e-d918-4a8c-a302-45719a6f4ea7} did not register dcom within required timeout. warning event occurred. eventid: 0x8000001d time generated: 08/05/2011 14:34:09 event string: key distribution center (kdc) cannot find suitable certificate use smart card logons, or kdc certificate not verified. smart card logon m ay not function correctly if problem not resolved. correct problem, either verify existing kdc certificate using certutil.exe or enroll new kdc certi ficate. warning event occurred. eventid: 0x000003f6 time generated: 08/05/2011 14:34:13 event string: name resolution name billsgs.net timed out after none of configured dns servers responded. error event occurred. eventid: 0xc0001b58 time generated: 08/05/2011 14:34:48 event string: dgivecp service failed start due following error: error event occurred. eventid: 0x0000168e time generated: 08/05/2011 14:34:55 event string: dynamic registration of dns record '6282bfca-ade1-41c8-84dc-516ce19b49be._msdcs.billsgs.net. 600 in cname bgs-hq-vrdsvr01.billsgs.net.' failed on follo wing dns server: error event occurred. eventid: 0x0000168e time generated: 08/05/2011 14:34:56 event string: dynamic registration of dns record '_kpasswd._udp.billsgs.net. 600 in srv 0 100 464 bgs-hq-vrdsvr01.billsgs.net.' failed on following dns server: warning event occurred. eventid: 0x00002724 time generated: 08/05/2011 14:34:56 event string: computer has @ least 1 dynamically assigned ipv6 address.for reliable dhcpv6 server operation, should use static ipv6 addresses. warning event occurred. eventid: 0x000003f6 time generated: 08/05/2011 14:34:55 event string: name resolution name billsgs.net timed out after none of configured dns servers responded. error event occurred. eventid: 0xc00110f1 time generated: 08/05/2011 14:35:09 event string: wins server not initialize security allow read-only operations. error event occurred. eventid: 0xc0002720 time generated: 08/05/2011 14:36:05 event string: application-specific permission settings not grant local launch permission com server application clsid warning event occurred. eventid: 0x000727aa time generated: 08/05/2011 14:38:30 event string: winrm service failed create following spns: wsman/bgs-hq-vrdsvr01.billsgs.net; wsman/bgs-hq-vrdsvr01. warning event occurred. eventid: 0x0000043d time generated: 08/05/2011 14:47:48 event string: windows failed apply folder redirection settings. folder redirection settings might have own log file. please click on "more information" link. error event occurred. eventid: 0x0000168e time generated: 08/05/2011 15:02:25 event string: dynamic registration of dns record '6282bfca-ade1-41c8-84dc-516ce19b49be._msdcs.billsgs.net. 600 in cname bgs-hq-vrdsvr01.billsgs.net.' failed on follo wing dns server: error event occurred. eventid: 0x0000168e time generated: 08/05/2011 15:02:26 event string: dynamic registration of dns record '_kpasswd._udp.billsgs.net. 600 in srv 0 100 464 bgs-hq-vrdsvr01.billsgs.net.' failed on following dns server: ......................... bgs-hq-vrdsvr01 failed test systemlog starting test: verifyreferences ......................... bgs-hq-vrdsvr01 passed test verifyreferences running partition tests on : forestdnszones starting test: checksdrefdom ......................... forestdnszones passed test checksdrefdom starting test: crossrefvalidation ......................... forestdnszones passed test crossrefvalidation running partition tests on : domaindnszones starting test: checksdrefdom ......................... domaindnszones passed test checksdrefdom starting test: crossrefvalidation ......................... domaindnszones passed test crossrefvalidation running partition tests on : schema starting test: checksdrefdom ......................... schema passed test checksdrefdom starting test: crossrefvalidation ......................... schema passed test crossrefvalidation running partition tests on : configuration starting test: checksdrefdom ......................... configuration passed test checksdrefdom starting test: crossrefvalidation ......................... configuration passed test crossrefvalidation running partition tests on : billsgs starting test: checksdrefdom ......................... billsgs passed test checksdrefdom starting test: crossrefvalidation ......................... billsgs passed test crossrefvalidation running enterprise tests on : billsgs.net starting test: locatorcheck ......................... billsgs.net passed test locatorcheck starting test: intersite ......................... billsgs.net passed test intersite
now, keep in mind pretty different everytime restart server. have issues related dcom being unable reach our specified dns servers! now.. here output of dns test...
c:\users\administrator>dcdiag /test:dns directory server diagnosis performing initial setup: trying find home server... home server = bgs-hq-vrdsvr01 * identified ad forest. done gathering initial info. doing initial required tests testing server: bgs-hq\bgs-hq-vrdsvr01 starting test: connectivity ......................... bgs-hq-vrdsvr01 passed test connectivity doing primary tests testing server: bgs-hq\bgs-hq-vrdsvr01 starting test: dns dns tests running , not hung. please wait few minutes... ......................... bgs-hq-vrdsvr01 passed test dns running partition tests on : forestdnszones running partition tests on : domaindnszones running partition tests on : schema running partition tests on : configuration running partition tests on : billsgs running enterprise tests on : billsgs.net starting test: dns test results domain controllers: dc: bgs-hq-vrdsvr01.billsgs.net domain: billsgs.net test: basic (basc) warning: adapter [00000007] intel(r) pro/1000 mt network connection has invalid dns server: 192.168.40.254 (<name unavailable>) test: records registration (rreg) network adapter [00000007] intel(r) pro/1000 mt network connection: warning: missing srv record @ dns server 192.168.40.13: _ldap._tcp.billsgs.net warning: missing srv record @ dns server 192.168.40.13: _ldap._tcp.22017278-29d1-493a-b72d-e44b31411a70.domains._msdcs.billsgs.net warning: missing srv record @ dns server 192.168.40.13: _kerberos._tcp.dc._msdcs.billsgs.net warning: missing srv record @ dns server 192.168.40.13: _ldap._tcp.dc._msdcs.billsgs.net warning: missing srv record @ dns server 192.168.40.13: _kerberos._tcp.billsgs.net warning: missing srv record @ dns server 192.168.40.13: _kerberos._udp.billsgs.net warning: missing srv record @ dns server 192.168.40.13: _kpasswd._tcp.billsgs.net error: missing srv record @ dns server 192.168.40.13: _ldap._tcp.bgs-hq._sites.billsgs.net error: missing srv record @ dns server 192.168.40.13: _kerberos._tcp.bgs-hq._sites.dc._msdcs.billsgs.net error: missing srv record @ dns server 192.168.40.13: _ldap._tcp.bgs-hq._sites.dc._msdcs.billsgs.net error: missing srv record @ dns server 192.168.40.13: _kerberos._tcp.bgs-hq._sites.billsgs.net warning: missing srv record @ dns server 192.168.40.13: _ldap._tcp.gc._msdcs.billsgs.net warning: missing record @ dns server 192.168.40.13: gc._msdcs.billsgs.net error: missing srv record @ dns server 192.168.40.13: _gc._tcp.bgs-hq._sites.billsgs.net error: missing srv record @ dns server 192.168.40.13: _ldap._tcp.bgs-hq._sites.gc._msdcs.billsgs.net error: missing srv record @ dns server 192.168.40.13: _ldap._tcp.pdc._msdcs.billsgs.net warning: missing cname record @ dns server 192.168.40.254: 6282bfca-ade1-41c8-84dc-516ce19b49be._msdcs.billsgs.net warning: missing record @ dns server 192.168.40.254: bgs-hq-vrdsvr01.billsgs.net error: missing srv record @ dns server 192.168.40.254: _ldap._tcp.billsgs.net error: missing srv record @ dns server 192.168.40.254: _ldap._tcp.22017278-29d1-493a-b72d-e44b31411a70.domains._msdcs.billsgs.net error: missing srv record @ dns server 192.168.40.254: _kerberos._tcp.dc._msdcs.billsgs.net error: missing srv record @ dns server 192.168.40.254: _ldap._tcp.dc._msdcs.billsgs.net error: missing srv record @ dns server 192.168.40.254: _kerberos._tcp.billsgs.net error: missing srv record @ dns server 192.168.40.254: _kerberos._udp.billsgs.net error: missing srv record @ dns server 192.168.40.254: _kpasswd._tcp.billsgs.net error: missing srv record @ dns server 192.168.40.254: _ldap._tcp.bgs-hq._sites.billsgs.net error: missing srv record @ dns server 192.168.40.254: _kerberos._tcp.bgs-hq._sites.dc._msdcs.billsgs.net error: missing srv record @ dns server 192.168.40.254: _ldap._tcp.bgs-hq._sites.dc._msdcs.billsgs.net error: missing srv record @ dns server 192.168.40.254: _kerberos._tcp.bgs-hq._sites.billsgs.net error: missing srv record @ dns server 192.168.40.254: _ldap._tcp.gc._msdcs.billsgs.net warning: missing record @ dns server 192.168.40.254: gc._msdcs.billsgs.net error: missing srv record @ dns server 192.168.40.254: _gc._tcp.bgs-hq._sites.billsgs.net error: missing srv record @ dns server 192.168.40.254: _ldap._tcp.bgs-hq._sites.gc._msdcs.billsgs.net error: missing srv record @ dns server 192.168.40.254: _ldap._tcp.pdc._msdcs.billsgs.net error: record registrations cannot found network adapters summary of test results dns servers used above domain controllers: dns server: 192.168.40.254 (<name unavailable>) 1 test failure on dns server name resolution not functional. _ldap._tcp.billsgs.net. failed on dns server 192.168.40.254 summary of dns test results: auth basc forw del dyn rreg ext _________________________________________________________________ domain: billsgs.net bgs-hq-vrdsvr01 pass warn pass pass pass fail n/a ......................... billsgs.net failed test dns c:\users\administrator>believe our main issue, i'm lost on whole thing. i've given netlogon restart trick few tries. i've ran following sequence:
net stop netlogon net stop dns ipconfig /flushdns net start dns net start netlogon
nothing seems work. recently, today, went "active directory users , computers", , under "domain controllers", hq server not listed. says unavailable. if willing provide insight mess, appreciated it!
also.. here ip config output...
microsoft windows [version 6.1.7600] copyright (c) 2009 microsoft corporation. rights reserved. c:\users\administrator>ipconfig /all windows ip configuration host name . . . . . . . . . . . . : bgs-hq-vrdsvr01 primary dns suffix . . . . . . . : billsgs.net node type . . . . . . . . . . . . : hybrid ip routing enabled. . . . . . . . : no wins proxy enabled. . . . . . . . : no dns suffix search list. . . . . . : billsgs.net ethernet adapter local area connection: connection-specific dns suffix . : description . . . . . . . . . . . : intel(r) pro/1000 mt network connection physical address. . . . . . . . . : 00-0c-29-03-ba-38 dhcp enabled. . . . . . . . . . . : no autoconfiguration enabled . . . . : yes ipv4 address. . . . . . . . . . . : 192.168.40.13(preferred) subnet mask . . . . . . . . . . . : 255.255.255.0 default gateway . . . . . . . . . : 192.168.40.254 dns servers . . . . . . . . . . . : 192.168.40.13 192.168.40.254 primary wins server . . . . . . . : 192.168.40.13 secondary wins server . . . . . . : 192.168.41.17 netbios on tcpip. . . . . . . . : enabled tunnel adapter isatap.{adec15a8-2603-40eb-964c-489ccbd11e08}: media state . . . . . . . . . . . : media disconnected connection-specific dns suffix . : description . . . . . . . . . . . : microsoft isatap adapter physical address. . . . . . . . . : 00-00-00-00-00-00-00-e0 dhcp enabled. . . . . . . . . . . : no autoconfiguration enabled . . . . : yes tunnel adapter local area connection* 11: media state . . . . . . . . . . . : media disconnected connection-specific dns suffix . : description . . . . . . . . . . . : teredo tunneling pseudo-interface physical address. . . . . . . . . : 00-00-00-00-00-00-00-e0 dhcp enabled. . . . . . . . . . . : no autoconfiguration enabled . . . . : yes c:\users\administrator> 192.168.40.13 hq , 192.168.41.17 cp. 192.168.40.254 hq firewall, , 192.168.41.254 cp firewall.
hello,
starting test: frsevent there warning or error events within last 24 hours after sysvol has been shared. failing sysvol replication problems may cause group policy problems.
check event viewer logs more information.
invalid service startup type: dfsr on bgs-hq-vrdsvr01, current value disabled, expected value auto_start dfsr service stopped on [bgs-hq-vrdsvr01] ......................... bgs-hq-vrdsvr01 failed test services
please start dfsr on bgs-hq-vrdsvr01.
event string: dynamic registration of dns record '6282bfca-ade1-41c8-84dc-516ce19b49be._msdcs.billsgs.net. 600 in cname bgs-hq-vrdsvr01.billsgs.net.' failed on follo wing dns server: error event occurred. eventid: 0x0000168e time generated: 08/05/2011 14:34:56
you have dns registration probems.
for ip configuration, change that:
windows ip configuration host name . . . . . . . . . . . . : bgs-hq-vrdsvr01 primary dns suffix . . . . . . . : billsgs.net node type . . . . . . . . . . . . : hybrid ip routing enabled. . . . . . . . : no wins proxy enabled. . . . . . . . : no dns suffix search list. . . . . . : billsgs.net ethernet adapter local area connection: connection-specific dns suffix . : description . . . . . . . . . . . : intel(r) pro/1000 mt network connection physical address. . . . . . . . . : 00-0c-29-03-ba-38 dhcp enabled. . . . . . . . . . . : no autoconfiguration enabled . . . . : yes ipv4 address. . . . . . . . . . . : 192.168.40.13(preferred) subnet mask . . . . . . . . . . . : 255.255.255.0 default gateway . . . . . . . . . : 192.168.40.254 dns servers . . . . . . . . . . . : 192.168.40.254 192.168.40.13 primary wins server . . . . . . . : 192.168.40.13 secondary wins server . . . . . . : 192.168.41.17 netbios on tcpip. . . . . . . . : enabled tunnel adapter isatap.{adec15a8-2603-40eb-964c-489ccbd11e08}: media state . . . . . . . . . . . : media disconnected connection-specific dns suffix . : description . . . . . . . . . . . : microsoft isatap adapter physical address. . . . . . . . . : 00-00-00-00-00-00-00-e0 dhcp enabled. . . . . . . . . . . : no autoconfiguration enabled . . . . : yes tunnel adapter local area connection* 11: media state . . . . . . . . . . . : media disconnected connection-specific dns suffix . : description . . . . . . . . . . . : teredo tunneling pseudo-interface physical address. . . . . . . . . : 00-00-00-00-00-00-00-e0 dhcp enabled. . . . . . . . . . . : no autoconfiguration enabled . . . . : yes
once done, run ipconfig /registerdns , restart netlogon on dc.
similar thing other dc, let point other 1 ip address primary dns server , private ip address secondary one. can add 127.0.0.1 third one.
also, please check needed ports ad replication opened: http://technet.microsoft.com/en-us/library/bb727063.aspx
this posting provided "as is" no warranties or guarantees , , confers no rights.
microsoft student partner 2010 / 2011
microsoft certified professional
microsoft certified systems administrator: security
microsoft certified systems engineer: security
microsoft certified technology specialist: windows server 2008 active directory, configuration
microsoft certified technology specialist: windows server 2008 network infrastructure, configuration
microsoft certified technology specialist: windows server 2008 applications infrastructure, configuration
microsoft certified technology specialist: windows 7, configuring
microsoft certified professional: enterprise administrator
Windows Server > Windows Server General Forum
Comments
Post a Comment