This is part 2 of my troubleshooting weeks where we were facing issues with Outlook losing connections to Exchange intermittently AND Database copies losing synchronization as well. The part 1 of this blog focused on what\/how we troubleshoot so far mostly using database connectivity alerts. In this part we\u2019ll be focusing on Outlook disconnections and how things got heated up for next few days.<\/p>
I believe you will all agree that most annoying and most interesting issue for troubleshooting with Outlook & Exchange is?\u00a0 Outlook intermittently disconnects or pops a message saying \u201cOutlook lost connection to server, trying to reconnect\u201d<\/p>
The pop-up basically indicates that Outlook either lost MAPI connection to exchange services or\/and is taking very long time to make complete connection causing Outlook to exhibit behavior such as it has lost connectivity.<\/p>
<\/p>
<\/p>
The issue in itself doesn\u2019t causes any critical business loss, it\u2019s more of an annoyance and if users in your environment are as \u201cpeculiar\u201d about Outlook\u2019s well-being then you should be facing same level of annoyance as we do. Sometimes the level of annoyance is so high, that you feel to mask the problem using custom registry discussed below with blog ;-).<\/p>
So here I am, discussing my mid-month of June where we had one such painful and interesting issue to troubleshoot where Outlook was generating RPC disconnection pop-up and lot of experts were basically head scratching along with us to remediate same.<\/p>
Stage 2: Outlook intermittent Disconnections<\/p>
The second and basically more critical sign of issues started happening few days after the database connectivity issues discussed in part 1 of this blog. Basically we started users complaining about:<\/p>
The Exchange hardware report<\/a>\/SCOM was not triggering any alerts for performance or RPC latency. When running server\u2019s MAPI connectivity test, it failed & below error was reported:<\/p> ____________________________________________________________________________________________<\/p> Error : [Microsoft.Exchange.Data.Storage.TooManyObjectsOpenedException]: Cannot open mailbox \/o=Contoso\/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)\/cn=Configuration\/cn=Servers\/cn=Server01\/cn=Microsoft System Attendant. Inner error [Microsoft.Mapi.MapiExceptionSessionLimit]: MapiExceptionSessionLimit: Unable to open message store. (hr=0x80040112, ec=1246)?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0 Diagnostic context:<\/p> _______________________________________________________________________________________________<\/p> The error basically indicates issue with system attendant out of RPC connections. However it was not adding up as we already had increased exchange store limits documented here<\/a>, majorly Maximum Allowed Sessions Per Userwas already set as high as 5000 (decimal). Running ExMon (Exchange User Monitor tool)<\/a> generally helps in these scenarios as it points out which user\/mailbox is spending maximum CPU time in server or is having highest operations against exchange store, but not in our scenario sadly.<\/p> After some expert help, we were able to determine that issue was not due to total number of MAPI connections, but was due to Exchange administrative connections open at a given time against exchange server store. The Exchange administrative connections are primarily used by:<\/p> Default value for exchange store limits for administrative connections Maximum Allowed Exchange Sessions Per Service is 10000, however as per performance monitors, we were going above it at which point exchange store was locking down all connections causing Outlook outage. At this point we increased the following registries to make sure we do not go into outage situation again:<\/p> And from here we monitored if we go above same. But for me, it was still not adding up as we were running fine all this time, how can we spike connections overnight?<\/p> Meanwhile, the Outlook continuous prompts for disconnection and reconnection kept bugging users intermittently. We provided store dump for analysis using below command and were told that the administrative connections are still spiking and we need to bump them up to 30000 for resolving issues, but I still wasn\u2019t sold on same.<\/p> After going through some network captures, logon statistics & RPC log dumps from Exchange CAS and MBX servers, I noticed below:<\/p> 2586723?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a03:02:58 PM 6\/21\/2013?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a01343.5206036?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0store.exe?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0Server01?\u00a0 ?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a010.0.0.2?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0TCP?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0TCP:[Segment Lost]Flags=\u2026A\u2026., SrcPort=38364, DstPort=32159, PayloadLen=1460, Seq=1546335157 \u2013 1546336617, Ack=3329138042, Win=511?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0{TCP:361, IPv4:11}<\/p> 2586724?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a03:02:58 PM 6\/21\/2013?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a01343.5206181?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0store.exe?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0Server01?\u00a0 ?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a010.0.0.2?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0TCP?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0TCP:[Continuation to #2586723]Flags=\u2026A\u2026., SrcPort=38364, DstPort=32159, PayloadLen=279, Seq=1546336617 \u2013 1546336896, Ack=3329138042, Win=511?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0{TCP:361, IPv4:11}<\/p> 2586737?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a03:02:58 PM 6\/21\/2013?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a01343.5227346?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0store.exe?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a010.0.0.2?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0Server01?\u00a0 ?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0TCP?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0TCP:[Request Fast-Retransmit #2586723]Flags=\u2026A\u2026., SrcPort=32159, DstPort=38364, PayloadLen=0, Seq=3329138042, Ack=1546335157, Win=3045?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0?\u00a0{TCP:361, IPv4:11}<\/p> At this point, I turned back to network team who\u2019ve given up on us and pushed them hard to check the network switches connected to our exchange VM farm, took some pushing but at the end they expanded monitoring traffic at switch level from not just exchange but basically all traffic coming to switch, at that point they were able to determine the cause of the issue.<\/p> The cause appears to have been another VM server in the data center that had 2 trunk ports that were flapping.?\u00a0 When the ports would flap it would trigger a spanning-tree change on the LAN.?\u00a0 During the spanning-tree re-convergence, traffic would get flooded out through all ports on the core switch.?\u00a0?\u00a0 The switch ports connected to this server were set up for \u2018spanning-tree portfast\u2019 (disable spanning tree) but since they were trunk ports this command was not preventing the spanning-tree change during a port flap.?\u00a0 We had to disable spanning-tree on each trunk port with the \u2018spanning-tree portfast trunk\u2019?\u00a0 command.?\u00a0?\u00a0 This stopped the spanning-tree issues during the port flaps.?\u00a0 The server was found to be down and was moved to a powered down state.?\u00a0 This stopped the port flaps from occurring.<\/p> So in a nut-shell, the powered down VM server was flooding traffic against all switches connected to it, exchange switch being one of them. Hence the switch getting overwhelmed, servers sending\/receiving Rx Pause packets as they\u2019re unable to keep up with network data speed and hence the packet loss and hence the database disconnections and hence the Outlook intermittent connectivity issue ! Phew !<\/p> Once the issue was determined, we removed the below two registries from server such that we do not allow more than default connections available for Exchange 2010 store service and added exchange admin client connection column in our Exchange hardware report<\/a> for monitoring:<\/p> Since then, the administrative connections which were going above 20000 now stay below 2000 even at peak business day, no more Outlook intermittent dis-connectivity, no more database synchronization issue and hence my interesting yet painful mid-month of June ended so I can take on other issues waiting in queue \ud83d\ude09<\/p> Other frequent factors that impact Outlook intermittent disconnections:<\/p> To read more about other factors that cause issue where users observe Outlook RPC dialog box discussed above, please read the MS blog below:<\/p>The Maximum Allowed Exchange Sessions Per Service registry to 20000 (default value 10000)<\/div><\/li>The Maximum Allowed Concurrent Exchange Sessions Per Service registry to 100 (default value 50)<\/div><\/li>The Maximum Allowed Service Sessions Per User registry to 64 (default value 32)<\/div><\/li><\/ul>All users have high number of connections to the server irrespective of client version or device they??\u2122re using to connect or server they??\u2122re on. Also in the logon statistics dump, I could see passive nodes of database copy were having logon connection against mailboxes as well other than CAS servers. At this time I learnt that search service also uses administrative connections and it is by Design that the Microsoft Exchange Search Indexer service on the passive node(s) indexes only the active node database. The design is that, in this way,?\u00a0 all content indexes of the database on all database copies of the DAG are always up to date. (Get-LogonStatistics Shows Logons to a Mailbox on the Active Node by the Exchange Search Indexer on the Passive Node(s) of a DAG in Exchange 2010<\/a>) So when we increased the registry for service sessions above, we basically allowed exchange search service to open more connections against mailboxes on database.<\/div><\/li>Parsing through CAS logs, I can see lot of session drops, attached is connection log for certain users.<\/div><\/li><\/ol>