We'll do a full post-game analysis in a future posting for now we're all working very hard to maintain a good user experience on our application.Ĭontact us at if there's anything we can do to help. (And while we don't anticipate the need to bring the site offline again for further changes, we'll reserve that option if the situation requires it.)Īll of us are truly sorry for the frustration and inconvenience this has caused many of our customers. Traffic has been very smooth this afternoon but load has been fairly low so we're still being cautious. ![]() ![]() If any problems do arise, we'll know immediately and we'll keep you posted on this blog. This has since been rectified and all system diagnostics indicate that. This issue was related to one of our load-balancing servers. 7:00 am Our Web Operations team located an issue that may have impacted some customers’ ability to login between 2 and 6 AM Pacific Daylight Time. We're also maintaining enhanced monitoring procedures overnight and into the typical Tuesday morning busy period. ClickTime Login Issues Resolved July 21, 2020. More complex changes will be made over the coming days and weeks. I hope anyone with questions will contact us: Author ClickTime Posted on FebruCategories Status Updates Leave a comment on Performance back to normal Feb 2nd End-of-Day UpdateĪs of this evening (Feb 2) we have implemented certain site changes that we believe will mitigate the root cause of today's performance problems. We've now got a clear idea of how to remedy those flaws. We recently expanded our hardware capacity to handle future transactional load, but the plumbing between those new systems couldn't keep up. Ironically, yesterday's problems were the result of mistakes made pursuing that goal. Some customers have asked if we're unprepared to handle the load of a growing customer base. Lastly we are accelerating our plans to totally replace our session-handling system with something many times faster. In coming releases we're going to limit the ability to re-try certain actions to avoid multiplying the load of any one user. More changes along those lines will come today. This is a step towards fixing the underlying cause of the original speed issue. So we've done several things to fix this: yesterday afternoon we made improvements to streamline our session-management. We believe this happened yesterday because a threshold was crossed where our session-management found its limit and became congested. We pass information between those servers behind-the-scenes to maintain a seamless user experience as users bounce from one machine to another.) This slowness led our users to re-attempt their actions and logins more and more often, which created more slowness, and this compounded itself. ![]() (A bit of background: all actions taken by our users are spread out across numerous web servers for scalability and maximum speed. In brief, we experienced excessive slowness while user sessions were being passed across our farm of web servers. It's not quite time to declare victory, but we do think that ongoing site performance should remain up to our standards moving forward. We've been closely monitoring our systems throughout the day today (Feb 3rd) and site speed has been better than average for a Tuesday.
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |