Dovecot Returns No New Mail

I have always regarded forums as a way for 1 persons experience to save the struggle for another. When I started this blog, I set out to offer my own experiences and fixes to help others that may experience the same. So I hope this reaches someone out there and is useful for them.

Last week my IT team spent several hours attempting to resolve an issue where 1 of our MX servers would return 0 new messages when, clearly there were over 60,000 new email between /home/vmail/%mailbox%/MailDir/cur and new. Now, we have a complex mail setup that pulls email from dovecot mailbox and inserts the email into a database for the the application. There are several layers involved, so troubleshooting a system like that can take several hours, going through each layer to determine if the program that picks up the mail was faulty or the dovecot server itself.

Below are the steps i took to troubleshoot the issue.

• Log into the MX server and run [ # tail -f /var/log/dovecot.log]
• open email client or start email client program to fetch new mail
• watch dovecot logs to determine if another program is picking up the mail prior to our mail program is able to fetch the new mail.
• Rule out another process picking up the mail.
• watch /home/vmail/%mailbox%/MailDir/new for new incoming mail.
• once mail is in new queue, run mail client to fetch new mail.
• At this point i noticed mail being moved from “new” to “cur” while the mail client still reported no new mail to pick up
• Next I looked at the mail client to see when the last new message was picked up. I made not of the date stamp on the last mail message that was picked up.
• I then went back to the dovecot MX server and ran [ # ls-l /home/vmail/%mailbox%/MailDir/ ]
• this returns a few directories and a few key files that are the focus of this blog post. Dovecot.index, dovecot.index.cache, and dovecot.index.log. These files are what dovecot uses to make loading of mail in the mailbox faster. They are an index of the transactions on that particular mailbox. What i noticed was that the dovecot.index file, had the same timestamp as the timestamp of the last successful email that was received by the client. Even though new mail was obviously coming in, the dovecot index was not updating.
• I then, renamed all three files to dovecot.index.old, dovecot.index.cache.old, and dovecot.index.log.old.
• I proceeded to restart the dovecot service by running [ # services dovecot restart]
• Verified that dovecot.index, dovecot.index.cache, and dovecot.index.log were rebuilt after the service restarted
• then I restarted my mail client and fetched new mail.
• BAM! 60,000 new messages now are seen and fetched.

I hope this finds usefulness to anyone out there having the same issue. Feel free to leave your comments or questions anytime!. Good Luck!.

Advertisement

IT Team Lessons Learned

I wanted to take the opportunity to share a bit about my experiences over the past year. I apologize for being a bit quiet recently. With new opportunities, comes new challenges. Over the past year, I have had the great opportunity to lead a team of IT admins. It has indeed had its challenges. Mixtures of personalities, business habits, and a rapid changing business model.

Granted when businesses are born, IT teams develop certain habits. As well as good habits unfortunately they also develop bad habits. These habits become SOP (Standard Operating Procedure) when handling implementations, service requests, and deployments. All IT teams go through this, and can be crippling as the business grows. Quality standards suffer, project timelines divert drastically, and SLA’s go un met. It can be a rather impossible task to get a ship back on course once it diverts. as that ship gets bigger, that task becomes more daunting.

It takes a strong leadership and team to realize that a ship has strayed off course, admit there is an issue, and make a clear objective to right it. In fact realizing ways to improve and change our methods as an IT team should never stop, regardless if you are able to correct core issues that may affect the teams deliverables. As the business objectives/needs change, technology advances, budgets shrink and grow, so should the IT department look for ways to improve services.

Other then looking for ways to improve services, we must also be willing to look at our methods as a team and ways to be more efficient. Today businesses constantly are looking for ways to streamline and become more efficient. As an IT team, we should be doing the same. One thing i have noticed, is that the way IT operates sometimes can suffer by the standard of “That is just the way we have always done it” syndrome. as an IT team we can loose our credibility when our methods become ineffective our quality suffers as well as our standards. We need to be able to rip off the bandaid at times, open up the wounds and look objectively for ways to improve rather then blame and perpetuate our problems. Sometimes admitting false and flaws can feel like career suicide. However i must object to this. Some of the greatest leaders of the largest corporations find themselves pointing at themselves for failures. It isn’t that we make mistakes that kills our careers, it is failure to acknowledge and develop a plan to right the failures that will ultimately lead to our demise. Many leaders inherit predecessors flawed plans or objectives. The best level 5 leaders out there are willing to identify and make the drastic and difficult to choice to right the flaws, so that the business can prosper. At all levels, leader or not, shouldn’t we do the same?

Processes I have found, can cripple an IT department if they are over done and not done enough. To many processes can frustrate and demoralize staff when it takes weeks to make a simple IP change due to paperwork. To little processes can stunt a businesses growth because it creates enormous chaos, lack of direction and demoralization as well. Staff will lack drive to improve and invest in the business because suddenly everything is a priority 1. So everything because rushed and half done, leading to a severely suffering quality standard. IT teams will stop growing with the business, and become more reactive to situations because now they cannot keep up.

So time to stop the cycle, rip off the bandaid, collaberate and find ways to improve a crippled system. Stop blaming the process, or lack there of and find a way to fix it. The greatest minds of our generation found ways to solve issues rather then let someone else fix it. They have become successful and legendary because they had a clear objective to get the best and brightest out there team to identify the issues, devise a plan and path to resolve, and then set out to change the course.