zydronium/imapsync
1
0
Fork 0
mirror of https://github.com/imapsync/imapsync.git synced 2025-06-10 14:44:32 +02:00
Code Issues Projects Releases Packages Wiki Activity

1.836

Browse source
This commit is contained in:
Nick Bebout 2017-09-23 16:54:48 -05:00
parent 3afeea4a16
commit 8d76e44c5e
243 changed files with 57452 additions and 10330 deletions
Download patch file Download diff file Expand all files Collapse all files

32
FAQ.d/FAQ.Admin_Authentication.txt Normal file
View file

@ -0,0 +1,32 @@
#!/bin/cat
$Id: FAQ.Admin_Authentication.txt,v 1.2 2017/01/06 14:11:13 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
==============================================
Imapsync tips about admin authentication.
==============================================
It can be useful to authenticate without knowing each user password.
Using an admin account to authenticate is not a standard feature
supported by all imap servers. Sometimes it is implemented via --authuser1
sometimes not. It depends on the server software and its configuration.
You have to figure out what is the imap software server,
find out if it supports an admin account and how,
if an admin account is actually configured and
finally check an authentication via this admin account for
a standard user.
Known imap server software supporting admin authentication:
* Exchange 2003/2007/2010/2013. See the file FAQ.Exchange.txt
* Office365. See the file FAQ.Exchange.txt
* Gmail. See the file FAQ.XOAUTH2.txt
* Dovecot. See the file FAQ.Dovecot.txt
* UW-imap. See the file FAQ.General.txt
* Cyrus-imap. See the file FAQ.General.txt
* Sun Java Enterprise System/SunOne/iPlanet. See the file FAQ.General.txt
=======================================================================
=======================================================================

28
FAQ.d/FAQ.Archiving.txt
View file

@ -1,5 +1,5 @@
#!/bin/cat
$Id: FAQ.Archiving.txt,v 1.2 2016/05/09 13:03:14 gilles Exp gilles $
$Id: FAQ.Archiving.txt,v 1.6 2017/07/13 14:44:07 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
@ -7,6 +7,30 @@ This documentation is also at http://imapsync.lamiral.info/#doc
Imapsync tips about archiving messages.
============================================
=======================================================================
Q. Can I archive different accounts on the same destination account,
each account on a separate folder?
R. Yes. Use --subfolder2
--subfolder2 str : Move whole host1 folders hierarchy under the
host2 folder str.
It is done it by adding two --regextrans2 options before
all others. Add --debug to see what's really going on.
Example:
imapsync ... --user1 foo --subfolder2 foo
imapsync ... --user1 bar --subfolder2 bar
In case you need a strict sync, add --delete2 --delete2foldersonly "foo"
(or "bar"), il will delete on account2 what is not on account1 but only
in the right place, ie, the subfolder "foo".
imapsync ... --user1 foo --subfolder2 foo --delete2 --delete2foldersonly "foo"
=======================================================================
Q. How to move emails from one IMAP folder to another either on the
same IMAP server or a different one?
@ -21,3 +45,5 @@ Add option --delete if you want to move messages, instead of just copy/sync them
R2. See also file FAQ.Folders_Mapping.txt
=======================================================================
=======================================================================

54
FAQ.d/FAQ.Authentication_failure.txt Normal file
View file

@ -0,0 +1,54 @@
#!/bin/cat
$Id: FAQ.Authentication_failure.txt,v 1.5 2017/09/04 11:03:57 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=======================================================================
Imapsync authentication issues
=======================================================================
Host1 failure: Error login on [imap.example.com] with user [foo] auth [LOGIN]: 2 NO [AUTHENTICATIONFAILED] Authentication failed
One over four imapsync syncs ends up quickly with the error message
"Authentication failed" or "NO LOGIN failed" or similar.
Authentication failure is the primary failure with imapsync
and since nothing tangible can be done without authentication,
this stage must succeed to go further.
Here are some advices to get you pass this difficult stage of authentication:
* Triple check each credendial parameter, there are three parameters at each side:
* triple check --host1
* triple check --user1
* triple check --password1
* triple check --host2
* triple check --user2
* triple check --password2
* If you can authenticate successfully with an other imap client software
like Thunderbird or Outlook or Sparrow then it is a very good sign to
authenticate successfully with imapsync. Examine the parameters of
this other imap client and copy them as is for imapsync.
* Use option --showpasswords
At the beginning of the output, imapsync dumps all its command
line parameters; it's the line after "Command line used:".
With --showpasswords, imapsync prints the passwords received
instead of the string MASKED. It helps for debugging quoting issues.
Option --showpasswords shows passwords again when the IMAP dialog
is dumped by --debugimap option. Search for a line like
"Sending: 2 LOGIN test1 secret1" (secret1 is the password here)
* It is sometimes very hard to quote correctly unusual characters,
especially on Windows. See
https://imapsync.lamiral.info/FAQ.d/FAQ.Passwords_on_Windows.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.Passwords_on_Unix.txt
The quicker trick might be to change the password temporarilly
with easy characters like the classical alphabet, a long
string will still ensure strong security.

29
FAQ.d/FAQ.Contacts_Calendars.txt
View file

@ -1,5 +1,5 @@
#!/bin/cat
$Id: FAQ.Contacts_Calendars.txt,v 1.3 2016/05/09 13:37:21 gilles Exp gilles $
$Id: FAQ.Contacts_Calendars.txt,v 1.5 2017/01/06 14:11:13 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
@ -7,6 +7,11 @@ This documentation is also at http://imapsync.lamiral.info/#doc
Imapsync issues about syncing Contacts & Calendars.
=========================================================
Questions anwswered in this FAQ are:
Q. Can I copy or sync Calendar or Contacts with imapsync?
Q. How can I avoid copying Calendar or Contacts folders?
Q. How can I copy or synchronize Calendars or Contacts?
=======================================================================
Q. Can I copy or sync Calendar or Contacts with imapsync?
@ -16,15 +21,29 @@ R. No, Imapsync can't migrate Contacts and Calendars.
events via IMAP. In other words, messages synced by imapsync from
Calendars or Contacts folders are not used by email servers to set
or get the contacts or calendars.
No way via IMAP, no way via imapsync.
No way via IMAP, no way via imapsync.
So it's a good idea to avoid syncing contacts and calendars.
But see next question.
=======================================================================
Q. How can I avoid copying Calendar or Contacts folders?
R1. You can avoid synchronizing Calendar or Contacts folders with
the --exclude option. First you have to search what is their
exact name. The folders listing printed by imapsync at the
beginning of a run will surely help to find their names.
Assuming their names are "Calendars" and "Contacts" use then:
imapsync ... --exclude "Calendar" --exclude "Contacts"
=======================================================================
Q. How can I copy or synchronize Calendars or Contacts?
R1. It can't be done with imapsync. See previous question for explanations.
R1. Synchronizing Calendars or Contacts can't be done with imapsync.
See the first question for detailed explanations.
R2. It can be done, depending on the email server softwares used.
R2. Synchronizing Calendars or Contacts can be done, not
with imapsync, depending on the email server softwares used.
a) From Exchange to Exchange, export contacts and calendar to
PST format files on host1 and import them on host2.
@ -46,4 +65,6 @@ d) I plan to make tools for that but so far nothing has began
http://www.linux-france.org/prj/imapsync_list/msg01797.html
http://www.linux-france.org/prj/imapsync_list/msg01811.html
=======================================================================
=======================================================================

38
FAQ.d/FAQ.Dates.txt
View file

@ -1,5 +1,5 @@
#!/bin/cat
$Id: FAQ.Dates.txt,v 1.3 2016/07/27 13:14:30 gilles Exp gilles $
$Id: FAQ.Dates.txt,v 1.6 2017/08/29 05:34:55 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
@ -7,10 +7,19 @@ This documentation is also at http://imapsync.lamiral.info/#doc
Imapsync tips about dates.
===============================
Questions anwswered in this FAQ are:
Q. We have found that the time and date displayed have been changed to
the time at which the file was synchronized. What happened? Any fix?
Q. Is there a way to set any message with
"Date: (Invalid)" to a valid one like
"Date: Thu, 1 Jun 2017 23:59:59 +0000"?
=======================================================================
Q. We have found that the time and date displayed have been changed to
the time at which the file was synchronized.
the time at which the file was synchronized. What happened? Any fix?
R. This is the case by default with some email readers like:
- Outlook 2003
@ -70,7 +79,30 @@ by sent date, the "Date:" header line.
b) Use a imap server that respects the imap RFC and accepts
the internal date set by imapsync.
c) Try to understand why the reader shows another date than the "Date:" line.
c) Try to understand why the email client software shows another date
than the "Date:" header line.
=======================================================================
Q. Is there a way to set any message with
"Date: (Invalid)" to a valid one like
"Date: Thu, 1 Jun 2017 23:59:59 +0000"?
R. Yes, there is a way with option --regexmess
First, let's select only messages with a buggy Date header:
--search "HEADER Date Invalid"
Second, let's change this line by a valid one,
on windows:
--regexmess "s{\A(.*?(?! ^$))^Date:\ \(Invalid\)(.*?)$}{$1Date: Thu, 1 Jun 2017 23:59:59 +0000}xms"
on Unix (replaced enclosing double-quotes " by single quotes ' ):
--regexmess 's{\A(.*?(?! ^$))^Date:\ \(Invalid\)(.*?)$}{$1Date: Thu, 1 Jun 2017 23:59:59 +0000}xms'
=======================================================================
=======================================================================

28
FAQ.d/FAQ.Domino.txt
View file

@ -1,5 +1,5 @@
#!/bin/cat
# $Id: FAQ.Domino.txt,v 1.6 2016/01/28 14:34:15 gilles Exp gilles $
# $Id: FAQ.Domino.txt,v 1.8 2017/03/02 12:54:15 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
@ -8,44 +8,44 @@ This documentation is also at http://imapsync.lamiral.info/#doc
=============================
======================================================================
Q. From Domino Notes to xxx
=======================================================================
Q. From Domino Notes to XXX
On Windows use:
imapsync.exe ... --sep1 "\\" --prefix1 ""
imapsync.exe ... --sep1 "\\" --prefix1 "" --messageidnodomain
On Unix use:
imapsync ... --sep1 '\' --prefix1 ''
imapsync ... --sep1 '\' --prefix1 '' --messageidnodomain
======================================================================
Q. From xxx to Domino Notes
=======================================================================
Q. From XXX to Domino Notes
Domino doesn't accept INBOX subfolders.
On Windows:
imapsync.exe ... ^
--sep2 "\\" --prefix2 "" ^
--sep2 "\\" --prefix2 "" --messageidnodomain ^
--regextrans2 "s,^Inbox\\(.*),$1,i"
On Unix:
imapsync ... \
--sep2 '\' --prefix2 '' \
--sep2 '\' --prefix2 '' --messageidnodomain \
--regextrans2 's,^Inbox\\(.*),$1,i'
If you want to sync the complete host1 mailbox in a subfolder called OLDBOX use:
On Windows:
imapsync.exe ... ^
--sep2 "\\" --prefix2 "" ^
--sep2 "\\" --prefix2 "" --messageidnodomain ^
--subfolder2 "OLDBOX" --justfolders --dry
On Unix:
imapsync ... \
--sep2 '\' --prefix2 '' \
--sep2 '\' --prefix2 '' --messageidnodomain \
--subfolder2 'OLDBOX' --justfolders --dry
If the output is correct for you then remove --dry and have a run.
@ -69,8 +69,10 @@ For Domino 853FP6 on Linux, we used this command on Unix:
--host2 hhh2 --user2 uuu2 --password2 ppp2 \
--exclude "^INBOX/Trash" --exclude 'Junk|Drafts' \
--regextrans2 's#^INBOX/Sent$#^Sent#' \
--prefix2 "" --sep2 "\/" \
--prefix2 "" --sep2 "\/" --messageidnodomain \
--regexmess 's{\A(.*?(?! ^$))^Date:(.*?)$}{$1Migratedbyus:$2\nx-MailDate:$2}gxms'
======================================================================
=======================================================================
=======================================================================

2
FAQ.d/FAQ.Dovecot.txt
View file

@ -1,5 +1,5 @@
#!/bin/cat
$Id: FAQ.Dovecot.txt,v 1.4 2016/01/28 14:34:15 gilles Exp gilles $
$Id: FAQ.Dovecot.txt,v 1.5 2017/01/06 14:21:06 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc

30
FAQ.d/FAQ.Duplicates.txt
View file

@ -1,11 +1,11 @@
#!/bin/cat
$Id: FAQ.Duplicates.txt,v 1.10 2016/04/17 19:06:39 gilles Exp gilles $
$Id: FAQ.Duplicates.txt,v 1.12 2017/08/31 03:06:45 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
======================================================================
=======================================================================
Imapsync tips about duplicated messages issues.
======================================================================
=======================================================================
=======================================================================
Q. How can I remove duplicates in an lone account?
@ -13,6 +13,27 @@ Q. How can I remove duplicates in an lone account?
R. Just run imapsync on the same account with option --delete2duplicates,
ie, with host1 == host2 and user1 == user2
=======================================================================
Q. How can I know if imapsync will generate duplicates on a second run?
R. To see if imapsync will generate duplicates on a second run, start
a second run with --dry option added. imapsync will then show if it
would mistakenly copy messages again, but without really copying them
imapsync ... --dry
The final stats should also show a positive value for the line
"Messages skipped:" since most of the skipped messages are skipped
because they are already on host2. Example of final stats:
++++ Statistics
Transfer started on : Thu Aug 31 04:28:32 2017
Transfer ended on : Thu Aug 31 04:28:44 2017
Transfer time : 11.7 sec
Folders synced : 1/1 synced
Messages transferred : 0
Messages skipped : 1555
=======================================================================
Q: Multiple copies, duplicates, when I run imapsync twice ore more.
@ -140,5 +161,4 @@ R2. With option --useuid imapsync doesn't use headers to identify
In that case duplicates on host1 are also transferred on host2.
=======================================================================
=======================================================================

7
FAQ.d/FAQ.Emptying.txt
View file

@ -1,11 +1,11 @@
#!/bin/cat
$Id: FAQ.Emptying.txt,v 1.4 2016/04/24 00:07:47 gilles Exp gilles $
$Id: FAQ.Emptying.txt,v 1.5 2017/01/06 14:11:13 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
============================================
=======================================================================
Imapsync tip to empty an account.
============================================
=======================================================================
=======================================================================
Q. How to delete all emails of all folders of an account with imapsync?
@ -33,3 +33,4 @@ Example:
./imapsync ... --delete2folders --foldersizes
=======================================================================
=======================================================================

84
FAQ.d/FAQ.Exchange.txt
View file

@ -1,5 +1,5 @@
#!/bin/cat
$Id: FAQ.Exchange.txt,v 1.23 2016/08/10 01:29:37 gilles Exp gilles $
$Id: FAQ.Exchange.txt,v 1.34 2017/07/18 21:27:18 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
@ -11,13 +11,16 @@ Questions anwswered in this FAQ are:
Q. Can I use imapsync to transfer from or to Exchange or Office365 accounts?
Q. How to sync from XXX to Exchange 2010/2013
Q. How to sync from XXX to Exchange 2010/2013/2016
Q. How to sync from XXX to Office365
Q. For Office365 I have double and triple checked the username and
password spelling but I still get a "LOGIN failed". Any clue?
Q. I see "NO Maximum size of appendable message has been exceeded"
What can I do with that?
Q. Exchange fails with "User is authenticated but not connected".
Q. Exchange fails with "BAD Command received in Invalid state".
@ -25,6 +28,9 @@ Q. Exchange fails with "BAD Command received in Invalid state".
Q. From XXX to Exchange 2013 or Office365, read receipts are all
resent again after a sync. Even for old messages. How can I fix that?
Q. DEBUG: IO/Socket/SSL.pm:1043: local error: SSL read error
DEBUG: IO/Socket/SSL.pm:1043: local error: SSL read error
Q. From XXX to Exchange 2010/2013 or Office365 I get this error message
sometimes: "BAD Command Argument Error 11". What does it mean?
@ -60,12 +66,14 @@ R. Yes. But IMAP access to a Exchange or Office365 account is not always
part:
=======================================================================
Q. How to sync from XXX to Exchange 2010/2013
Q. How to sync from XXX to Exchange 2010/2013/2016
R. Here is a command line resume that solves most encountered issues when
migrating to Exchange. To understand or change the details you have
to read next Q/R sections.
R0. IMAP is not enable by default on Exchange, see how to enable it:
https://technet.microsoft.com/en-us/library/bb124489(v=exchg.150).aspx
R1. Following is a command line resume that solves most encountered
issues when migrating to Exchange. To fully understand or change
the details you have to read the next Q/R sections.
On Windows:
@ -155,13 +163,39 @@ R2. Miguel Alameda reported understanding and solving this issue
like this, the context was admin/authuser:
"The admin user had not permission in the target mailbox."
=======================================================================
Q. I see "NO Maximum size of appendable message has been exceeded"
What can I do with that?
R. Office 365 supports send/receive max message sizes of up to 150MB
but you need to make changes in your tenant(s) to support it.
The following PowerShell command will increase the message sizes that
can be sent/received. The trick in getting IMAPSync to work is to
apply these settings to the accounts performing the migration,
NOT the accounts associated with the target mailbox (assuming you're
using service accounts to perform transfers on behalf of users).
Set-mailbox -Identity $UPN -MaxReceiveSize 150mb -MaxSendSize 150mb
e.g.
Set-mailbox -Identity "migrationaccount@testtenant.onmicrosoft.com" -MaxReceiveSize 150mb -MaxSendSize 150mb
We're transferring data between Office 365 tenants so we set these
values on the migration acounts in the source and target tenants.
Thanks to Sean McDougall, Ian Thomas & Matt Wilks from Toronto
for this FAQ item.
=======================================================================
Q. Exchange fails with "BAD Command received in Invalid state".
R. This message might happens when authentication without ssl nor tls.
R. This message might happens when authenticating without ssl nor tls.
Add --tls1 or --ssl1 if this error message comes from host1.
Add --tls2 or --ssl2 if this error message comes from host1.
Add --tls2 or --ssl2 if this error message comes from host2.
(Never add --tlsX and --sslX on the same side X (1 or 2), it won't work)
=======================================================================
Q. From XXX to Exchange 2013 or Office365, read receipts are all
resent again after a sync. Even for old messages. How can I fix that?
@ -196,6 +230,19 @@ X-Disposition-Notification-To: blabla
Thanks to David Karnowski for pointing and solving this issue.
=======================================================================
Q. DEBUG: IO/Socket/SSL.pm:1043: local error: SSL read error
DEBUG: IO/Socket/SSL.pm:1043: local error: SSL read error
R1. "SSL read or write error" happens sometimes, it isn't related to
imapsync directly but to the ssl underlying library when communicating
with Exchange in TLS/SSL encrypted mode.
Next runs should put the sync further, so rerun the syncs
until it is well completed.
R2. Another solution is to remove --tls or --ssl options for Exchange
and accept a clear text sync.
=======================================================================
Q. From XXX to Exchange 2010/2013 or Office365 I get this error message
sometimes: "BAD Command Argument Error 11". What does it mean?
@ -273,11 +320,19 @@ Q. Exchange and Office365 have throttle mechanisms to limit any huge
usage. Sometimes imapsync transfers are too stressful for servers.
How to deal with that?
R. It looks like limiting 4 messages per second is enough to never
reach any throttle limit.
R. It looks like limiting 4 messages per second is enough to avoid
the throttle limits.
imapsync ... --maxmessagespersecond 4
In case throttle appears anyway, fix them with:
https://technet.microsoft.com/en-us/library/jj863577(v=exchg.150).aspx
See also:
http://www.linux-france.org/prj/imapsync_list/msg02072.html
Sometimes restarting the Exchange server is needed to take
into account the change in the configuration.
======================================================================
Q. How to migrate from or to Exchange 2007/2010/2013 with an
admin/authuser account?
@ -347,6 +402,12 @@ So don't use --authmech1 SOMETHING with --authuser1 admin_user,
it will not work.
Same behavior with the --authuser2 option.
Note from Martin Paulucci:
I had to remove the domain part for the user
but not for the admin. Example:
imapsync ... --authuser2 user_admin@domain.com --user2 user_to_be_migrated
See also:
http://www.linux-france.org/prj/imapsync_list/msg02203.html
@ -458,3 +519,4 @@ R. imapsync ... --prefix1 "INBOX."
=======================================================================
=======================================================================

184
FAQ.d/FAQ.Flags.txt
View file

@ -1,11 +1,11 @@
#!/bin/cat
$Id: FAQ.Flags.txt,v 1.7 2016/01/28 14:34:15 gilles Exp gilles $
$Id: FAQ.Flags.txt,v 1.17 2017/07/27 15:39:57 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
===============================
=======================================================================
Imapsync tips about flags.
===============================
=======================================================================
Questions answered here are:
@ -15,7 +15,11 @@ Q. Is there a way to only sync messages with a specific flag set,
for example, the \Seen flag?
Q. How to convert flags?
Q. Exchange sends an email to any sender whose email is deleted
without reading. It's called "unread notifications".
How to set the \Seen flag on host1 (source system) before syncing?
Q. Does imapsync retain the \Answered and $Forwarded flags?
Q. How to fix this error: BAD Invalid system flag \FORWARDED
@ -25,9 +29,16 @@ Q. How to convert flags with $ to \ character?
Q. imapsync fails with the following error:
flags from : [\Seen NonJunk]
Error trying to append string: 58 NO APPEND Invalid flag list
Q. Flags are not well synchronized. Is it a bug?
Q. Flags are resynced at each run for already synced/copied messages,
how can I avoid this feature?
Q. Is it possible to sync labels and stars made by Thunderbird to
Exchange categories? Or a way in Outlook to show labels created by
Thunderbird?
=======================================================================
Q. How to debug flag issues?
@ -50,35 +61,15 @@ or
or ...
The complete list of search things are listed below
The complete list of search things related to flags are listed below
http://www.faqs.org/rfcs/rfc3501.html
6.4.4. SEARCH Command
...
ALL
All messages in the mailbox; the default initial key for
ANDing.
ANSWERED
Messages with the \Answered flag set.
BCC <string>
Messages that contain the specified string in the envelope
structure's BCC field.
BEFORE <date>
Messages whose internal date (disregarding time and timezone)
is earlier than the specified date.
BODY <string>
Messages that contain the specified string in the body of the
message.
CC <string>
Messages that contain the specified string in the envelope
structure's CC field.
DELETED
Messages with the \Deleted flag set.
@ -88,25 +79,9 @@ http://www.faqs.org/rfcs/rfc3501.html
FLAGGED
Messages with the \Flagged flag set.
FROM <string>
Messages that contain the specified string in the envelope
structure's FROM field.
HEADER <field-name> <string>
Messages that have a header with the specified field-name (as
defined in [RFC-2822]) and that contains the specified string
in the text of the header (what comes after the colon). If the
string to search is zero-length, this matches all messages that
have a header line with the specified field-name regardless of
the contents.
KEYWORD <flag>
Messages with the specified keyword flag set.
LARGER <n>
Messages with an [RFC-2822] size larger than the specified
number of octets.
NEW
Messages that have the \Recent flag set but not the \Seen flag.
This is functionally equivalent to "(RECENT UNSEEN)".
@ -119,10 +94,6 @@ http://www.faqs.org/rfcs/rfc3501.html
functionally equivalent to "NOT RECENT" (as opposed to "NOT
NEW").
ON <date>
Messages whose internal date (disregarding time and timezone)
is within the specified date.
OR <search-key1> <search-key2>
Messages that match either search key.
@ -132,42 +103,6 @@ http://www.faqs.org/rfcs/rfc3501.html
SEEN
Messages that have the \Seen flag set.
SENTBEFORE <date>
Messages whose [RFC-2822] Date: header (disregarding time and
timezone) is earlier than the specified date.
SENTON <date>
Messages whose [RFC-2822] Date: header (disregarding time and
timezone) is within the specified date.
SENTSINCE <date>
Messages whose [RFC-2822] Date: header (disregarding time and
timezone) is within or later than the specified date.
SINCE <date>
Messages whose internal date (disregarding time and timezone)
is within or later than the specified date.
SMALLER <n>
Messages with an [RFC-2822] size smaller than the specified
number of octets.
SUBJECT <string>
Messages that contain the specified string in the envelope
structure's SUBJECT field.
TEXT <string>
Messages that contain the specified string in the header or
body of the message.
TO <string>
Messages that contain the specified string in the envelope
structure's TO field.
UID <sequence set>
Messages with unique identifiers corresponding to the specified
unique identifier set. Sequence set ranges are permitted.
UNANSWERED
Messages that do not have the \Answered flag set.
@ -197,6 +132,44 @@ For example to convert flag IMPORTANT to flag CANWAIT
option --debugflags is usefull to see in details what imapsync
does with flags.
=======================================================================
Q. Exchange sends an email to any sender whose email is deleted
without reading. It's called "unread notifications".
How to set the \Seen flag on host1 (source system) before syncing?
R. You can add \Seen (if missing) to the host1 account by applying a
first sync to the same account, same at source and destination,
and the help of option --regexflag. It can also be done on the fly
from account1 to account2 in case account1 has to stay as it is.
Add the \Seen flag to all messages like this:
On Winwows:
imapsync.exe ... --regexflag "s,^((?!\\Seen)).*$,$1 \\Seen,"
On Unix:
imapsync ... --regexflag 's,^((?!\\Seen)).*$,$1 \\Seen,'
R2. You can also filter with --search1 UNSEEN and use a simpler
regex:
On Winwows:
imapsync.exe ... --search1 UNSEEN --regexflag "s/(.*)/$1 \\Seen/"
On Unix:
imapsync ... --search1 UNSEEN --regexflag 's/(.*)/$1 \\Seen/'
R3. Fix it on the server Exchange:
Google translate:
https://translate.google.com/translate?sl=auto&tl=en&u=https%3A%2F%2Fwww.ci-solution.com%2Fblog%2Fartikel%2Fungelesen-geloescht-verhindern.html
German original:
https://www.ci-solution.com/blog/artikel/ungelesen-geloescht-verhindern.html
(Link from Oliver B.)
=======================================================================
Q. Does imapsync retain the \Answered and $Forwarded flags?
@ -224,7 +197,7 @@ R. Filter flag \FORWARDED with --regexflag like this:
On Windows:
imapsync ... --regexflag "s/\\FORWARDED//gi"
imapsync.exe ... --regexflag "s/\\FORWARDED//gi"
On Unix:
@ -234,6 +207,19 @@ or
imapsync ... --regexflag "s/\\\\FORWARDED//gi"
Other related flags to remove are \Indexed and \ATTACHED
Windows: imapsync.exe ... --regexflag "s/\\Indexed//gi"
Unix: imapsync ... --regexflag 's/\\Indexed//gi'
Windows: imapsync.exe ... --regexflag "s/\\ATTACHED//gi"
Unix: imapsync ... --regexflag 's/\\ATTACHED//gi'
All usually problematic flags in one line:
Windows: imapsync.exe ... --regexflag "s/\\FORWARDED|\\Indexed|\\ATTACHED//gi"
Unix: imapsync ... --regexflag 's/\\FORWARDED|\\Indexed|\\ATTACHED//gi'
=======================================================================
Q. How to convert flags with $ to \ character?
@ -280,5 +266,37 @@ Two solutions:
option --syncflagsaftercopy does it again using the imap STORE
command.
=======================================================================
Q. Flags are resynced at each run for already synced/copied messages,
how can I avoid this feature?
R. It is not possible to avoid this feature by an option for now.
If you really need this then ask for it and I might code it.
=======================================================================
Q. Is it possible to sync labels and stars made by Thunderbird to
Exchange categories? Or a way in Outlook to show labels created by
Thunderbird?
R. Imapsync syncs all flags possible by default, so if it doesn't do
that it might means there are not on the server but stay only on the
client or that the host2 server claims to accept only a given set
of flags.
Check those claims by accessing the same mailbox on the same server
from another thunderbird on another host, you should not retrieve
those labels. If you do find them it then might mean that host2
server don't want them, try --nofilterflags
imapsync ... --nofilterflags
You can try --nofilterflags straightaway without the
"other thunderbird" proposal.
There is also the possibility to map flags across servers
with --regexflags
=======================================================================
=======================================================================

33
FAQ.d/FAQ.Folders_Mapping.txt
View file

@ -1,11 +1,11 @@
#!/bin/cat
$Id: FAQ.Folders_Mapping.txt,v 1.10 2016/01/28 14:34:15 gilles Exp gilles $
$Id: FAQ.Folders_Mapping.txt,v 1.13 2017/05/24 19:18:41 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
===============================================
=======================================================================
Imapsync tips about changing folders names.
===============================================
=======================================================================
Folders names are by default reproduced identical except for
the prefix and the separator which are automatically adapted
@ -46,22 +46,22 @@ Here
*) Good method to elaborate any --regextrans2 string
First elaborate the --regextrans2 string with --dry --justfolders
--nofoldersizes options.
With --dry imapsync shows the transformations it will do without
really doing them.
With --justfolders imapsync will work only with folders,
messages won't be taken into account.
With --nofoldersizes imapsync won't spend time useless time on
evaluating folders sizes.
First, elaborate the --regextrans2 string with --dry --justfolders options.
When the output shows what you escape imapsync to do with folders
imapsync ... --dry --justfolders
With --dry imapsync shows the transformations it will do without
really doing them, --dry is the "do nothing" mode.
With --justfolders imapsync will work only with folders,
messages won't be taken into account, so it will be fast.
When the output shows what you expect imapsync to do with folders
names, you can remove the --dry option. Keep the --justfolders
option in order to see if the destination server host2 accepts
to create the folders.
When everything is ok with folders you can remove --justfolders
and --nofoldersizes imapsync will also transfer messages.
When everything is ok with folders you might remove --justfolders,
imapsync will also transfer messages.
Showing folders sizes is good then transferring messages, it allows
ETA calculation and it's a supplementary check on folders.
@ -183,7 +183,7 @@ to folder INBOX only on host2:
1) First try (safe mode):
--regextrans2 "s/(.*)/INBOX/" --dry --justfolders
--regextrans2 "s/.*/INBOX/" --dry --justfolders
2) See if the output says everything you want imapsync to do,
--dry option is safe and does nothing real.
@ -247,5 +247,4 @@ imapsync ... \
--folder INBOX
=======================================================================
=======================================================================

37
FAQ.d/FAQ.Folders_Selection.txt
View file

@ -1,11 +1,11 @@
#!/bin/cat
$Id: FAQ.Folders_Selection.txt,v 1.4 2016/06/07 22:19:04 gilles Exp gilles $
$Id: FAQ.Folders_Selection.txt,v 1.6 2017/07/20 11:02:11 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=====================================
=======================================================================
Imapsync tips to select folders.
=====================================
=======================================================================
By default, Imapsync syncs all folders, one by one, in alphanumeric order.
@ -19,7 +19,7 @@ square brackets, the right column is the human utf8 view.
======================================================================
=======================================================================
Q. How can I sync only one folder?
R. Use --folder option.
@ -31,25 +31,24 @@ If you have more specific folders to sync just add several --folder
imapsync ... --folder MyFolder --folder ThisFolder --folder ThatFolder
======================================================================
=======================================================================
Q. What are --subscribe and --subscribed for, and how can they be used?
R. In the IMAP protocol each user can subscribe to one or more folders.
Then one can configure his email software to just see his subscribed
folders. That's an IMAP feature.
Knowing that, the imapsync help mentions:
Imapsync can use this imap feature to select subscribed folders
and also subscribe to folders on host2. Here are the options:
imapsync --help
...
--subscribed : transfers subscribed folders.
--subscribe : subscribe to the folders transferred on the
host2 that are subscribed on host1.
--subscribe_all : subscribe to the folders transferred on the
host2 even if they are not subscribed on host1.
--subscribed : Transfers subscribed folders.
--subscribe : Subscribe to the folders transferred on the
host2 that are subscribed on host1. On by default.
--subscribeall : Subscribe to the folders transferred on the
host2 even if they are not subscribed on host1.
======================================================================
=======================================================================
Q. I want to exclude a folder hierarchy like "public"
R. Use:
@ -65,7 +64,7 @@ output line :
From folders list : [INBOX] [public.dreams] [etc.]
======================================================================
=======================================================================
Q. I want to exclude only INBOX
R. Use:
@ -76,7 +75,7 @@ A good way to see what will be done is to first use:
imapsync ... --exclude "^INBOX$" --justfolders --nofoldersizes --dry
======================================================================
=======================================================================
Q. I want to exclude folders matching SPAM no matter the case,
aka how to be case insensitive
@ -89,7 +88,7 @@ A good way to see what will be done is to first use:
imapsync ... --exclude "(?i)spam" --justfolders --nofoldersizes --dry
======================================================================
=======================================================================
Q. I want the --folder "MyFolder" option be recursive.
Two solutions:
@ -102,5 +101,5 @@ R2. Use --include "^MyFolder"
Then the folder "MyFolder" and all its subfolders will be handled
and only them.
=======================================================================
=======================================================================

797
FAQ.d/FAQ.General.txt Normal file
View file

@ -0,0 +1,797 @@
#!/bin/cat
# $Id: FAQ.General.txt,v 1.233 2017/09/07 22:08:42 gilles Exp gilles $
=======================================================================
General FAQ for imapsync
=======================================================================
This document is also available at
https://imapsync.lamiral.info/#doc
https://imapsync.lamiral.info/FAQ.d/FAQ.General.txt
Questions anwswered in this FAQ are:
Q. Do I need to create IMAP mailboxes at the destination platform?
Q. Am I forced to publish the IMAP service on internet since the two
environment are not in the same location or same LAN?
Q. What are the most important differences between the Unix shell syntax
and the Windows batch syntax.
Q. How to install imapsync?
Q. How to use imapsync?
Q. Can you give some configuration examples?
Q. How can I have commercial support?
Q. How can I have gratis support?
Q. Where I can find old imapsync releases?
Q. Where I can find free open and gratis imapsync releases?
Q. Is is legal to find imapsync gratis (or not) elsewhere?
Q. How "Facts and figures" are known
https://imapsync.lamiral.info/#NUMBERS
Q. I use --useuid which uses a cache in /tmp or --tmpdir, the hostnames
host1 or host2 has changed but mailboxes are the same. Will imapsync
generate duplicate messages on next runs?
Q. How can I speed up transfers?
Q. I see warning messages like the following:
"Host1 Sent/15 size 1428 ignored (no header so we ignore this message.
To solve this: use --addheader)".
What can I do to transfer those messages?
Q. How can I try imapsync with latest Mail::IMAPClient 3.xx perl module?
Q. How can I use imapsync with Mail::IMAPClient 2.2.9 perl module?
Q. How to verify imapsync.exe I got is the right file bit per bit?
Q. Can I use imapsync to migrate emails from pop3 server to imap server?
Q. Folders are not created on host2. What happens?
Q. I am interested in creating a local clone of the IMAP on a LAN
server for faster synchronizations, email will always be delivered
to the remote server and so the synchronization will be one way - from
remote to local. How suited is imapsync for continuous one-way
synchronization of mailboxes? Is there a better solution?
Q. I need to log every output on a file named log.txt
Q. Quantifier in {,} bigger than 32766 in regex; marked by <-- HERE in
m/(.{ <-- HERE 1,49947})(?:,|$)/ at Mail/IMAPClient.pm line 2121.
Q. Couldn't create [INBOX.Ops/foo/bar]: NO Invalid mailbox name:
INBOX.Ops/foo/bar
Q. Is it possible to sync also the UIDL of the POP3 server?
Q. Is it possible to sync also the UIDs of the IMAP server?
Q. The option --subscribe does not seem to work
Q. With huge account (many messages) when it comes to reading the
destination server it comes out this error:
"To Folder [INBOX.foobar] Not connected"
What can I do?
Q. Can Imapsync filter Spam during the sync?
Q. How to migrate from uw-imap with an admin/authuser account?
Q. How to migrate from cyrus with an admin account?
Q: How to migrate from Sun Java Enterprise System / Sun One / iPlanet /
Netscape servers with an admin account?
Q. Is there a way to delete the destination folder when the source
folder is no longer there?
Q. I would love to have a function to inject lines in the header.
Things like "X-migrated-from-foo: 20100617"
Q. I want to play with headers line and --regexmess but I want to leave
the body as is.
Q. My imap server does not accept a message and warns
"Invalid header". What is the problem?
Q. The contact folder isn't well copied.
How to copy the contact folder?
Q: How can I write an .rpm with imapsync
Q. Where I can read up on the various IMAP RFCs?
=======================================================================
Q. Do I need to create IMAP mailboxes at the destination platform?
R. Yes!
Imapsync does only IMAP and there is no way to create an account
with the standard IMAP protocol. So you have to create them first.
=======================================================================
Q. Am I forced to publish the IMAP service on internet since the two
environment are not in the same location or same LAN?
R. The host where you run imapsync has to contact both servers via imap.
You are not obliged to publish the imap service on internet if the host
where you run imapsync can contact both imap servers via imap.
Use their names or their IP addresses.
=======================================================================
Q. What are the most important differences between the Unix shell syntax
and the Windows batch syntax.
R. There are several differences between Unix and Windows
in the command line syntax.
- Character \ on Unix versus ^ on Windows
- Character ' on Unix versus " on Windows
Details:
A) \ versus ^
On Unix shells you can write a single command on multiple lines
by using the escape character \ at the end of each line
(except the last one). On Windows this character is ^
Unix example:
./imapsync \
--host1 imap.truc.org --user1 foo --password1 secret1 \
--host2 imap.trac.org --user2 bar --password2 secret2
Windows example:
imapsync ^
--host1 imap.truc.org --user1 foo --password1 secret1 ^
--host2 imap.trac.org --user2 bar --password2 secret2
Of course you can write the command on a single line without
characters \ nor ^. I use them because the output is
better, no truncation, pretty print. It's just sugar!
In this FAQ I use \ for examples. Transcript to ^ if
you're on a Windows system.
B) Quote vs Double-quote, ie ' versus "
On Windows the single quote character ' doesn't work
like on Unix so in the examples of this FAQ the
command containing single quotes ' will fail on Windows.
To fix this, just replace single quotes ' by double quotes "
Also on Windows, in examples with \$1, replace
any \$1 by $1 (remove the \ before $).
=======================================================================
Q. How to install imapsync?
R. Read the INSTALL files in the tarball. also available at
https://imapsync.lamiral.info/#doc
https://imapsync.lamiral.info/INSTALL.d/
=======================================================================
Q. How to use imapsync?
R. Read the TUTORIAL_Unix.txt file, maybe the README and, if you
encounter problems, the FAQ.d/* files in the tarball.
All are also available and updated at:
https://imapsync.lamiral.info/#doc
=======================================================================
Q. Can you give some configuration examples?
R1. Basic usage is described there:
https://imapsync.lamiral.info/#DOC_BASIC
It is:
imapsync --host1 test1.lamiral.info --user1 test1 --password1 secret1 \
--host2 test2.lamiral.info --user2 test2 --password2 secret2
R2. Some FAQ files contains many examples for several scenarios
https://imapsync.lamiral.info/#doc
(Gmail, Office365, Exchange, Darwin, etc.)
=======================================================================
Q. How can I have commercial support?
R. Buy support from imapsync author: Gilles LAMIRAL
https://imapsync.lamiral.info/#buy_all
=======================================================================
Q. How can I have gratis support?
R. Write to the imapsync author: Gilles LAMIRAL
https://imapsync.lamiral.info/#AUTHOR
I help all users as long as I have time to help them all,
users who bought the support get my help first.
=======================================================================
Q. Where I can find old imapsync releases?
R. https://imapsync.lamiral.info/dist/old_releases/
=======================================================================
Q. Where I can find free open and gratis imapsync releases?
R. https://imapsync.lamiral.info/dist/
Github has a copy at
https://github.com/imapsync/imapsync
Q. Is is legal to find imapsync gratis (or not) elsewhere?
R. Yes, the license permits it
https://imapsync.lamiral.info/NOLIMIT
=======================================================================
Q. How "Facts and figures" are known
https://imapsync.lamiral.info/#NUMBERS
R. To know wether a newer imapsync exists or not, imapsync does a http
GET to http://imapsync.lamiral.info/VERSION
Via the User-agent parameter it also send:
* imapsync release
* Perl version
* Mail::IMAPClient version
* Operating System
You can remove this behavior by adding option --noreleasecheck on the
command line (or by setting $releasecheck = 0 in the source code)
or by using the github release.
=======================================================================
Q. I use --useuid which uses a cache in /tmp or --tmpdir, the hostnames
host1 or host2 has changed but mailboxes are the same. Will imapsync
generate duplicate messages on next runs?
R. Yes
Q. How can I fix this?
R. The cache path reflects exactly hostnames or ip addresses given via
--host1 and --host2 values. So just change the directory names
of host1 or host2. Use --dry to see if next runs will generate
duplicates.
By default on Unix the cache is like
/tmp/imapsync_cache/host1/user1/host2/user2/...
=======================================================================
Q. How can I speed up transfers?
R. By using --useuid imapsync avoid getting messages headers and build
a cache. On Unix a good thing is to add also --tmpdir /var/tmp
to keep the cache since /tmp is often cleared on reboot.
imapsync ... --useuid
On Unix:
imapsync ... --useuid --tmpdir /var/tmp/
R. Add also --nofoldersizes since the default behavior is to compute
folder sizes. Folder sizes are useless for the transfer, just
useful to see what has to be done on each folder and guess when
the transfer will end (ETA).
R. Add also --noexpungeaftereach if you use --delete.
But be warn that an interrupted transfer can loose messages
on host2 in a second run if you use a (silly) combination like
imapsync ... --delete --noexpunge --noexpungeaftereach --expunge2
R. Add also --nocheckmessageexists
--nocheckmessageexists is on by default since release 1.520.
Since transfer can be long on a huge mailbox imapsync checks
a message exist before copying it, but it takes time and
cpu on the host1 server.
Notes about --useuid
Case where building the cache first is necessary (to avoid multiples transfers)
If you run again imapsync with --useuid on a transfer already done without
--useuid then, to avoid messages be copied again, first run imapsync
with --usecache but without --useuid, example scenario:
A] Running with the default options, I began without --useuid
1) First run with default options
imapsync ...
Too slow, I want to speed up!
2) Build the cache
imapsync ... --usecache
3) Speed up now
imapsync ... --useuid
B] I began with --useuid from the first time
1) First run and next runs with --useuid
imapsync ... --useuid
Inodes number issue.
The cache is simple, it uses the file-system natively,
it's just an empty file per message transfered.
When mailboxes are huge the cache can exhaust the number of inodes
allowed in the filesystem, that's a limitation like limitation
size but it's less often encountered.
On Unix, to predict whether your tmpdir filesystem used by imapsync
will support the whole cache, just run the command "df -i /var/tmp",
if /var/tmp is the --tmpdir argument.
On windows, search and drop me a note about how to count the number
of files allowed in the filesystem.
It seems FAT32 supports 268 435 445 clusters.
Choosing the number of inodes allowed by a filesystem can be done
at the creation of it with "mkfs -N number-of-inodes ..."
imapsync can predict how many messages have to be synced with the
option --justfoldersizes (no transfer will be done)
imapsync ... --justfoldersizes
=======================================================================
Q. I see warning messages like the following:
"Host1 Sent/15 size 1428 ignored (no header so we ignore this message.
To solve this: use --addheader)".
What can I do to transfer those messages?
R1. Like suggested inline, use --addheader option.
Option --addheader will add an header line like
Message-Id: <15@imapsync>
where 15 is the message UID number on host1.
Then imapsync will transfer the changed message on host2.
Duplicates won't happen on next runs.
imapsync ... --addheader
R2. Other solution.
Use --useuid then imapsync will avoid dealing with headers.
imapsync ... --useuid
=======================================================================
Q. How can I try imapsync with latest Mail::IMAPClient 3.xx perl module?
Three solutions at least.
R1 - Look at the script named "i3" in the tarball, it can be used to
run imapsync with the included Mail-IMAPClient-3.39/ wherever you
unpacked the imapsync tarball
R2 Run:
cpanm Mail::IMAPClient # this uses cpanminus
or
cpan -i Mail::IMAPClient
or
perl -MCPAN -e "install Mail::IMAPClient"
R3 If you want to install the Perl module locally in a directory
- Download latest Mail::IMAPClient 3.xx at
http://search.cpan.org/dist/Mail-IMAPClient/
- untar it anywhere:
tar xzvf Mail-IMAPClient-3.xx.tar.gz
- Get any imapsync (latest is better).
- run imapsync with perl and -I option tailing to use the perl
module Mail-IMAPClient-3.xx. Example:
perl -I./Mail-IMAPClient-3.39/lib ./imapsync ...
or if imapsync is in directory /path/
perl -I./Mail-IMAPClient-3.39/lib /path/imapsync ...
=======================================================================
Q. How can I use imapsync with Mail::IMAPClient 2.2.9 perl module?
R. Mail::IMAPClient 2.2.9 is no longer supported.
=======================================================================
Q. How to verify imapsync.exe I got is the right file bit per bit?
R. Use md5sum to check integrity of the file.
Get md5sum.exe at http://etree.org/md5com.html
md5sum imapsync.exe
Then compare the checksum with the one given by the author.
=======================================================================
Q. Can I use imapsync to migrate emails from pop3 server to imap server?
R1. No, but you can migrate emails from a pop3 server to an imap server
with the command line tool pop2imap:
http://www.linux-france.org/prj/pop2imap/
R2. Yes, sometimes, because many pop3 servers runs in parallel
with an imap server on exactly the same mailboxes. They serve
the same INBOX (imap serves INBOX and several other folders,
pop3 serves only INBOX).
So have a try with imapsync on the same host1.
=======================================================================
Q. Folders are not created on host2. What happens?
R. Do you use IMAP or POP3 with your client software?
It looks like you use POP3 instead of IMAP, POP3 sees only INBOX.
=======================================================================
Q. I am interested in creating a local clone of the IMAP on a LAN
server for faster synchronizations, email will always be delivered
to the remote server and so the synchronization will be one way - from
remote to local. How suited is imapsync for continuous one-way
synchronization of mailboxes? Is there a better solution?
R. If messages are delivered remotely and you play locally with the
copy, in order to have fast access, then the synchronization can't
be one way. You may change flags, you may move messages in
different folders etc. The issue described is clearly
two-ways sync.
A better tool with this scenario is offlineimap,
designed for this issue, and faster than imapsync.
=======================================================================
Q. I need to log every output on a file named log.txt
R1. imapsync logs on a file by default, its name is given at the
beginning and the end of each run. This name is unique since
it is compound of the current date and time and user2 value.
R2. To change this default name, use --logfile log.txt
imapsync ... --logfile log.txt
=======================================================================
Q. Quantifier in {,} bigger than 32766 in regex; marked by <-- HERE in
m/(.{ <-- HERE 1,49947})(?:,|$)/ at Mail/IMAPClient.pm line 2121.
R. Do not use a bigger value than 3276 with --split1 or --split2
=======================================================================
Q. Couldn't create [INBOX.Ops/foo/bar]: NO Invalid mailbox name:
INBOX.Ops/foo/bar
Let begin by an explanation.
Example:
sep1 = /
sep2 = .
imapsync reverts each separator automatically.
a) All / character coming from host1 are converted to . (convert the separator)
b) All . character coming from host1 are converted to / (to avoid
intermediate unwanted folder creation).
So
INBOX/Ops.foo.bar (Ops.foo.bar is just one folder name) will be translated to
INBOX.Ops/foo/bar
Sometimes the sep1 character is not valid on host2 (character "/" usually)
R. Try :
--regextrans2 "s,/,X,g"
It'll convert / character to X
Choose X as you wish: _ or SEP or
any string (including the empty string).
This issue is automatically fixed by default since imapsync
release 1.513, use --nofixslash2 to suppress the fix.
=======================================================================
Q. Is it possible to sync also the UIDL of the POP3 server?
R. imapsync doesn't talk POP3 but I think you mean UID in IMAP.
See next question.
=======================================================================
Q. Is it possible to sync also the UIDs of the IMAP server?
R. UIDs in IMAP are chosen and created by the server, not by the
client software. imapsync is a client software. So UIDs cannot
be synced by any imap method.
UIDs may be synced via a rsync command between the imap servers but
it implies they are the same software, among other constraints.
=======================================================================
Q. The option --subscribe does not seem to work
R1. Use it with --subscribed
R2. There is also the --subscribe_all option that subscribe
to all folders on host2.
=======================================================================
Q. With huge account (many messages) when it comes to reading the
destination server it comes out this error:
"To Folder [INBOX.foobar] Not connected"
What can I do?
R. May be spending too much time on the source server, the connection
timed out on the destination server.
Try options --nofoldersizes
======================================================================
Q. Can Imapsync filter Spam during the sync?
R. No, imapsync doesn't detect Spam by itself. But I've seen blogs and
Spamassassin documentation explaining solutions to apply Spamassassin
to a imap mailbox. So you can apply one of these solutions on the host1
source mailbox before the imapsync run or on the destination host2
mailbox after the imapsync transfer.
http://www.stearns.org/doc/spamassassin-setup.current.html#isbg
http://euer.krebsco.de/using-spamassassin-on-a-remote-imap-host.html
https://github.com/ook/isbg
Imapsync can delegate this job during its IMAP syncs via the
--pipemess option but the underlying spam tool has to be written.
======================================================================
Q. How to migrate from uw-imap with an admin/authuser account?
R. Use the following syntax:
imapsync ... --user1="loginuser*admin_user" --password1 "admin_user_password"
======================================================================
Q. How to migrate from cyrus with an admin account?
R. Use:
imapsync ... \
--authuser1 admin_user ----password1 admin_user_password \
--user1 foo_user --ssl1
Instead of --ssl1 the alternative --tls1 can be used.
With --authuser1, the option --authmech1 PLAIN is set
automatically, you don't have to add it.
PLAIN authentication is the only way to go with --authuser1 for now.
So don't use --authmech1 SOMETHING with --authuser1 admin_user,
it will not work.
Same behavior with the --authuser2 option.
Do not forget the option --ssl1 or --tls1 since PLAIN auth is only
supported with ssl encryption most of the time. But it can
work without --ssl1 nor --tls1 if PLAIN is permitted in clear text
transmissions (the normal mode).
Add the AdminAccount to admins line in /etc/imapd.conf
Give AdminAccount lrswipkxtecda to the Cyrus Imap account
being migrated from, "joe" here.
Here is an example:
imapsync \
--host1 server1 \
--user1 joe \
--authuser1 AdminAccount \
--password1 AdminAccountPassword \
--ssl1 \
--host2 server2 \
--user2 joe \
--password2 joespassonserver2 \
--exclude "^user\."
======================================================================
Q: How to migrate from Sun Java Enterprise System / Sun One / iPlanet /
Netscape servers with an admin account?
R: Those imap servers don't allow the typical use of --authuser1 to use an
administrative account. They expect the use of an IMAP command called
proxyauth that is issued after login in as an administrative account.
For example, consider the administrative account 'administrator' and your
real user 'real_user'. The IMAP sequence would be:
OK [CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS
CHILDREN BINARY UNSELECT LANGUAGE STARTTLS XSENDER X-NETSCAPE XSERVERINFO
AUTH=PLAIN] imap.server IMAP4 service (Sun Java(tm) System Messaging
Server ...))
1 LOGIN administrator password
1 OK User logged in
2 PROXYAUTH real_user
2 OK Completed
In imapsync, you can achieve this by using the following options:
--host1 source.imap.server \
--user1 real_user \
--authuser1 administrator \
--proxyauth1 \
--passfile admin.txt
======================================================================
Q. Is there a way to delete the destination folder when the source
folder is no longer there?
R. Yes, use --delete2folders
--delete2folders : Delete folders in host2 that are not in host1 server.
For safety, first try it like this (it is safe):
--delete2folders --dry --justfolders --nofoldersizes
--delete2foldersonly <regex>: Deleted only folders matching regex.
Example: --delete2foldersonly "/^Junk$|^INBOX.Junk$/"
--delete2foldersbutnot <regex>: Do not delete folders matching regex.
Example: --delete2foldersbutnot "/Tasks$|Contacts$|Foo$/"
=======================================================================
Q. I would love to have a function to inject lines in the header.
Things like "X-migrated-from-foo: 20100617"
R. You can do that with:
imapsync ... --regexmess 's/\A/X-migrated-from-foo: 20100617\n/'
It will insert a first header line containing "X-migrated-from-foo: 20100617"
=======================================================================
Q. I want to play with headers line and --regexmess but I want to leave
the body as is.
R. The header/body separation is a blank line so an example:
--regexmess 's{\A(.*?(?! ^$))^Date:(.*?)$}{$1Date:$2\nX-Date:$2}gxms'
Will replace the next three lines
Message-ID: <499EF800.4030002@blabla.fr>
Date: Fri, 20 Feb 2009 19:35:44 +0100
From: Gilles LAMIRAL <lamiral@linux-france.org>
by the next four lines
Message-ID: <499EF800.4030002@blabla.fr>
Date: Fri, 20 Feb 2009 19:35:44 +0100
X-Date: Fri, 20 Feb 2009 19:35:44 +0100
From: Gilles LAMIRAL <lamiral@linux-france.org>
This example just add an header line "X-Date:" based on "Date:" line.
=======================================================================
Q. My imap server does not accept a message and warns
"Invalid header". What is the problem?
R. You fall in the classical mbox versus Maildir/ format
problem. May be you use a misconfigured procmail rule.
A header beginning like the following one is in the mbox
format, header line 1 has no colon behind "From", header
lines 2 through N do have a colon :
From foo@yoyo.org Sat Jun 22 01:10:21 2002
Return-Path: <foo@yoyo.org>
Received: ...
Any Maildir/ configured imap server may refuse this message since its
header is invalid. The first "From " line is not valid. It lacks a
colon character ":". To solve this issue you have several solutions
a) Remove manually this first "From " line for each message before
using imapsync.
b) Replace manually the whitespace by a colon in string "From " but you
might end with two "From:" lines (just have a look at the other
header lines of the message)
c) Run imapsync with the following option (this replaces "From "by "From:"):
--regexmess 's/\AFrom /From: /'
or may be better (no other "From:" collision):
d) Run imapsync with the following option (this replaces "From "by "X-om:"):
--regexmess 's/\AFrom /X-From: /'
e) Run imapsync with the following option (this removes the whole "From " line):
--regexmess 's{\AFrom\ [^\n]*(\n)?}{}gxms'
Solution e) is solution a) made by imapsync itself.
Solutions c) and d) keep "From " lines information
(normally it's useless to keep them)
Best solutions are e) or d).
=======================================================================
Q. The contact folder isn't well copied.
How to copy the contact folder?
R. Forget the destination server (choose the same)
Change the script around line 1426
# ITSD
$new_id = $from->copy($t_fold,$f_msg);
#$new_id = $to->append_string($t_fold,$string, $flags_f, $d);
and tried a copy of the mail instead an append_string. Because we are
using the same server, we can use $from->copy Therefore we seem to not
download and upload the message and therefore we do not have any
format issues. And now it works fine. (Thanks to Hansjoerg.Maurer)
======================================================================
Q: How can I write an .rpm with imapsync
R. You'll find an RPM imapsync.spec file in the directory learn/rpm/
It has been downloaded from
https://svn.fysik.dtu.dk/projects/rpmbuild/trunk/SPECS/imapsync.spec
It has been tested with imapsync 1.434 (May 2011) on CentOS5
and RedHat RHEL5 Linux. (Thanks to Ole Holm Nielsen).
This imapsync.spec is coming from Neil Brown work in 2007.
=======================================================================
Q. Where I can read up on the various IMAP RFCs?
R. Here:
RFC 3501 - INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1
http://www.faqs.org/rfcs/rfc3501.html
RFC2683 - IMAP4 Implementation Recommendations
http://www.faqs.org/rfcs/rfc2683.html
RFC 2595 - Using TLS with IMAP, POP3 and ACAP
http://www.faqs.org/rfcs/rfc2595.html
RFC 2822 - Internet Message Format
http://www.faqs.org/rfcs/rfc2822.html
RFC 2342 - IMAP4 Namespace
http://www.faqs.org/rfcs/rfc2342.html
RFC2180 - IMAP4 Multi-Accessed Mailbox Practice
http://www.faqs.org/rfcs/rfc2180.html
RFC 4549 - Synchronization Operations for Disconnected IMAP4 Clients
http://www.faqs.org/rfcs/rfc4549.html
=======================================================================
=======================================================================

82
FAQ.d/FAQ.Gmail.txt
View file

@ -1,11 +1,11 @@
#!/bin/cat
$Id: FAQ.Gmail.txt,v 1.25 2016/07/22 00:18:35 gilles Exp gilles $
$Id: FAQ.Gmail.txt,v 1.33 2017/09/05 15:11:20 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=====================================
Imapsync tips for Gmail accounts.
=====================================
=======================================================================
Imapsync tips for Gmail accounts.
=======================================================================
Questions anwswered in this FAQ are:
@ -19,17 +19,22 @@ Q. How to synchronize from XXX to Gmail?
Q. How to synchronize from Gmail to XXX?
Q. Can I use the Extension of the SEARCH command: X-GM-RAW described at
https://support.google.com/mail/answer/7190?hl=en
Q. How to avoid the [IMAP] prefix on Gmail side?
Q. I can't authenticate with Gmail via IMAP
and Gmail says "Please log in via your web browser"
Q. Can not open imap connection on [imap.gmail.com]
Unable to connect to imap.gmail.com
Q. Can I safely use --useuid for Gmail transfers?
Q. Gmail does not really delete messages in folder [Gmail]/All Mail
What happens? What can I do?
Q. Can I use the Extension of the SEARCH command: X-GM-RAW described at
https://support.google.com/mail/answer/7190?hl=en
https://developers.google.com/gmail/imap_extensions#extension_of_the_search_command_x-gm-raw
Q. How to avoid the [IMAP] prefix on Gmail side?
Q. Does imapsync have the capability to do 2 stage authentication?
@ -55,13 +60,25 @@ it has to be allowed in the Gmail configuration part:
Q. How many days does it take to transfer X GB?
R. Basically it takes X days to transfer X GB per account.
Gmail has usage limits per day
Gmail has usage limits per day and use throttlers when
they are overtaken
http://support.google.com/a/bin/answer.py?hl=en&answer=1071518
From the previous link,
it's 2X days to upload X GB to Gmail
it's X/2 days to download X BG from Gmail
but that's the theory.
From the previous link:
* it's 2X days to upload X GB to Gmail, it's why I suggest to add
--maxbytespersecond 10000
for uploading messages to Gmail
* it's X/2 days to download X BG from Gmail, it's why I suggest to add
--maxbytespersecond 20000
for downloading messages from Gmail
That's theoretical values that always work in practice. Try
upper values and see if they still work.
How Gmail says limits are reached?
This is either a disconnection with
"BYE Session expired, please login again"
or a very small rate, less than 1 Kib/s
=======================================================================
Q. How to synchronize from Gmail to Gmail?
@ -275,6 +292,8 @@ label CanWait and only it.
--skipcrossduplicates, will only put in "[Gmail]/All Mail"
the messages that are not labeled at all.
=======================================================================
Q. I can't authenticate with Gmail via IMAP
and Gmail says "Please log in via your web browser"
@ -310,10 +329,16 @@ R0. It looks like this issue is related to ipv6. Both ipv4 and ipv6
protocols should work with gmail and imapsync, I test that regularly,
imapsync works fine for both ipv4 and ipv6.
If you disable ipv6 then disable also ipv6 resolution!
Or at least, make ipv4 answers be taken before ipv6 since the default
names resolution order is to present ipv6 name resolutions first.
R1. A first simple solution is to use directly gmail ipv4 ip address:
The default names resolution order is to present ipv6 name resolutions
first. If you know how to make ipv4 answers be taken before ipv6
then tell me.
R1. First solution, run imapsync with the option --inet4:
imapsync ... --inet4
R2. A second solution is to use directly gmail ipv4 ip address:
imapsync ... --host1 64.233.184.108
@ -326,12 +351,23 @@ name resolution, try one of those:
Or go to http://ping.eu/nslookup/ to get the resolution.
R2. Second solution. Fix imapsync with the line:
use IO::Socket::SSL 'inet4' ;
Thanks to Chris Nolan to report, understand and fix this issue!
=======================================================================
Q. Can I safely use --useuid for Gmail transfers?
R. Yes, but I suggest to not use --useuid for Gmail transfers.
Using UIDs is useless with Gmail in the case of global duplicates
(duplicates across different folders). Gmail always accept a global
duplicate message as a new message, giving imapsync a new UID for this
message, and throw it away because it already has it. Gmail
will do this at each run so imapsync will always try to copy the
message, and Gmail will always accept and throw away the new copy. It
ends up with no duplicates on Gmail but a waste of bandwith and time,
which is the opposite goal of --usecache implied by --useuid.
=======================================================================
Q. Gmail does not really delete messages in folder [Gmail]/All Mail
What happens? What can I do?
@ -407,6 +443,8 @@ Q. How to migrate email from gmail to google apps?
R. Take a look at:
http://www.linux-france.org/prj/imapsync_list/msg00639.html
http://biasecurities.com/blog/2009/migrate-email-from-gmail-to-google-apps/
http://biasecurities.com/2009/02/migrate-email-from-gmail-to-google-apps/
http://www.thamtech.com/blog/2008/03/29/gmail-to-google-apps-email-migration/
=======================================================================
=======================================================================

64
FAQ.d/FAQ.ISP.txt
View file

@ -1,11 +1,11 @@
#!/bin/cat
$Id: FAQ.ISP.txt,v 1.4 2016/01/28 14:34:15 gilles Exp gilles $
$Id: FAQ.ISP.txt,v 1.8 2017/06/21 18:09:00 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=========================================================
Imapsync tips for ISP. Specific issues and solutions.
=========================================================
=======================================================================
Imapsync tips for ISP. Specific issues and solutions.
=======================================================================
* IMAPSync - usage scenario with ISP - by Flávio Zarur Lucarelli.
@ -18,12 +18,43 @@ Lamiral for all his help and hard work.
First of all, remember to use --dry to test things first always and
check the log file to see what would actually happen.
Initially, I used a method where I'd do an exact sync of source to
destination, deleting messages which were in destination, but not
source. I call this "Method 2", below. In this cenario, customer
shouldn't be using the destination account yet. Then, after I switch MX,
I'd do a final sync based on date. The big advantage of this is, you
get an exact sync.
Easier, however, is method 1, where I sync based on message ID, this
way, I can use the same syntax before and after MX change, with no
worries. Only disadvantage, you might not get an exact sync, there
might be some difference in terms of total emails in folders, due to
duplicates, emails that had same message ID in source server.
* Method 1 - sync based on message ID, can use same syntax before and
after MX change
imapsync --host1 imap.myisp.com --user1 user@domain.com --password1 pwd \
--host2 imap.myisp.com --user2 user@domain.com --password2 pwd \
--addheader
Note: add header adds message ID when it doesn't exist.
This syntax can also be used to sync different source accounts to one
same destination account, simply execute it as many times as desired,
switching source user (user1).
* Method 2 - exact sync source do destination, then sync based on date
after MX change
My first goal is to have an exact sync of an account from
current/source host to the new/destination host and be able to sync
several times. The --useuid parameter is very important for that
purpose. This is what I use:
imapsync --host1 imap.gmail.com --user1 user@domain.com --password1 pwd --ssl1 --host2 imap.myisp.com --user2 user@domain.com --password2 pwd --ssl2 --useuid --delete2 --delete2folders
imapsync --host1 imap.gmail.com --user1 user@domain.com --password1 pwd --ssl1 \
--host2 imap.myisp.com --user2 user@domain.com --password2 pwd --ssl2 \
--useuid --delete2 --delete2folders
This makes it so imap.myisp.com (destination) is an exact copy of the
account at imap.gmail.com (source). This is not a problem, since the
@ -51,7 +82,9 @@ cache) and delete such emails from source server. This way, customer's
mailbox is still intact on the source server, except new emails, which
get synced to new server and deleted from source.
imapsync --host1 imap.gmail.com --user1 user@domain.com --password1 pwd --ssl2 --folder INBOX --useuid --noexpungeaftereach --skipemptyfolders --maxage 1 --delete1
imapsync --host1 imap.gmail.com --user1 user@domain.com --password1 pwd --ssl1 \
--host2 imap.myisp.com --user2 user@domain.com --password2 pwd --ssl2 \
--folder INBOX --useuid --noexpungeaftereach --skipemptyfolders --maxage 1 --delete1
I personally prefer to keep a copy of users box intact in source, but
if that's not an issue for you, you can remove --folder INBOX and even
@ -70,5 +103,22 @@ lixeira (trash) and rascunhos (drafts).
So this was added:
--regextrans2 "s,\[Gmail\].,," --regextrans2 "s,E-mails enviados,Sent," --regextrans2 "s,/Lixeira,Trash," --regextrans2 "s,/Rascunhos,Drafts,"
--regextrans2 "s,\[Gmail\].,," \
--regextrans2 "s,E-mails enviados,Sent," \
--regextrans2 "s,Lixeira,Trash," \
--regextrans2 "s,Rascunhos,Drafts,"
*** Other cenarios
- Sync entire account into 1 folder of another account
imapsync --host1 xxx --user1 user1@domain.com --password1 secret1 --ssl1 \
--host2 yyy --user2 user2@domain.com --password2 secret2 --ssl2 \
--useuid --subfolder2 otheraccountfolder --delete2 --delete2foldersonly /otheraccountfolder/
Above is based on message UID, advantage of no dupes, however, user
must not be using such destination folder in destination account until
you finish syncing.
=======================================================================
=======================================================================

181
FAQ.d/FAQ.Massive.txt
View file

@ -1,20 +1,20 @@
#!/bin/cat
$Id: FAQ.Massive.txt,v 1.9 2016/03/25 19:59:45 gilles Exp gilles $
$Id: FAQ.Massive.txt,v 1.15 2017/06/17 13:45:15 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
==============================================
Imapsync tips for massive/bulk migrations.
==============================================
=======================================================================
Imapsync tips for massive/bulk migrations.
=======================================================================
Questions answered here are:
Q. I need to migrate hundred accounts, how can I do?
Q. I have to migrate 500k users using 400 TB of disk space.
How do I proceed?
How do I proceed? How about speed?
Q. How to determine what is the bottleneck in my current imapsync process?
Q. How to determine where is the bottleneck in an imapsync process?
Q. Can I run more instances of imapsync in parallel on a Windows host?
@ -29,7 +29,7 @@ Q. I need to migrate hundred accounts, how can I do?
R. If you have many mailboxes to migrate think about a little
script program. Write a file called file.txt (for example)
containing hosts users and passwords on both sides.
containing hosts, users and passwords on both sides.
The separator used in this example is ";"
The file.txt file contains for example:
@ -40,16 +40,23 @@ host003_1;user003_1;password003_1;host003_2;user003_2;password003_2;
host004_1;user004_1;password004_1;host004_2;user004_2;password004_2;
etc.
Most of the times, the first column (host001_1, host002_1 ...) will
contains the same value, the value of --host1 parameter. Same
thing for the third column (host001_2, host002_2).
On Unix the shell script can be:
#!/bin/sh
{ while IFS=';' read h1 u1 p1 h2 u2 p2 fake
do
imapsync --host1 "$h1" --user1 "$u1" --password1 "$p1" \
--host2 "$h2" --user2 "$u2" --password2 "$p2"
--host2 "$h2" --user2 "$u2" --password2 "$p2" "$@"
done
} < file.txt
You can add extra options inside this script, just after the variable "$@".
You can also pass extra options via the parameters of this script
since they will go in "$@"
Here is a complete Unix example ready to use:
http://imapsync.lamiral.info/examples/sync_loop_unix.sh
@ -59,54 +66,142 @@ On Windows the batch script can be:
CD /D %~dp0
SET csvfile=file.txt
FOR /F "tokens=1,2,3,4,5,6 delims=; eol=#" %%G IN (%csvfile%) DO (
FOR /F "tokens=1,2,3,4,5,6,7 delims=; eol=#" %%G IN (%csvfile%) DO (
imapsync ^
--host1 %%G --user1 %%H --password1 %%I ^
--host2 %%J --user2 %%K --password2 %%L ...
--host2 %%J --user2 %%K --password2 %%L %%M ...
)
The final ... can be replaced by nothing or any supplementary imapsync option.
You can add extra options inside this script, just after the variable %%M.
You can add extra options inside the file.txt, in the last column. Add
an extra semicolon at the end (optional)
Example:
host001_1;user001_1;password001_1;host001_2;user001_2;password001_2;
host002_1;user002_1;password002_1;host002_2;user002_2;password002_2;
becomes
host001_1;user001_1;password001_1;host001_2;user001_2;password001_2; --automap --addheader
host002_1;user002_1;password002_1;host002_2;user002_2;password002_2; --automap --addheader
Here is a complete Windows example nearly ready to use:
With this solution, options can be added, changed or removed per account.
Technically those options will go in %%M in the loop body
Here is a complete Windows example ready to use:
http://imapsync.lamiral.info/examples/sync_loop_windows.bat
Another solution to add extra arguments is to write another .bat that
calls sync_loop_windows.bat with the extra arguments, like this
for example:
sync_loop_windows.bat --automap --addheader --maxmessagespersecond 4
Technically those options will go in %arguments% in the loop body
of sync_loop_windows.bat
=======================================================================
Q. I have to migrate 500k users using 400 TB of disk space.
How do I proceed?
How do I proceed? How about speed?
R. Solution to this issue is two words: parallelism and measurements.
Since all 500k mailboxes are independent against each other,
they can be processed independently.
500k on 400TB is 800 MB per account on average.
Since all mailboxes are functionnaly independent, they can be processed
independently, here comes parallelism, lunching several imapsync
processes in parallel.
On any process involving several mechanisms there is always a
bottleneck among all elements taking part on the process. No one knows
in advance what is the first bottleneck. The first bottleneck has to
be determined, by measurements, not by guesses. Once this first
Meanwhile, mailboxes usually belong to the same server and syncs
share the same imapsync host via the same bandwidth, here come
some limitations and bottlenecks.
How many syncs can we run in parallel? here comes measurements.
1) Measure the total transfer rate by adding each one printed in each run.
Since adding this way is not so easy, just look at the overall
network rate of the imapsync host.
On Linux, nload is good candidate to measure this overall
network rate, every 6 seconds, on eth0 interface, values in Kbytes:
nload -t 6000 eth0 -u K
Another excellent network tool is dstat:
dstat -n -N eth0 6
On Windows, get the overall network rate with the classical
task manager (Ctrl-Alt-Sup), there is a network tab in it.
Don't hesitate to send me free good tools to measure the
overall transfer rate (the best would be one to sum up only
imap traffic but that's not mandatory at all).
2) Launch new parallel runs, one by one, as long as the total
transfer rate increase.
3) When the total transfer rate starts to diminish, stop new launches.
Note N as the number of parallel runs you got until then.
4) Only keep N-2 parallel runs for the future.
=======================================================================
Q. How to determine where is the bottleneck in an imapsync process?
R1. Divide and conquer.
In order to detect whether host1/link1 is the bottleneck or
host2/link2, we have several tests to explore:
1) run a sync from host1 to host1, with a host1 test account as destination.
This way, only host1+link1 are tested, host2 is not directly concerned.
If performances increase a lot then host2/link2 is the bottleneck.
2) run a sync from host2 to host2, with a host2 test account as destination.
This way, only host2+link2 are tested, host1 is not concerned.
If performances increase a lot then host1/link1 is the bottleneck.
If performances increase on both tests 1) and 2), I have no clue to explain that.
Same thing if they both decrease!
R2. Isolating and overcoming bottlenecks
On any process involving several mechanisms, among all elements taking
part on the process there is always a bottleneck. No one knows in
advance what is the first bottleneck. The first bottleneck has to be
determined, by measurements, not by guesses. Once this first
bottleneck is known and overcome then the next bottleneck has to be
determined and overcome too, if needed. Repeat the process of looking
for the next bottleneck and its resolution until you estimate the
for the next bottleneck and its elimination until you estimate the
transfer rates, money costs and final dates are good enough to proceed
the whole 500k/400TB migration.
the whole huge migration.
Possible bottlenecks:
- IMAP servers have artificial limits. For example Gmail and Office365
have throttle limits.
- Throttles.
IMAP servers have artificial limits.
For example Gmail, Office365, Exchange have throttle limits.
- Bandwidth, on any side, especially on small Internet connexions. But
usually bandwidth is not a bottleneck.
- Bandwidth.
Usually available bandwidth is not a bottleneck.
Meanwhile, it can be a bottleneck on small Internet connexions.
Imapsync downloads messages from host1 and upload messages to host2,
consider this in case the connexion are asymetric.
- I/O on disks.
I/O are a classical bottleneck, almost always forgotten.
Unlike CPU and RAM, Input/Output performances don't improve
very much as time goes on so it's often a bottleneck.
- Memory, on any side. Monitor your system doesn't swap on disk.
- RAM memory.
On all sides, monitor that your systems don't swap on disk,
because swapping memory on disks decreases performance by
a factor of 20, at least.
- CPU, on any side. When measuring that CPU is always 100% during a
transfer then it's useless to add imapsync processus on that host.
- I/O on disks. A classical one always forgotten. Unlike CPU and RAM
Input/Output performances don't improve very much as time goes on.
- CPU.
100% CPU during a whole transfer means the system is busy.
Usually CPU is not a problem with imapsync but it can be a problem
with one of the imap servers.
Most often CPU is not the real bottleneck, I/O are.
Other possible bottlenecks:
- Number of hosts available to run imapsync processes.
- Imapsync itself.
- Errors management.
@ -116,25 +211,6 @@ Possible bottlenecks:
- Bad luck.
- ...
=======================================================================
Q. How to determine what is the bottleneck in my current imapsync process?
R. Divide and conquer.
In order to detect whether host1/link1 is the bottleneck or
host2/link2, we have several tests to explore:
1) run a sync from host1 to host1, with a host1 test account as destination.
This way, only host1+link1 are tested. host2 is not concerned.
If performances increase a lot then host2/link2 is the bottleneck.
2) run a sync from host2 to host2, with a host2 test account as destination.
This way, only host2+link2 are tested. host1 is not concerned.
If performances increase a lot then host1/link1 is the bottleneck.
If performances increase on both tests 1) and 2), I have no clue to explain that.
Same thing if they both decrease!
=======================================================================
Q. Can I run more instances of imapsync in parallel on a Windows host?
@ -181,4 +257,5 @@ This file is removed at the end of a normal run.
You can safely ignore the warning if you don't use imapsync.pid file
to manage imapsync processes.
=======================================================================
=======================================================================

11
FAQ.d/FAQ.Memory.txt
View file

@ -1,11 +1,11 @@
#!/bin/cat
$Id: FAQ.Memory.txt,v 1.3 2016/05/09 13:01:22 gilles Exp gilles $
$Id: FAQ.Memory.txt,v 1.4 2017/01/06 14:11:13 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
============================================
Imapsync tips about memory issues.
============================================
=======================================================================
Imapsync tips about memory issues.
=======================================================================
=======================================================================
@ -27,4 +27,5 @@ R2. Usually "Out of memory" errors are related to old days,
Look at imapsync output first lines to get the Mail::IMAPClient
release used. Then upgrade Mail::IMAPClient Perl module if needed.
=======================================================================
=======================================================================
=======================================================================

71
FAQ.d/FAQ.Messages_Selection.txt
View file

@ -1,22 +1,55 @@
#!/bin/cat
$Id: FAQ.Messages_Selection.txt,v 1.6 2016/04/18 12:45:20 gilles Exp gilles $
$Id: FAQ.Messages_Selection.txt,v 1.12 2017/09/05 15:11:20 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=====================================
Imapsync tips to select messages.
=====================================
=======================================================================
Imapsync tips to select messages.
=======================================================================
By default, Imapsync syncs all messages, avoiding duplicates.
Questions anwswered in this FAQ are:
Q. What messages imapsync syncs by default?
Q. Is there a way we can specify a date range to sync emails?
If yes, can you please share an example?
Q. Is there a way we can specify an age to sync emails?
If yes, can you please share some examples?
Q. I want to sync messages based on their UID.
Q. Can I migrate only mails with attachments?
Q. How can I move messages marked \Deleted from all folders to
a dedicated folder?
Q. What are the selection criteria available with --search option?
=======================================================================
Q. What messages imapsync syncs by default?
R. By default, Imapsync syncs all messages, except duplicates.
=======================================================================
Q. Is there a way we can specify a date range to sync emails?
If yes, can you please share an example?
R. Yes, with the --search option.
R. Yes, a date range is possible with the --search option.
imapsync ... --search "SENTSINCE 1-Jan-2010"
or
imapsync ... --search "SENTBEFORE 31-Dec-2010"
or
imapsync ... --search "SENTSINCE 1-Jan-2010 SENTBEFORE 31-Dec-2010"
Months are specified like this:
Jan
Feb
Mar
@ -80,7 +113,8 @@ timezone) is earlier than the specified date.
If --noabletosearch is on then --minage and --maxage deal
with the internal dates given by a FETCH imap command but
not the Date: header. Internal date is the arrival date
in the mailbox.
in the mailbox. Same remark for --noabletosearch1 and
--noabletosearch2 but only for one side then.
=======================================================================
Q. I want to sync messages based on their UID.
@ -109,9 +143,29 @@ UIDs 20000 20002 20004:
If you search n UIDs then you have to put n-1 OR in the search line,
that's IMAP.
=======================================================================
Q. Can I migrate only mails with attachments?
R. Use:
imapsync ... --search "HEADER Content-Type multipart/mixed"
or more generally:
imapsync ... --search "OR HEADER Content-Disposition attachment HEADER Content-Type multipart/mixed"
=======================================================================
Q. What is the selection criteria available with --search option?
Q. How can I move messages marked \Deleted from all folders to
a dedicated folder?
R. To move \Deleted messages from all folders to a specific folder,
let's call it Trash, use:
imapsync ... --search DELETED --regextrans2 "s/.*/Trash/"
=======================================================================
Q. What are the selection criteria available with --search option?
R. The list of search criteria are listed below, an excerpt from RFC3501.
@ -250,4 +304,5 @@ http://www.faqs.org/rfcs/rfc3501.html
Messages that do not have the \Seen flag set.
=======================================================================
=======================================================================

36
FAQ.d/FAQ.OnlineUI.txt Normal file
View file

@ -0,0 +1,36 @@
#!/bin/cat
$Id: FAQ.OnlineUI.txt,v 1.1 2017/05/04 23:24:55 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=====================================================================
Imapsync tips about the online visual user interface
https://imapsync.lamiral.info/X/
=====================================================================
Q .Will I have any issues with browser timing out? What happens
if the connection is closed for whatever reason?
R. It should be ok
With the /X interface there is two connections (three connections in
fact but let simplify the picture), 1 is the Browser-WebServer
connection, 2 is the WebServer-ImapServers connections (imapsync
stuff).
If the Browser-WebServer connection is timeout (but it shouldn't
because of the log refresh), the imapsync sync might continue
anyway. To see if it continues or not, just do a sync again and the
interface will tell you that a sync is already going on, if the
"Sync!" button is grey/inactive then just reload the page (F5 or
similar).
Anyway, on the /X you can try to do several parallel runs on the same
mailbox even if there is no timeout, open a new tab/windows with /X
and start a same sync, it's safe, the /X will say, if any, that there
is already a current sync.
You can stop this sync with the "Abort!" button from any /X
tab/window, even from another browser or place. To doing this with
success, you have to give the same account parameters, same
credentials, or imapsync will ignore the demand.

10
FAQ.d/FAQ.Oracle-UCS.txt
View file

@ -1,11 +1,11 @@
#!/bin/cat
$Id: FAQ.Oracle-UCS.txt,v 1.3 2016/01/28 14:34:15 gilles Exp gilles $
$Id: FAQ.Oracle-UCS.txt,v 1.4 2017/01/06 14:11:13 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
==================================================
Imapsync tips for Oracle-UCS. Specific issues.
==================================================
=======================================================================
Imapsync tips for Oracle-UCS. Specific issues.
=======================================================================
Oracle-UCS was previously Sun JES, IPlanet, etc.
@ -16,3 +16,5 @@ Oracle-UCS was previously Sun JES, IPlanet, etc.
--skipmess 'm/[\x80-\xff]/'
=======================================================================
=======================================================================

21
FAQ.d/FAQ.Passwords_on_Unix.txt Normal file
View file

@ -0,0 +1,21 @@
#!/bin/cat
$Id: FAQ.Passwords_on_Unix.txt,v 1.1 2017/01/06 14:11:13 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=======================================================================
Imapsync issues with passwords on Unix.
=======================================================================
=======================================================================
Q. On Unix, some passwords contain some *(),;& characters. Login fails.
R. Use double-quotes within single-quotes in order to enclose the
password within double-quotes in the imap LOGIN command:
imapsync ... --password1 '"passw*(),;&rd"'
=======================================================================
=======================================================================

12
FAQ.d/FAQ.Passwords_on_Windows.txt
View file

@ -1,16 +1,18 @@
#!/bin/cat
$Id: FAQ.Passwords_on_Windows.txt,v 1.2 2016/05/10 19:19:16 gilles Exp gilles $
$Id: FAQ.Passwords_on_Windows.txt,v 1.3 2017/01/06 14:11:13 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
======================================================
Imapsync issues with passwords on Windows.
======================================================
=======================================================================
Imapsync issues with passwords on Windows.
=======================================================================
http://www.robvanderwoude.com/escapechars.php
http://stackoverflow.com/questions/3288552/how-can-i-escape-an-exclamation-mark-in-cmd-scripts
In case you're brave and relentless, understand and try this:
http://www.dostips.com/forum/viewtopic.php?f=3&t=1733
=======================================================================
Q. On Windows, some passwords contain $ characters. Login fails.
@ -42,3 +44,5 @@ or even
imapsync ... --password1 "==secret"
=======================================================================
=======================================================================

22
FAQ.d/FAQ.Release_Checklist.txt Normal file
View file

@ -0,0 +1,22 @@
#!/bin/cat
$Id: FAQ.Release_Checklist.txt,v 1.5 2017/09/07 12:38:45 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
===================================
Imapsync developper notes
===================================
Checklist before release a new release:
- Generate the README
- Run a spell checker on the README
- Read the README again slowly. Fix all issues, all
- Read the OPTIONS section of README and read it very slowly
- Read slowly README_Windows.txt
- Read slowly the TUTORIAL_Unix file in html
- Review the newsletter by running:
m4 -P W/ml_announce.in
- Review the general FAQ.d/FAQ.General.txt

29
FAQ.d/FAQ.Reporting_Bugs.txt Normal file
View file

@ -0,0 +1,29 @@
#!/bin/cat
$Id: FAQ.Reporting_Bugs.txt,v 1.2 2017/09/07 12:10:49 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
===================================================
Imapsync bugs reports
===================================================
=======================================================================
Q. How can I report bugs or problems I encountered with Imapsync?
R. Help me to help you: follow the following guidelines.
Report any bug or feature request directly to the author by
email at <gilles.lamiral@laposte.net>
Put a useful title with word "imapsync" in it: my spam filters
won't filter it.
Provide me any useful information. The simplest way is to attach
the complete log file, in case it is not too big, let say less
than 1MB. Don't zip it, it will slow my response.
Thanks!
=======================================================================

56
FAQ.d/FAQ.SSL_errors.txt Normal file
View file

@ -0,0 +1,56 @@
#!/bin/cat
$Id: FAQ.SSL_errors.txt,v 1.1 2017/05/24 19:16:40 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=====================================================
Imapsync SSL errors
=====================================================
Questions anwswered in this FAQ are:
Q. What are the errors
DEBUG: .../IO/Socket/SSL.pm:1165: local error: SSL write error
or
DEBUG: .../IO/Socket/SSL.pm:1088: local error: SSL read error
Q. What can I do to avoid those "SSL read/write errors"?
=======================================================================
Q. What are the errors
DEBUG: .../IO/Socket/SSL.pm:1165: local error: SSL write error
or
DEBUG: .../IO/Socket/SSL.pm:1088: local error: SSL read error
R. Like they claim, those errors are SSL errors. SSL is not directly
done by imapsync but by an underlying Perl module called
IO::Socket::SSL. Those errors arise sometimes and sometimes
they form a serie that ends with imapsync auto-abortion.
Those errors happen with some hosts but not with others,
it's often Exchange or Office365. I don't know what exactly happens.
Those errors happen more often on Windows than on Linux.
=======================================================================
Q. What can I do to avoid those "SSL read/write errors"?
R1. Remove all ssl/tls encryption
imapsync ... --nossl1 --notls1 --nossl2 --notls2
R2. If you don't want to quit encryption, rerun imapsync until the
complete sync is over. Those errors are not at the same place
each time, so imapsync will sync remaining messages at each run
until none remains.
R3. Run imapsync on a Linux machine, a VM is ok, there are less
SSL errors on Unix.
R4. Use https://imapsync.lamiral.info/X/
It's a Linux host so R3 applies there.
R5. Set up a ssltunnel proxy to the host.
Read the file FAQ.Security.txt
=======================================================================

152
FAQ.d/FAQ.Security.txt
View file

@ -1,44 +1,108 @@
#!/bin/cat
# $Id: FAQ.Security.txt,v 1.9 2016/08/19 17:53:21 gilles Exp gilles $
# $Id: FAQ.Security.txt,v 1.16 2017/04/26 21:50:00 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=======================================================
Imapsync tips about security. Issues and solutions.
=======================================================
=======================================================================
Imapsync tips about security. Issues and solutions.
=======================================================================
SSL: going to encryption before the imap session start.
It is on port 993.
TLS: going to encryption after the imap session start but before
the credential are sent. It is on port 143.
=======================================================================
Q. Is running this program a secure method of transferring emails?
Are there any security concerns?
R. Well, it depends. Use encryption, secure the access to the host running
imapsync and everything shall be safe.
=======================================================================
Q. I noticed that the online UI has no option for TLS/SSL
Is this secure? Is this more secure that using the .bat file
on my computer?
R1. The online UI does TLS/SSL imap connections if the imap
servers support TLS/SSL. If you are concerned by security then
using the .bat file or .sh on your computer should be more secure
since you can examine and secure it by yourself, no matter high is
your paranoïd spirit. The online UI security is mine, I am
concerned by security, not to the upmost high level possible
but I won't give you direct access to the host to discover my level.
With a good guy spirit, feel free to try to break the online UI
security and report me any security issue you encounter, I'll do my
best to fix them as soon as possible. Drop me a note before
starting because I may detect a sort of abuse and ban
definitively your IPs.
=======================================================================
Q. Are transferred emails/attachments stored on any other
server/location aside from my originating/destination server(s)?
R. No!
=======================================================================
Q. Other than changing passwords on the originating/destination email
accounts once the relevant emails have been moved,
are there any other security tips I should know?
R. Secure the host where imapsync is running since credentials are
on it.
=======================================================================
Q. I need to transfer mail from an imap server to an other imap server.
Which ports need to be open on the firewall to make this possible?
R. It depends. Open either:
* port 143 in basic (no special option) or tls mode (--tls1 or --tls2)
* port 993 in ssl mode (--ssl1 or --ssl2)
=======================================================================
Q. Does imapsync support IMAP TLS?
R. Use --tls1 and/or --tls2 options
R1. Yes.
Use --tls1 and/or --tls2 options:
--tls1 tells imapsync to use tls on host1.
--tls2 tells imapsync to use tls on host2.
--tls1 tells imapsync to use tls on host1.
--tls2 tells imapsync to use tls on host2.
R2. Since imapsync release 1.755 TLS mode is activated automatically
if the server announce it supports it, STARTTLS inside the response
to CAPABILITY, and if neither --notls nor --ssl is explicitely
mentioned on the command line options.
=======================================================================
Q. Does imapsync support IMAP over SSL (IMAPS)?
R. Yes natively since release 1.161.
still, 2 ways, at least to use ssl:
Still, there are 2 ways, at least, to use ssl:
a) Use native --ssl1 and/or --ssl2 options
--ssl1 tells imapsync to use ssl on host1.
--ssl2 tells imapsync to use ssl on host2.
--ssl1 tells imapsync to use ssl on host1.
--ssl2 tells imapsync to use ssl on host2.
b) Use stunnel
http://www.stunnel.org/
Assuming there is an imaps (993) server on imap.foo.org,
on your localhost machine (or bar machine) run :
stunnel -c -d imap -r imap.foo.org:imaps
or using names instead of numbers
stunnel -c -d 143 -r imap.foo.org:993
then use imapsync on localhost (or bar machine) imap (143) port.
If the local port 143 is already taken then use a free one, 10143.
on your localhost machine (or bar machine), run:
c) Other example for gmail with no root access to open port 143
stunnel -c -d imap -r imap.foo.org:imaps
or using numbers instead of names:
stunnel -c -d 143 -r imap.foo.org:993
then use imapsync on localhost (or bar machine) imap (143) port.
If the local port 143 is already taken then use a free one,
like 10143 for example.
c) Other example for accessing gmail with no local root access
to open port 143
stunnel -f -P '' -c -d 9993 -r imap.gmail.com:993
@ -47,12 +111,36 @@ Then, to access gmail as host2 use:
imapsync ... --host2 localhost --port2 9993 --nossl2
=======================================================================
Q.How can I test an ssl imap connection without imapsync?
R1. Use either ncat or telnet-ssl or openssl commands like in the
following examples with imap.gmail.com server:
ncat --ssl -C imap.gmail.com 993
telnet-ssl -z ssl imap.gmail.com 993
openssl s_client -crlf -connect imap.gmail.com:993
The previous commands are interactive, hit ctrl-c
to finish them. If you want to finish automatically, then use:
{ sleep 2; echo "a logout"; sleep 1; } | ncat --ssl -C imap.gmail.com 993
=======================================================================
Q. How can I manually test a login via ssl?
R. Use ncat or telnet-ssl like in this example:
R. Use either ncat or telnet-ssl or openssl commands like in the
following examples with imap.gmail.com server:
ncat --ssl -C imap.gmail.com 993
ncat --ssl -C imap.gmail.com 993
telnet-ssl -z ssl imap.gmail.com 993
openssl s_client -crlf -connect imap.gmail.com:993
Tipical dialog for an imap LOGIN command:
* OK Gimap ready for requests from 78.196.254.58 q1mb175739668wix
a LOGIN "gilles.lamiral@gmail.com" "secret"
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE ... ESEARCH
@ -61,21 +149,27 @@ b LOGOUT
* BYE LOGOUT Requested
b OK 73 good day (Success)
The client part to type is "a LOGIN ..." and "b LOGOUT" without
the double-quotes.
The client part you have to type is
a LOGIN ...
b LOGOUT
while replacing ... by your credentials values.
=======================================================================
Q.How to test a ssl imap connection without imapsync?
Q.How can I test an tls imap connection without imapsync?
R1.Use openssl command like the following,
an example with imap.gmail.com server:
R1. Use openssl command like the following example with
an outlook.office365.com server:
openssl s_client -crlf -connect imap.gmail.com:993
openssl s_client -crlf -starttls imap -connect outlook.office365.com:143
The previous command is an interactive connection, hit ctrl-c
to finish it. If you want to finish it gently, then use:
The previous commands are interactive, hit ctrl-c
to finish them. If you want to finish automatically, then use:
{ sleep 2; echo "a logout"; sleep 1; } | openssl s_client -crlf -starttls imap -connect outlook.office365.com:143
Replace outlook.office365.com with your imap server name.
{ sleep 2; echo "a logout"; sleep 1; } | openssl s_client -crlf -connect imap.gmail.com:993
======================================================================
Q. Imapsync used to use SSL_VERIFY_PEER now it uses SSL_VERIFY_NONE.
@ -130,3 +224,5 @@ b) or use stunnel :
c) or use stunnel on inetd
imaps stream tcp nowait cyrus /usr/sbin/stunnel -s cyrus -p /etc/ssl/certs/imapd.pem -r localhost:imap2
=======================================================================
=======================================================================

17
FAQ.d/FAQ.SmarterMail.txt
View file

@ -1,11 +1,11 @@
#!/bin/cat
$Id: FAQ.SmarterMail.txt,v 1.8 2016/01/28 14:34:15 gilles Exp gilles $
$Id: FAQ.SmarterMail.txt,v 1.11 2017/09/05 15:11:20 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=================================================================
Imapsync tips for SmarterMail. Specific issues and solutions.
=================================================================
=======================================================================
Imapsync tips for SmarterMail. Specific issues and solutions.
=======================================================================
=======================================================================
@ -14,7 +14,7 @@ Q. Synchronizing from SmarterMail to XXX
On Unix:
imapsync --host1 imap.d1.org --user1 joe --password1 secret1 \
--host2 imap.d2.org --user2 joe --password2 secret2 \
--sep1 "/" --prefix1 "" --useheader Message-Id \
--sep1 "/" --prefix1 "" --useheader Message-Id --noabletosearch1 \
--regextrans2 "s,Deleted Items,Trash," \
--regextrans2 "s,Junk E-Mail,Junk," \
--regextrans2 "s,Sent Items,Sent,"
@ -22,7 +22,7 @@ imapsync --host1 imap.d1.org --user1 joe --password1 secret1 \
On Windows:
imapsync.exe --host1 imap.d1.org --user1 joe --password1 secret1 ^
--host2 imap.d2.org --user2 joe --password2 secret2 ^
--sep1 "/" --prefix1 "" --useheader Message-Id ^
--sep1 "/" --prefix1 "" --useheader Message-Id --noabletosearch1 ^
--regextrans2 "s,Deleted Items,Trash," ^
--regextrans2 "s,Junk E-Mail,Junk," ^
--regextrans2 "s,Sent Items,Sent,"
@ -36,13 +36,14 @@ Q. Synchronizing from XXX to SmarterMail
On Unix:
imapsync --host1 imap.d1.org --user1 joe --password1 secret1 \
--host2 imap.d2.org --user2 joe --password2 secret2 \
--sep2 "/" --prefix2 "" --useheader Message-Id
--sep2 "/" --prefix2 "" --useheader Message-Id --noabletosearch2
On Windows:
imapsync.exe --host1 imap.d1.org --user1 joe --password1 secret1 ^
--host2 imap.d2.org --user2 joe --password2 secret2 ^
--sep2 "/" --prefix2 "" --useheader Message-Id
--sep2 "/" --prefix2 "" --useheader Message-Id --noabletosearch2
=======================================================================
=======================================================================

32
FAQ.d/FAQ.TTL.txt Normal file
View file

@ -0,0 +1,32 @@
#!/bin/cat
$Id: FAQ.TTL.txt,v 1.1 2017/05/03 22:27:45 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=====================================================================
Imapsync tips about TTL when changing name resolution of hosts
=====================================================================
Why decrease the TTL (Time To Live) delay in DNS configuration, down
to 5 minutes?
A small TTL is not mandatory, it's safer and more easy to work with when
migrating.
It's about how long it takes to be sure the users are using the new
imap host, and how long it takes to be sure all new incoming messages
are going right now at the right place. Will you shut down the old
server just after the resolution change? I guess you won't and you'll
be right.
The TTL is just a value, very well supported by machines, with a
little name resolution supplementary work for them when it is set to a
small value like 5 minutes, but it's a tremendous comfort for migrator
people like us.
Be sure to wait 24h after this TTL change before changing any
resolution since the TTL change has to be propagated as well. After
the migration done, no problem to set back the TTL to 24h or more. If
you can't decrease TTL under 4h or even 24h, it's ok anyway, imapsync
can sync the new messages dropped in the old server.

19
FAQ.d/FAQ.Use_addheader.txt
View file

@ -1,11 +1,11 @@
#!/bin/cat
$Id: FAQ.Use_addheader.txt,v 1.1 2016/06/01 12:25:56 gilles Exp gilles $
$Id: FAQ.Use_addheader.txt,v 1.4 2017/01/25 23:54:02 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
============================================
Imapsync --addheader suggestion.
============================================
=======================================================================
Imapsync --addheader suggestion.
=======================================================================
=======================================================================
Q. What means this log message:
@ -22,9 +22,12 @@ R. In order to sync messages from one account to another Imapsync has
messages except for special folders like Sent or Draft where
messages in those folders don't have "Message-Id:" nor
"Received:" headers.
Here comes -addheader option. Option --addheader adds
a "Message-Id" header consisting of the imap UID of the message
on the host1 folder, like "Message-Id: 12345@imapsync".
This way, messages are well identified on both sides.
Here comes --addheader option. When a message has no "Message-Id:"
nor "Received:", option --addheader adds a "Message-Id" header
consisting of the imap UID of the message on the host1 folder,
like "Message-Id: 12345@imapsync".
This way, messages are well identified on both sides,
transferred, and only once.
=======================================================================
=======================================================================

99
FAQ.d/FAQ.Use_cache.txt Normal file
View file

@ -0,0 +1,99 @@
$Id: FAQ.Use_cache.txt,v 1.4 2017/02/14 17:35:22 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
============================================
Imapsync --usecache option
============================================
Questions anwswered in this FAQ are:
Q. On Windows, with --useuid or --usecache a problem occurs with long
nested folder names. The error message is:
"No such file or directory; The filename or extension is too long"
Q. Inode problem with --usecache on Linux
Questions and their answers:
=======================================================================
Q. On Windows, with --useuid or --usecache a problem occurs with long
nested folder names. The error message is:
"No such file or directory; The filename or extension is too long"
R. This comes from a Windows limitation on pathnames.
No more than 260 characters are allowed for pathnames.
See more details on page
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247.aspx#maxpath
The workaround solution given at the previous link,
ie using \\?\D:", does not work for imapsync.
So this imapsync Windows bug is still pending and needs a fix using
a different technique to cache, like a database file for example.
A solution to fix the issue is to use a Linux virtual host on a
Windows box, with VirtualBox or VmWare etc. There is no bug this way,
pathnames can be several thousands characters long.
Better said that done but not so difficult nor expensive these days,
VirtualBox is free and VmWare Player is free for personal or test use.
If you have to stick on Windows, there are two good workarounds
to reduce the cache directory name:
1) Use --tmpdir "D:\\temp" or simply --tmpdir "D:" and imapsync
will build and use the cache in the sub directory
D:\imapsync_cache\
2) add two equivalent entries in the etc/hosts for host1 imap.truc.org
and host2 imap.trac.org.
If you map the ip of imap.truc.org just with the letter a
and same thing for imap.trac.org then you gain characters
etc/hosts
192.168.12.1 a
192.168.55.3 b
Then use:
imapsync --host1 a --host2 b ...
You can get the ip of a host with the ping command line
C:\> ping imap.truc.org
3) A third solution is to not use options --useuid nor --usecache
Fixing this long path problem directly in imapsync is in the TODO file
for a very long time.
=======================================================================
Q. Inode problem with --usecache on Linux
R. You may run out of inodes using --usecache, especially with large
migration. Option --usecache creates a empty file per email message
in order to keep the UIDs mapping between account1 and account2.
So, if you plan to sync regularly 10 millions messages over a period
of migration then the filesystem of --tmpdir needs 10 millions of free
inodes. If it hasn't those free inodes then create a new special
filesystem devoted to the imapsync cache.
# Create a file of 10 GB (10 millions*1024):
dd if=/dev/zero of=/var/tmp/fscache bs=1M count=10000
# Create a filesystem where each file is only 1024 bytes per inode:
mkfs.ext2 -F -i 1024 /var/tmp/fscache
# Mount this brand new filesystem
mount -o loop /var/tmp/fscache /var/tmp/cachedir
# Tell imapsync to use it
imapsync ... --tmpdir /var/tmp/cachedir/
=======================================================================
=======================================================================

9
FAQ.d/FAQ.User_Concurrent_Access.txt
View file

@ -1,11 +1,11 @@
#!/bin/cat
$Id: FAQ.User_Concurrent_Access.txt,v 1.1 2016/07/22 00:00:08 gilles Exp gilles $
$Id: FAQ.User_Concurrent_Access.txt,v 1.2 2017/01/06 14:11:13 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=======================================
Imapsync and user concurrent access.
=======================================
=======================================================================
Imapsync and user concurrent access.
=======================================================================
=======================================================================
Q. What happens if a user access the mailbox during the transfer process?
@ -33,3 +33,4 @@ on what he does, where he does and when:
* message flag change => might be canceled by the sync
=======================================================================
=======================================================================

46
FAQ.d/FAQ.Various_Server_Softwares.txt
View file

@ -1,11 +1,11 @@
#!/bin/cat
$Id: FAQ.Various_Server_Softwares.txt,v 1.4 2016/01/28 14:34:15 gilles Exp gilles $
$Id: FAQ.Various_Server_Softwares.txt,v 1.8 2017/09/05 15:11:20 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
====================================================
Imapsync tips for various imap server softwares.
====================================================
=======================================================================
Imapsync tips for various imap server softwares.
=======================================================================
@ -43,22 +43,6 @@ http://www.safetynet-it.com/it-support/mac-kerio-server-to-microsoft-exchange-20
http://www.safetynet-it.com/it-support/mac-kerio-server-to-microsoft-exchange-2010-migration-2/
=======================================================================
Q. Synchronizing from Yahoo to XXX
R. Use --host1 imap.mail.yahoo.com --ssl1
./imapsync \
--host1 imap.mail.yahoo.com \
--user1 billy \
--password1 secret \
--ssl1 \
--host2 XXX \
--user2 billy \
--password2 secret
SSL is mandatory for yahoo since november 2011.
=======================================================================
Q. from Microsoft's Exchange 2007 to Google Apps for your Domain
(GAFYD)
@ -228,14 +212,14 @@ R. Old Softalk releases don't support the IMAP SEARCH command.
Here are the options to get it working.
imapsync ... --sep1 '.' --prefix1 '' \
--noabletosearch --nocheckmessageexists --addheader
--noabletosearch1 --nocheckmessageexists --addheader
(Thanks to Andrew Tucker)
======================================================================
Q. From or to QQMail IMAP4Server
R. imapsync ... --noabletosearch
R. imapsync ... --noabletosearch1
======================================================================
Q. From FirstClass to XXX
@ -264,8 +248,20 @@ Here is a command line used to migrate from FirtClass release 12:
--regextrans2 "s,&AC8-,-,g" \
--regextrans2 "s,&APg-,oe,g"
On Windows, in the previous example containing \$1 you have to
replace the two \$1 by $1 (remove the \ before $).
On Windows:
imapsync.exe ... ^
--automap ^
--usecache ^
--useheader Message-ID ^
--idatefromheader ^
--addheader ^
--regextrans2 "s,(/|^) +,$1,g" ^
--regextrans2 "s, +(/|$),$1,g" ^
--regextrans2 "s/[\^]/_/g" ^
--regextrans2 "s/['\\]/_/g" ^
--regextrans2 "s,^&AC8-,-,g" ^
--regextrans2 "s,^&APg-,oe,g"
Special thanks to Kristian Wind and Joey Alexander for helping me
writing this FAQ item.
@ -283,3 +279,5 @@ R. Do NOT use --usecache since new UIDs are not given by FTGate and also
--sep2 / --prefix2 "" \
--useheader Message-Id \
=======================================================================
=======================================================================

34
FAQ.d/FAQ.Virus.txt Normal file
View file

@ -0,0 +1,34 @@
$Id: FAQ.Virus.txt,v 1.2 2017/07/03 19:30:19 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=======================================================================
Imapsync and virus scanners
=======================================================================
=======================================================================
Q. My virus scanner claims imapsync.exe is a virus/malware/trojan etc.
What the hell?
R. Yes, I found the same. Two antivirus, Baidu and Jiangmin, report that
imapsync might have a trojan in it.
All others virus scanner say imapsync.exe is ok.
I've done this test on imapsync.exe release 1.727 on the two following
meta-virus-scanners:
https://www.virustotal.com/
https://www.metadefender.com/
R2. Explanation:
It may come from the fact that imapsync checks whether there is a
new realease available at http://imapsync.lamiral.info/VERSION
It's explained here:
https://imapsync.lamiral.info/#NUMBERS
Any other explanation is welcome!
R3. There is no virus alerts reported for the Perl script imapsync
itself. The binary imapsync.exe is just a compiled version of
the script imapsync, with perl interpreter itself and all modules
needed by imapsync in order to make imapsync.exe a standalone software.

8
FAQ.d/FAQ.XOAUTH2.txt
View file

@ -1,11 +1,11 @@
#!/bin/cat
$Id: FAQ.XOAUTH2.txt,v 1.11 2016/07/27 23:08:19 gilles Exp gilles $
$Id: FAQ.XOAUTH2.txt,v 1.12 2017/01/06 14:11:13 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
======================================================================
=======================================================================
Imapsync tips to use XOAUTH2 authentication (Gmail) and old XOAUTH
======================================================================
=======================================================================
=======================================================================
@ -187,4 +187,6 @@ Some notes about configuring the Google Apps XOAUTH:
"https://mail.google.com/" configured
(https://support.google.com/a/bin/answer.py?answer=162106).
=======================================================================
=======================================================================

54
FAQ.d/FAQ.Yahoo.txt Normal file
View file

@ -0,0 +1,54 @@
$Id: FAQ.Yahoo.txt,v 1.6 2017/06/17 14:42:01 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=======================================================================
Imapsync tips for Yahoo.
=======================================================================
=======================================================================
Q. Synchronizing from Yahoo to XXX
R. Use --host1 imap.mail.yahoo.com --ssl1
./imapsync \
--host1 imap.mail.yahoo.com \
--user1 billy \
--password1 secret \
--ssl1 \
--host2 XXX \
--user2 billy \
--password2 secret
SSL is mandatory for yahoo since November 2011.
You also need to go to Yahoo, security and enable
"Allow less secure apps to login",
otherwise the login will not work.
To enable less secure apps to login:
* Login to the Yahoo mail account,
* click on the account name or the avatar and select "Account Info",
* click on "Account security",
* turn off "Two steps verification"
* turn on "Allow apps that use less secure authentication".​
Thanks to Eugen Mayer for this last point.
Another solution:
* Login to the Yahoo mail account,
* click on the account name or the avatar and select "Account Info",
* click on "Account security",
* Turn on "Two-step verification"
* Click on "Manage app passwords" and
generate a specific password for imapsync,
choose "Other app" at the bottom and type imapsync
since it is not in the predefined apps.
* Use this password with imapsync.
=======================================================================
=======================================================================

57
FAQ.d/htaccess.txt
View file

@ -1,31 +1,40 @@
# $Id: htaccess.txt,v 1.10 2016/07/22 00:20:13 gilles Exp gilles $
# $Id: htaccess.txt,v 1.18 2017/09/03 03:14:48 gilles Exp gilles $
AddDescription "<b>Back to Imapsync main page</b>." ..
AddDescription "<b>Archiving tips</b>." FAQ.Archiving.txt
AddDescription "<b>Connection issues</b>." FAQ.Connection.txt
AddDescription "<b>Contacts & Calendars issues</b>." FAQ.Contacts_Calendars.txt
AddDescription "<b>Dates issues</b>." FAQ.Dates.txt
AddDescription "<b>Domino</b>." FAQ.Domino.txt
AddDescription "<b>Dovecot</b>." FAQ.Dovecot.txt
AddDescription "<b>Duplicated</b> messages issues." FAQ.Duplicates.txt
AddDescription "<b>Emptying</b> an account." FAQ.Emptying.txt
AddDescription "<b>Exchange 20xx</b> and <b>Office365</b>." FAQ.Exchange.txt
AddDescription "<b>Changing folders names</b>." FAQ.Folders_Mapping.txt
AddDescription "<b>Selecting folders</b>." FAQ.Folders_Selection.txt
AddDescription "<b>Flags</b>." FAQ.Flags.txt
AddDescription "<b>Gmail</b> accounts." FAQ.Gmail.txt
AddDescription "<b>ISP tips</b>." FAQ.ISP.txt
AddDescription "<b>Massive/bulk migrations</b>." FAQ.Massive.txt
AddDescription "<b>Back to Imapsync main page</b>." ..
AddDescription "<b>Authenticate via an admin account</b>." FAQ.Admin_Authentication.txt
AddDescription "<b>Authentication failures</b>." FAQ.Authentication_failure.txt
AddDescription "<b>Archiving tips</b>." FAQ.Archiving.txt
AddDescription "<b>Connection issues</b>." FAQ.Connection.txt
AddDescription "<b>Contacts & Calendars issues</b>." FAQ.Contacts_Calendars.txt
AddDescription "<b>Dates issues</b>." FAQ.Dates.txt
AddDescription "<b>Domino</b>." FAQ.Domino.txt
AddDescription "<b>Dovecot</b>." FAQ.Dovecot.txt
AddDescription "<b>Duplicated</b> messages issues." FAQ.Duplicates.txt
AddDescription "<b>Emptying</b> an account." FAQ.Emptying.txt
AddDescription "<b>Exchange 20xx</b> and <b>Office365</b>." FAQ.Exchange.txt
AddDescription "<b>Changing folders names</b>." FAQ.Folders_Mapping.txt
AddDescription "<b>Selecting folders</b>." FAQ.Folders_Selection.txt
AddDescription "<b>Flags</b>." FAQ.Flags.txt
AddDescription "<b>General</b> and <b>pot-pourri</b> issues" FAQ.General.txt
AddDescription "<b>Gmail</b> accounts." FAQ.Gmail.txt
AddDescription "<b>ISP tips</b>." FAQ.ISP.txt
AddDescription "<b>Massive/bulk migrations</b>." FAQ.Massive.txt
AddDescription "<b>Memory issues</b>." FAQ.Memory.txt
AddDescription "<b>Password & special characters on Windows</b>." FAQ.Passwords_on_Windows.txt
AddDescription "<b>Password & special characters on Unix</b>." FAQ.Passwords_on_Unix.txt
AddDescription "<b>Selecting messages</b>." FAQ.Messages_Selection.txt
AddDescription "<b>Oracle-UCS</b>." FAQ.Oracle-UCS.txt
AddDescription "<b>Security</b>." FAQ.Security.txt
AddDescription "<b>SmarterMail</b>." FAQ.SmarterMail.txt
AddDescription "<b>User concurrent access</b>." FAQ.User_Concurrent_Access.txt
AddDescription "<b>Why use --addheader?</b>." FAQ.Use_addheader.txt
AddDescription "<b>Various imap server</b> softwares." FAQ.Various_Server_Softwares.txt
AddDescription "<b>XOAUTH2</b> (<b>Gmail</b>)." FAQ.XOAUTH2.txt
AddDescription "<b>Oracle-UCS</b>." FAQ.Oracle-UCS.txt
AddDescription "<b>Guidelines to report bugs</b>." FAQ.Reporting_Bugs.txt
AddDescription "<b>Security</b>." FAQ.Security.txt
AddDescription "<b>SSL errors</b>." FAQ.SSL_errors.txt
AddDescription "<b>SmarterMail</b>." FAQ.SmarterMail.txt
AddDescription "<b>Option --usecache and inodes</b>." FAQ.Use_cache.txt
AddDescription "<b>User concurrent access</b>." FAQ.User_Concurrent_Access.txt
AddDescription "<b>Why use --addheader?</b>." FAQ.Use_addheader.txt
AddDescription "<b>Various imap server</b> softwares." FAQ.Various_Server_Softwares.txt
AddDescription "<b>XOAUTH2</b> (<b>Gmail</b>)." FAQ.XOAUTH2.txt
AddDescription "<b>Yahoo</b>." FAQ.Yahoo.txt
AddDescription "<b>Where this Description column comes from</b>." htaccess.txt