zydronium/imapsync
1
0
Fork 0
mirror of https://github.com/imapsync/imapsync.git synced 2025-07-25 11:38:15 +02:00
Code Issues Projects Releases Packages Wiki Activity

1.684

Browse source
This commit is contained in:
Nick Bebout 2016-09-19 10:15:41 -05:00
parent d0595b7dfd
commit 8d24f07718
110 changed files with 6018 additions and 4489 deletions
Download patch file Download diff file Expand all files Collapse all files

25
FAQ.d/FAQ.Connection.txt Normal file
View file

@ -0,0 +1,25 @@
#!/bin/cat
$Id: FAQ.Connection.txt,v 1.2 2016/03/16 13:33:51 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
============================================
Imapsync tips about connection issues.
============================================
=======================================================================
Q.How to test a ssl imap connection without imapsync?
R1.Use openssl command like the following,
an example with imap.gmail.com server:
openssl s_client -connect imap.gmail.com:993
The previous command is an interactive connection, hit ctrl-c
to finish it. If you want to finish it gently, then use:
{ sleep 1; echo "a LOGOUT"; } | openssl s_client -connect imap.gmail.com:993
=======================================================================

71
FAQ.d/FAQ.Dates.txt Normal file
View file

@ -0,0 +1,71 @@
#!/bin/cat
$Id: FAQ.Dates.txt,v 1.2 2016/03/07 02:08:12 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
===============================
Imapsync tips about dates.
===============================
=======================================================================
Q. We have found that the time and date displayed have been changed to
the time at which the file was synchronized.
R. This is the case by default with some email readers like:
- Outlook 2003
- Ipad
but not with:
- Mutt
- Thunderbird
- Zimbra
- Gmail
A thing to keep in mind, imapsync does not touch any byte of messages
unless told to do so by option --regexmess or --pipemess
Messages on both parts should be identical.
Now I explain the whole picture about dates of messages.
There are several different dates for any message.
First, there is the "Date:" header. Most of the time, this date is set
by the MUA (MUA means Mail User Agent) it can be Outlook, Thunderbird,
a webmail or Mutt. The Date header is usually the date the message was written
or sent the first time, at the writer side.
This Date: header line is never changed by any transfer or copy.
If an email reader uses the "Date:" header for displaying the date of a
message then no problem should arise.
Another date is the internal date. In IMAP, the internal date is handled
by the server and normally it corresponds to the arrival date in the mailbox.
The IMAP protocol allows the internal date to be set by a email client
when a message is delivered by the imap APPEND command.
Imapsync synchronizes internal dates by default, internal dates on host2
should then be the same as the internal dates on host1.
If an email reader uses the internal date for displaying the date of a
message then the sync date problem only occurs when the host2 server software
ignores the internal date given by imapsync during the APPEND imap
command. It happens with some imap servers.
There are also the "Received:" header lines. Each time a message travels
a SMTP server, this one adds a "Received:" header line. Sometimes
some email clients use the last "Received:" header date as the date of the
message. And some IMAP servers software add a "Received:" line after
and imap transfer. If those both conditions are met then the date
displayed become the transfer date even if imapsync
has done its best to keep all the dates synchronized. Bad luck.
Solutions:
a) Use a another email client or configure it in order it sorts messages
by sent date, the "Date:" header line.
b) Use a imap server that respects the imap RFC and accepts
the internal date set by imapsync.
c) Try to understand why the reader shows another date than the "Date:" line.
=======================================================================

4
FAQ.d/FAQ.Domino.txt
View file

@ -1,5 +1,7 @@
#!/bin/cat
# $Id: FAQ.Domino.txt,v 1.5 2016/01/18 18:14:12 gilles Exp gilles $
# $Id: FAQ.Domino.txt,v 1.6 2016/01/28 14:34:15 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=============================
Imapsync tips for Domino.

4
FAQ.d/FAQ.Dovecot.txt
View file

@ -1,5 +1,7 @@
#!/bin/cat
$Id: FAQ.Dovecot.txt,v 1.3 2016/01/18 18:14:12 gilles Exp gilles $
$Id: FAQ.Dovecot.txt,v 1.4 2016/01/28 14:34:15 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=======================================================================
Imapsync tips for Dovecot. Specific issues and solutions.

34
FAQ.d/FAQ.Duplicates.txt
View file

@ -1,5 +1,7 @@
#!/bin/cat
$Id: FAQ.Duplicates.txt,v 1.6 2016/01/18 18:14:12 gilles Exp gilles $
$Id: FAQ.Duplicates.txt,v 1.9 2016/03/08 21:17:21 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
======================================================================
Imapsync tips about duplicated messages issues.
@ -8,8 +10,8 @@ $Id: FAQ.Duplicates.txt,v 1.6 2016/01/18 18:14:12 gilles Exp gilles $
=======================================================================
Q. How can I remove duplicates in an lone account?
R. Just run imapsync on the same account with option --delete2,
ie with host1 == host2 and user1 == user2
R. Just run imapsync on the same account with option --delete2duplicates,
ie, with host1 == host2 and user1 == user2
=======================================================================
Q: Multiple copies, duplicates, when I run imapsync twice ore more.
@ -29,7 +31,7 @@ Keep in mind it uses a local cache.
imapsync ... --useuid
A big issue with --useuid is that it doesn't generate duplicates if
used from the first time. But it does generate duplicates after a previous
used from the first time but it does generate duplicates after a previous
run without --useuid (because it uses a different method to identify
the messages). A solution? it depends. --delete2 solves this problem
if you are permitted to use it.
@ -53,11 +55,11 @@ want to forward me the output.
A way to avoid this problem is by using option --useheader with
a different set than the default ones used by imapsync.
The default set is like:
The default set is equivalent to:
imapsync ... --useheader "Message-Id" --useheader "Received"
The big problem is that what can be used instead of Message-Id
The problem now is that what can be used instead of Message-Id
and Received lines? Often standalone Message-Id works:
imapsync ... --useheader "Message-Id"
@ -89,5 +91,25 @@ trick to keep the --useheader filtering behaviour is to use
Read it as "skip every header except Message-Id".
=======================================================================
Q. imapsync calculates 479 messages in a folder but only transfers 400
messages. What's happen?
R1. Unless --useuid is used, imapsync considers a header part
of a message to identify a message on both sides.
By default the header part used is lines "Message-Id:" "Message-ID:"
and "Received:" or specific lines depending on --useheader
--skipheader. Whole header can be set by --useheader ALL
Consequences:
1) Duplicate messages on host1 (identical header) are not transferred.
The result is that you can have more messages on host1 than on host2.
R2. With option --useuid imapsync doesn't use headers to identify
messages on both sides but it uses their imap uid identifier.
In that case duplicates on host1 are also transferred on host2.
=======================================================================

15
FAQ.d/FAQ.Exchange.txt
View file

@ -1,5 +1,7 @@
#!/bin/cat
$Id: FAQ.Exchange.txt,v 1.17 2016/01/18 18:14:12 gilles Exp gilles $
$Id: FAQ.Exchange.txt,v 1.20 2016/03/03 15:44:17 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=================================================================================
Imapsync tips for Exchange 20xx and Office365. Specific issues and solutions.
@ -18,6 +20,8 @@ Q. For Office365 I have double and triple checked the username and
Q. Exchange fails with "User is authenticated but not connected".
Q. Exchange fails with "BAD Command received in Invalid state".
Q. From XXX to Exchange 2013 or Office365, read receipts are all
resent again after a sync. Even for old messages. How can I fix that?
@ -148,6 +152,13 @@ http://unix.stackexchange.com/questions/164823/user-is-authenticated-but-not-con
Thanks to James Abbottsmith for this link and explanation at
https://github.com/imapsync/imapsync/issues/32#issuecomment-153561647
=======================================================================
Q. Exchange fails with "BAD Command received in Invalid state".
R. This message might happens when authentication without ssl nor tls.
Add --tls1 or --ssl1 if this error message comes from host1.
Add --tls2 or --ssl2 if this error message comes from host1.
=======================================================================
Q. From XXX to Exchange 2013 or Office365, read receipts are all
resent again after a sync. Even for old messages. How can I fix that?
@ -191,6 +202,8 @@ R. This error message comes from Exchange IMAP server when it
* Some messages are bigger than the size limit. 10 MB by default
on Exchange. It can be upped by configuration for Exchange.
Sometimes a reboot is needed to take configuration changes
into account.
If you can't fix this limit on Exchange then use option
--maxsize 10000000 for 10 MB, change it if needed) to tell
imapsync to skip those messages.

4
FAQ.d/FAQ.Flags.txt
View file

@ -1,5 +1,7 @@
#!/bin/cat
$Id: FAQ.Flags.txt,v 1.6 2016/01/18 18:14:12 gilles Exp gilles $
$Id: FAQ.Flags.txt,v 1.7 2016/01/28 14:34:15 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
===============================
Imapsync tips about flags.

4
FAQ.d/FAQ.Folders_Mapping.txt
View file

@ -1,5 +1,7 @@
#!/bin/cat
$Id: FAQ.Folders_Mapping.txt,v 1.9 2016/01/18 18:14:12 gilles Exp gilles $
$Id: FAQ.Folders_Mapping.txt,v 1.10 2016/01/28 14:34:15 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
===============================================
Imapsync tips about changing folders names.

96
FAQ.d/FAQ.Folders_Selection.txt Normal file
View file

@ -0,0 +1,96 @@
#!/bin/cat
$Id: FAQ.Folders_Selection.txt,v 1.3 2016/02/07 22:01:56 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=====================================
Imapsync tips to select folders.
=====================================
By default, Imapsync syncs all folders, one by one, in alphanumeric order.
======================================================================
Q. How can I sync only one folder?
R. Use --folder option.
imapsync ... --folder MyFolder
If you have more specific folders to sync just add several --folder
imapsync ... --folder MyFolder --folder ThisFolder --folder ThatFolder
======================================================================
Q. What are --subscribe and --subscribed for, and how can they be used?
R. In the IMAP protocol each user can subscribe to one or more folders.
Then one can configure his email software to just see his subscribed
folders. That's an IMAP feature.
Knowing that, the imapsync help mentions:
imapsync --help
...
--subscribed : transfers subscribed folders.
--subscribe : subscribe to the folders transferred on the
host2 that are subscribed on host1.
--subscribe_all : subscribe to the folders transferred on the
host2 even if they are not subscribed on host1.
======================================================================
Q. I want to exclude a folder hierarchy like "public"
R. Use:
--exclude "^public\."
or maybe
--exclude '^"public\.'
In the example given the character "." is the folder separator, you
can omit it. Just take the string as it appears on the imapsync
output line :
From folders list : [INBOX] [public.dreams] [etc.]
======================================================================
Q. I want to exclude only INBOX
R. Use:
imapsync ... --exclude "^INBOX$"
A good way to see what will be done is to first use:
imapsync ... --exclude "^INBOX$" --justfolders --nofoldersizes --dry
======================================================================
Q. I want to exclude folders matching SPAM no matter the case,
aka how to be case insensitive
R. Use:
imapsync ... --exclude "(?i)spam"
A good way to see what will be done is to first use:
imapsync ... --exclude "(?i)spam" --justfolders --nofoldersizes --dry
======================================================================
Q. I want the --folder "MyFolder" option be recursive.
Two solutions:
R1. Use
--folderrec "MyFolder"
R2. Use --include "^MyFolder"
Then the folder "MyFolder" and all its subfolders will be handled
and only them.

24
FAQ.d/FAQ.Gmail.txt
View file

@ -1,5 +1,7 @@
#!/bin/cat
$Id: FAQ.Gmail.txt,v 1.18 2016/01/18 18:14:12 gilles Exp gilles $
$Id: FAQ.Gmail.txt,v 1.21 2016/03/17 20:14:14 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=====================================
Imapsync tips for Gmail accounts.
@ -108,9 +110,7 @@ when selecting this not used folder.
=======================================================================
Q. How to synchronize from XXX to Gmail?
R. There are some details to get the special [Gmail] sub-folders
right. Here's an example of migrating an old "Sent" folder to
Gmail's structure:
R. Use the following example:
imapsync --host1 mail.oldhost.com \
--user1 my_email@oldhost.com \
@ -186,12 +186,6 @@ not accepted by gmail, character ^ to character _ underscore.
--regextrans2 "s/['\"\\\\]/_/g" is optional. It converts
characters ' or " or \ to character _ underscore.
You can add --folder "INBOX.Sent" in the example in case
you want to sync only the "Sent" folder.
You can select folders exported to imap within the gmail preferences.
Select or unselect some "System labels", depending on your needs.
@ -324,7 +318,7 @@ Q. Can not open imap connection on [imap.gmail.com]:
Unable to connect to imap.gmail.com
R0. It looks like this issue is related to ipv6. Both ipv4 and ipv6
protocols should work with gmail and imapsync, I test that regurlarly,
protocols should work with gmail and imapsync, I test that regularly,
imapsync works fine for both ipv4 and ipv6.
If you disable ipv6 then disable also ipv6 resolution!
Or at least, make ipv4 answers be taken before ipv6 since the default
@ -332,10 +326,10 @@ names resolution order is to present ipv6 name resolutions first.
R1. A first simple solution is to use directly gmail ipv4 ip address:
imapsync ... --host1 74.125.133.108
imapsync ... --host1 64.233.184.108
In case it changes, get with any command showing the imap.gmail.com
name resolution:
In case it changes, get it with any command showing the imap.gmail.com
name resolution, try one of those:
nslookup imap.gmail.com
host imap.gmail.com
@ -343,7 +337,7 @@ name resolution:
Or go to http://ping.eu/nslookup/ to get the resolution.
R2. Fix imapsync with the line:
R2. Second solution. Fix imapsync with the line:
use IO::Socket::SSL 'inet4' ;

4
FAQ.d/FAQ.ISP.txt
View file

@ -1,5 +1,7 @@
#!/bin/cat
$Id: FAQ.ISP.txt,v 1.3 2016/01/18 18:14:12 gilles Exp gilles $
$Id: FAQ.ISP.txt,v 1.4 2016/01/28 14:34:15 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=========================================================
Imapsync tips for ISP. Specific issues and solutions.

61
FAQ.d/FAQ.Massive.txt
View file

@ -1,5 +1,7 @@
#!/bin/cat
$Id: FAQ.Massive.txt,v 1.6 2016/01/18 18:14:12 gilles Exp gilles $
$Id: FAQ.Massive.txt,v 1.8 2016/02/07 17:21:40 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
==============================================
Imapsync tips for massive/bulk migrations.
@ -8,10 +10,20 @@ $Id: FAQ.Massive.txt,v 1.6 2016/01/18 18:14:12 gilles Exp gilles $
Questions answered here are:
Q. I need to migrate hundred accounts, how can I do?
Q. I have to migrate 500k users using 400 TB of disk space.
How do I proceed?
Q. How to determine what is the bottleneck in my current imapsync process?
Q. Can I run more instances of imapsync in parallel on a Windows host?
Q. I run multiple imapsync applications at the same time then get a
warning "imapsync.pid already exists, overwriting it".
Is this a potential problem when trying to sync multiple
IMAP account in parallel?
=======================================================================
Q. I need to migrate hundred accounts, how can I do?
@ -123,3 +135,50 @@ If performances increase a lot then host1/link1 is the bottleneck.
If performances increase on both tests 1) and 2), I have no clue to explain that.
Same thing if they both decrease!
=======================================================================
Q. Can I run more instances of imapsync in parallel on a Windows host?
R. Yes!
Q. Any performance issue?
You have to try and check the transfer rates, sum them up to
have a uniq numeric criteria.
There is always a limit, depending on remote imap servers
and the one running imapsync;
CPU, memory, Inputs/Outputs are the classical bottlenecks,
the worst bottleneck is the winner that sets the limit.
examples/sync_loop_windows.bat says
...
REM ==== Parallel executions ====
REM If you want to do parallel runs of imapsync then this current script is a good start.
REM Just copy it several times and replace, on each copy, the csvfile variable value.
REM Instead of SET csvfile=file.txt write for example
REM SET csvfile=file01.txt in the first copy
REM then also
REM SET csvfile=file02.txt in the second copy etc.
REM Of course you also have to split the data contained in file.txt
REM into file01.txt file02.txt etc.
REM After that, just double-click on each batch file to launch each process
=======================================================================
Q. I run multiple imapsync applications at the same time then get a
warning "imapsync.pid already exists, overwriting it".
Is this a potential problem when trying to sync multiple
IMAP account in parallel?
R1. No issue with the file imapsync.pid if you don't use its content
by yourself.
This file can help you to manage multiple runs by sending signals
to the processes (sigterm or sigkill) using their PID.
Each run can have its own pid file with --pidfile option.
The file imapsync.pid contains the PID of the current imapsync process.
This file is removed at the end of a normal run.
You can safely ignore the warning if you don't use imapsync.pid file
to manage imapsync processes.

4
FAQ.d/FAQ.Messages_selection.txt → FAQ.d/FAQ.Messages_Selection.txt
View file

@ -1,5 +1,7 @@
#!/bin/cat
$Id: FAQ.Messages_selection.txt,v 1.3 2016/01/18 18:14:12 gilles Exp gilles $
$Id: FAQ.Messages_Selection.txt,v 1.5 2016/02/05 08:22:26 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=====================================
Imapsync tips to select messages.

4
FAQ.d/FAQ.Oracle-UCS.txt
View file

@ -1,5 +1,7 @@
#!/bin/cat
$Id: FAQ.Oracle-UCS.txt,v 1.2 2016/01/18 18:14:12 gilles Exp gilles $
$Id: FAQ.Oracle-UCS.txt,v 1.3 2016/01/28 14:34:15 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
==================================================
Imapsync tips for Oracle-UCS. Specific issues.

102
FAQ.d/FAQ.Security.txt
View file

@ -1,10 +1,69 @@
#!/bin/cat
# $Id: FAQ.Security.txt,v 1.3 2016/01/18 18:14:12 gilles Exp gilles $
# $Id: FAQ.Security.txt,v 1.6 2016/02/07 21:49:52 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=======================================================
Imapsync tips about security. Issues and solutions.
=======================================================
=======================================================================
Q. Does imapsync support IMAP TLS?
R. Use --tls1 and/or --tls2 options
--tls1 tells imapsync to use tls on host1.
--tls2 tells imapsync to use tls on host2.
=======================================================================
Q. Does imapsync support IMAP over SSL (IMAPS)?
R. Yes natively since release 1.161.
still, 2 ways, at least to use ssl:
a) Use native --ssl1 and/or --ssl2 options
--ssl1 tells imapsync to use ssl on host1.
--ssl2 tells imapsync to use ssl on host2.
b) Use stunnel
http://www.stunnel.org/
Assuming there is an imaps (993) server on imap.foo.org,
on your localhost machine (or bar machine) run :
stunnel -c -d imap -r imap.foo.org:imaps
or using names instead of numbers
stunnel -c -d 143 -r imap.foo.org:993
then use imapsync on localhost (or bar machine) imap (143) port.
If the local port 143 is already taken then use a free one, 10143.
c) Other example for gmail with no root access to open port 143
stunnel -f -P '' -c -d 9993 -r imap.gmail.com:993
Then, to access gmail as host2 use:
imapsync ... --host2 localhost --port2 9993 --nossl2
=======================================================================
Q. How can I manually test a login via ssl?
R. Use ncat or telnet-ssl like in this example:
ncat --ssl -C imap.gmail.com 993
* OK Gimap ready for requests from 78.196.254.58 q1mb175739668wix
a LOGIN "gilles.lamiral@gmail.com" "secret"
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE ... ESEARCH
a OK gilles.lamiral@gmail.com Gilles Lamiral authenticated (Success)
b LOGOUT
* BYE LOGOUT Requested
b OK 73 good day (Success)
The client part to type is "a LOGIN ..." and "b LOGOUT" without
the double-quotes.
======================================================================
Q. Imapsync used to use SSL_VERIFY_PEER now it uses SSL_VERIFY_NONE.
@ -24,5 +83,42 @@ R2. After imapsync 1.673, for example
--sslargs1 SSL_verify_mode=1 \
--sslargs2 SSL_verify_mode=0 \
C1. Don't do this in function "sub set_tls" since it won't work by principle,
tls is done AFTER the application level connexion is established
See perldoc IO::Socket::SSL for all possibilities, also at
http://search.cpan.org/perldoc?IO%3A%3ASocket%3A%3ASSL
C1. Don't do this manual change in function "sub set_tls",
it won't work by principle, because tls is done AFTER
the application level connexion is established.
=======================================================================
Q. How can I manually test a login via ssl?
R. Use ncat or telnet-ssl like in this example:
ncat --ssl -C imap.gmail.com 993
* OK Gimap ready for requests from 78.196.254.58 q1mb175739668wix
a LOGIN "gilles.lamiral@gmail.com" "secret"
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE ... ESEARCH
a OK gilles.lamiral@gmail.com Gilles Lamiral authenticated (Success)
b LOGOUT
* BYE LOGOUT Requested
b OK 73 good day (Success)
The client part to type is "a LOGIN ..." and "b LOGOUT" without
the double-quotes.
=======================================================================
Q: How to have an imaps server?
R.
a) Install one
b) or use stunnel :
Assuming there is an imap (143) server on localhost
stunnel -d 993 -r 143 -f
c) or use stunnel on inetd
imaps stream tcp nowait cyrus /usr/sbin/stunnel -s cyrus -p /etc/ssl/certs/imapd.pem -r localhost:imap2

4
FAQ.d/FAQ.SmarterMail.txt
View file

@ -1,5 +1,7 @@
#!/bin/cat
$Id: FAQ.SmarterMail.txt,v 1.7 2016/01/18 18:14:12 gilles Exp gilles $
$Id: FAQ.SmarterMail.txt,v 1.8 2016/01/28 14:34:15 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
=================================================================
Imapsync tips for SmarterMail. Specific issues and solutions.

4
FAQ.d/FAQ.Various_Server_Softwares.txt
View file

@ -1,5 +1,7 @@
#!/bin/cat
$Id: FAQ.Various_Server_Softwares.txt,v 1.3 2016/01/18 18:14:12 gilles Exp gilles $
$Id: FAQ.Various_Server_Softwares.txt,v 1.4 2016/01/28 14:34:15 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
====================================================
Imapsync tips for various imap server softwares.

4
FAQ.d/FAQ.XOAUTH2.txt
View file

@ -1,5 +1,7 @@
#!/bin/cat
$Id: FAQ.XOAUTH2.txt,v 1.8 2016/01/18 18:14:12 gilles Exp gilles $
$Id: FAQ.XOAUTH2.txt,v 1.9 2016/01/28 14:34:15 gilles Exp gilles $
This documentation is also at http://imapsync.lamiral.info/#doc
======================================================================
Imapsync tips to use XOAUTH2 authentication (Gmail) and old XOAUTH