zydronium/imapsync
1
0
Fork 0
mirror of https://github.com/imapsync/imapsync.git synced 2025-08-03 07:31:48 +02:00
Code Issues Projects Releases Packages Wiki Activity

1.644

Browse source
This commit is contained in:
Nick Bebout 2015-08-03 20:44:40 -05:00
parent 9ff89e7df2
commit 4a1d71d8fd
53 changed files with 5656 additions and 10351 deletions
Download patch file Download diff file Expand all files Collapse all files

88
FAQ.d/FAQ.Duplicates.txt Normal file
View file

@ -0,0 +1,88 @@
#!/bin/cat
$Id: FAQ.Duplicates.txt,v 1.3 2015/04/02 23:40:08 gilles Exp gilles $
======================================================================
Imapsync and message duplicates issues
======================================================================
=======================================================================
Q. How can I remove duplicates in an lone account?
R. Just run imapsync on the same account with option --delete2,
ie with host1 == host2 and user1 == user2
=======================================================================
Q: Multiple copies, duplicates, when I run imapsync twice ore more.
R0.
Normally and by default, imapsync doesn't generate duplicates.
So if it does generate duplicates it means a problem occurs
with message identification. It happens sometimes with IMAP
servers changing the "Message-Id" line or "Received:" in
the header part of messages.
R1.
You can use option --useuid, with it, imapsync won't use
header lines to compare messages in folders.
Keep in mind it uses a local cache.
imapsync ... --useuid
A big issue with --useuid is that it doesn't generate duplicates if
used from the first time. But it does generate duplicates after a previous
run without --useuid (because it uses a different method to identify
the messages). A solution? it depends. --delete2 solves this problem
if you are permitted to use it.
R2.
Best way if you can follow it.
Multiple copies of the emails on the destination server. Some IMAP
servers (Domino for example) change some headers for each message
transferred. All messages are transferred again and again each time you
run imapsync. This is bad of course. The explanation is that imapsync
considers messages are not the same on each side, default headers used
to identify the messages have changed.
You can look at the headers found by imapsync by using the --debug
option (and search for the message on both part), Header lines from
the source server begin with a "FH:" prefix, Header lines from the
destination server begin with a "TH:" prefix. Since --debug is very
verbose I suggest to isolate a email in a specific folder in case you
want to forward me the output.
A way to avoid this problem is by using option --useheader with
a different set than the default ones used by imapsync.
The default set is like:
imapsync ... --useheader "Message-Id" --useheader "Received"
The big problem is that what can be used instead of Message-Id
and Received lines? Often standalone Message-Id works:
imapsync ... --useheader "Message-Id"
Another good way toward a solution is to isolate two or three messages
in a BUG folder and send me the --debug output by email at
gilles.lamiral@laposte.net
imapsync ... --debug --folder BUG
I will take a close look at the log and modify imapsync to fix
this faulty duplicate behaviour.
Remark. (Trick found by Tomasz Kaczmarski)
Option --useheader "Message-Id" asks the server to send only header
lines beginning with "Message-Id". Some (buggy) servers send the whole
header (all lines) instead of the "Message-Id" line. In that case, a
trick to keep the --useheader filtering behaviour is to use
--skipheader with a negative lookahead pattern:
imapsync ... --skipheader "^(?!Message-Id)"
Read it as "skip every header except Message-Id".
=======================================================================

17
FAQ.d/FAQ.Exchange.txt
View file

@ -1,5 +1,5 @@
#!/bin/cat
$Id: FAQ.Exchange.txt,v 1.2 2015/03/16 15:11:32 gilles Exp gilles $
$Id: FAQ.Exchange.txt,v 1.4 2015/06/03 15:33:48 gilles Exp gilles $
=======================================================================
Exchange 20xx and Office365 specific issues and solutions
@ -16,11 +16,12 @@ R. Here is a command line resume that solves most encountered issues when
On Windows:
imapsync ... ^
imapsync.exe ... ^
--maxsize 10000000 ^
--maxlinelength 9900 ^
--regexflag "s/\\Flagged//g" ^
--disarmreadreceipts
--disarmreadreceipts ^
--maxlinelength 9000
On Unix:
@ -29,7 +30,12 @@ On Unix:
--maxsize 10000000 \
--maxlinelength 9900 \
--regexflag "s/\\Flagged//g" \
--disarmreadreceipts
--disarmreadreceipts \
--maxlinelengthcmd 'reformime -r7'
To get the "reformime" command on Linux install the "maildrop" package
No "reformime" on Windows so for now messages with too long line length
can't be synced to Exchange or Office365.
=======================================================================
@ -197,7 +203,8 @@ So don't use --authmech1 SOMETHING with --authuser1 admin_user,
it will not work.
Same behavior with the --authuser2 option.
See also:
http://www.linux-france.org/prj/imapsync_list/msg02203.html

282
FAQ.d/FAQ.Flags.txt Normal file
View file

@ -0,0 +1,282 @@
#!/bin/cat
$Id: FAQ.Flags.txt,v 1.3 2015/04/03 21:05:11 gilles Exp gilles $
======================================================================
Imapsync and flags
======================================================================
Questions answered here are:
Q. How to debug flag issues?
Q. Is there a way to only sync messages with a specific flag set,
for example, the \Seen flag?
Q. How to convert flags?
Q. Does imapsync retain the \Answered and $Forwarded flags?
Q. How to fix this error: BAD Invalid system flag \FORWARDED
Q. How to convert flags with $ to \ character?
Q. imapsync fails with the following error:
flags from : [\Seen NonJunk]
Error trying to append string: 58 NO APPEND Invalid flag list
Q. Flags are not well synchronized. Is it a bug?
=======================================================================
Q. How to debug flag issues?
R. Use --debugflags
imapsync ... --debugflags
=======================================================================
Q. Is there a way to only sync messages with a specific flag set,
for example, the \Seen flag?
R. use --search
imapsync ... --search SEEN
or
imapsync ... --search UNSEEN
or ...
The complete list of search things are listed below
http://www.faqs.org/rfcs/rfc3501.html
6.4.4. SEARCH Command
...
ALL
All messages in the mailbox; the default initial key for
ANDing.
ANSWERED
Messages with the \Answered flag set.
BCC <string>
Messages that contain the specified string in the envelope
structure's BCC field.
BEFORE <date>
Messages whose internal date (disregarding time and timezone)
is earlier than the specified date.
BODY <string>
Messages that contain the specified string in the body of the
message.
CC <string>
Messages that contain the specified string in the envelope
structure's CC field.
DELETED
Messages with the \Deleted flag set.
DRAFT
Messages with the \Draft flag set.
FLAGGED
Messages with the \Flagged flag set.
FROM <string>
Messages that contain the specified string in the envelope
structure's FROM field.
HEADER <field-name> <string>
Messages that have a header with the specified field-name (as
defined in [RFC-2822]) and that contains the specified string
in the text of the header (what comes after the colon). If the
string to search is zero-length, this matches all messages that
have a header line with the specified field-name regardless of
the contents.
KEYWORD <flag>
Messages with the specified keyword flag set.
LARGER <n>
Messages with an [RFC-2822] size larger than the specified
number of octets.
NEW
Messages that have the \Recent flag set but not the \Seen flag.
This is functionally equivalent to "(RECENT UNSEEN)".
NOT <search-key>
Messages that do not match the specified search key.
OLD
Messages that do not have the \Recent flag set. This is
functionally equivalent to "NOT RECENT" (as opposed to "NOT
NEW").
ON <date>
Messages whose internal date (disregarding time and timezone)
is within the specified date.
OR <search-key1> <search-key2>
Messages that match either search key.
RECENT
Messages that have the \Recent flag set.
SEEN
Messages that have the \Seen flag set.
SENTBEFORE <date>
Messages whose [RFC-2822] Date: header (disregarding time and
timezone) is earlier than the specified date.
SENTON <date>
Messages whose [RFC-2822] Date: header (disregarding time and
timezone) is within the specified date.
SENTSINCE <date>
Messages whose [RFC-2822] Date: header (disregarding time and
timezone) is within or later than the specified date.
SINCE <date>
Messages whose internal date (disregarding time and timezone)
is within or later than the specified date.
SMALLER <n>
Messages with an [RFC-2822] size smaller than the specified
number of octets.
SUBJECT <string>
Messages that contain the specified string in the envelope
structure's SUBJECT field.
TEXT <string>
Messages that contain the specified string in the header or
body of the message.
TO <string>
Messages that contain the specified string in the envelope
structure's TO field.
UID <sequence set>
Messages with unique identifiers corresponding to the specified
unique identifier set. Sequence set ranges are permitted.
UNANSWERED
Messages that do not have the \Answered flag set.
UNDELETED
Messages that do not have the \Deleted flag set.
UNDRAFT
Messages that do not have the \Draft flag set.
UNFLAGGED
Messages that do not have the \Flagged flag set.
UNKEYWORD <flag>
Messages that do not have the specified keyword flag set.
UNSEEN
Messages that do not have the \Seen flag set.
=======================================================================
Q. How to convert flags?
R. use --regexflag
For example to convert flag IMPORTANT to flag CANWAIT
imapsync ... --regexflag "s/IMPORTANT/CANWAIT/g" --debugflags
option --debugflags is usefull to see in details what imapsync
does with flags.
=======================================================================
Q. Does imapsync retain the \Answered and $Forwarded flags?
R. It depends on the destination server.
a) If the destination server honours the "PERMAENTFLAGS \*"
directive (meaning it accepts any flag) or no PERMAENTFLAGS at all
then imapsync synchronizes all flags except the \Recent flag
(RFC 3501 says about \Recent flag "This flag can not be
altered by the client.").
b) If the destination server honours the "PERMAENTFLAGS without the
special "\*" then imapsync synchronizes only the flags listed
in PERMANENTFLAGS.
Some imap servers have problems with flags not beginning with
the backslash character \
(see next question to find a solution to this issue)
=======================================================================
Q. How to fix this error: BAD Invalid system flag \FORWARDED
R. Filter flag \FORWARDED with --regexflag like this:
On Windows:
imapsync ... --regexflag "s/\\FORWARDED//g"
On Unix:
imapsync ... --regexflag 's/\\FORWARDED//g'
or
imapsync ... --regexflag "s/\\\\FORWARDED//g"
=======================================================================
Q. How to convert flags with $ to \ character?
R. $ and \ are special characters we have to "escape" them.
For example to convert flag $label1 to \label1
imapsync ... --regexflag "s/\$label1/\\label1/g" --debugflags
======================================================================
Q. imapsync fails with the following error:
flags from : [\Seen NonJunk]
Error trying to append string: 58 NO APPEND Invalid flag list
R. For some servers, flags have to begin with a \ character.
The flag "NonJunk" may be a invalid flag for your server
so use for example:
imapsync ... --regexflag "s/NonJunk//g"
Remark (thanks to Arnt Gulbrandsen):
IMAP system flags have to begin with \ character.
Any other flag must begin with another character.
System flags are just flags defined by an RFC instead of by users.
Conclusion, some imap server coders don't read the RFCs (so do I).
Recent imapsync deals with this issue by filter with PERMANENTFLAGS
automatically.
=======================================================================
Q. Flags are not well synchronized. Is it a bug?
R. It happens with some servers on the first sync.
Also, it was a bug from revision 1.200 to revision 1.207
Two solutions:
* Run imapsync a second time. imapsync synchronizes flags on each run.
* Use option --syncflagsaftercopy. With this option imapsync will
also sync flags after each message transfer. Flags are already
synced during the transfer with the imap APPEND command but
option --syncflagsaftercopy does it again using the imap STORE
command.
=======================================================================

221
FAQ.d/FAQ.Folders_Mapping.txt Normal file
View file

@ -0,0 +1,221 @@
#!/bin/cat
$Id: FAQ.Folders_Mapping.txt,v 1.5 2015/05/11 10:36:33 gilles Exp gilles $
===========================================
Imapsync changing folders names
===========================================
Things to know and understand before playing with --regextrans2
*) --regextrans2 is used to transform folders names.
*) --regextrans2 applies after the default
inversion prefix1 <-> prefix2 and sep1 <-> sep2
So when elaborating the regex you should focus on
the right part of the default mapping printed by
imapsync, the part showing the host2 folder name.
The section to look at is within the folder loop:
++++ Looping on each folder
Here
++++ End looping on each folder
*) Several --regextrans2 is possible, they will be applied in the order
of the command line, each one on the result of the previous one.
*) --regextrans2 uses Perl regex mechanism so it may be hard to master
this part. It is powerful but not very simple.
*) Windows vs Unix quotes.
On windows don't use single quotes ' around the regex string,
use double quotes instead, like --regextrans2 "myregex"
On Linux/Unix use single quotes ' around the regex string, it is
easier to get what we want with single quotes since the shell
won't change the inner string. Like --regextrans2 'myregex'
*) Good method to elaborate any --regextrans2 string
First elaborate the --regextrans2 string with --dry --justfolders
--nofoldersizes options.
With --dry imapsync shows the transformations it will do without
really doing them.
With --justfolders imapsync will work only with folders,
messages won't be taken into account.
With --nofoldersizes imapsync won't spend time useless time on
evaluating folders sizes.
When the output shows what you escape imapsync to do with folders
names, you can remove the --dry option. Keep the --justfolders
option in order to see if the destination server host2 accepts
to create the folders.
When everything is ok with folders you can remove --justfolders
and --nofoldersizes imapsync will also transfer messages.
Showing folders sizes is good then transferring messages, it allows
ETA calculation and it's a supplementary check on folders.
=======================================================================
Q. Give examples about --regextrans2
Examples:
1) To remove INBOX. in the name of destination folders:
--regextrans2 's/^INBOX\.(.+)/$1/'
2a) To sync all folders to INBOX:
imapsync ... --regextrans2 "s/.*/INBOX/"
2b) To sync a complete account in a subfolder called FOO:
Since imapsync release 1.641 simply use:
imapsync ... --subfolder2 FOO
Next examples are subfolder solutions for any release.
a) Separator is dot character "." and "INBOX" prefixes every folder
On Linux/Unix:
--regextrans2 's,^INBOX(.*),INBOX.FOO$1,'
On Windows:
--regextrans2 "s,^INBOX(.*),INBOX.FOO$1,"
or:
b) Separator is the slash character "/" and there is no prefix
On Linux/Unix:
--regextrans2 's,(.*),FOO/$1,'
On Windows:
--regextrans2 "s,(.*),FOO/$1,"
or:
c) Any separator, any prefix solution, FOO is the subfolder:
It is a complicated line because every case is taken into account.
Type it in one line (or with the \ at the end of first line on Unix shells.
On Linux/Unix:
--regextrans2 's,${h2_prefix}(.*),${h2_prefix}FOO${h2_sep}$1,' \
--regextrans2 's,^INBOX$,${h2_prefix}FOO${h2_sep}INBOX,'
On Windows:
--regextrans2 "s,${h2_prefix}(.*),${h2_prefix}FOO${h2_sep}$1," ^
--regextrans2 "s,^INBOX$,${h2_prefix}FOO${h2_sep}INBOX,"
3) to substitute all characters dot "." by underscores "_"
--regextrans2 "s,\.,_,g"
3b) to substitute all doublequotes " by underscores _
On Linux/Unix:
--regextrans2 's,\",_,g'
On Windows:
--regextrans2 s,\^",_,g
4) to change folder names like this:
[mail/Sent Items] -> [Sent]
[mail/Test] -> [INBOX/Test]
[mail/Test2] -> [INBOX/Test2]
On Linux/Unix:
--regextrans2 's,^mail/Sent Items$,Sent,' \
--regextrans2 's,^mail/,INBOX/,'
======================================================================
Q. Is it possible to synchronize all messages from one server to
another without recreating the folder structure and the target server.
R. Yes.
For example, to synchronize all messages in all folders on host1
to folder INBOX only on host2:
1) First try (safe mode):
--regextrans2 "s/(.*)/INBOX/" --dry --justfolders
2) See if the output says everything you want imapsync to do,
--dry option is safe and does nothing real.
3) Remove --dry
Check the imap folder tree on the target side, you should
only have one: the classical INBOX.
4) Remove --justfolders
======================================================================
Q. I have moved from Braunschweig to Graz, so I would like to have my
whole Braunschweig mail sorted into a sub-folder INBOX.Braunschweig
of my new mail account.
R.
1) First try (safe mode):
imapsync \
...
--regextrans2 "s/INBOX(.*)/INBOX.Braunschweig\$1/" \
--dry --justfolders
On Windows, in the previous example containing \$1 you have to
replace the two \$1 by $1 (remove the \ before $).
2) See if the output says everything you want imapsync to do,
--dry option is safe and does nothing real.
3) Remove --dry
Check the imap folder tree on the target side
4) Remove --justfolders
=======================================================================
Q. I would like to move emails from InBox to a sub-folder called,
say "2010-INBOX" based on the date (Like all emails received in the
Year 2010 should be moved to the folder called "2010-INBOX").
R. 2 ways :
a) Manually:
------------
1) You create a folder INBOX.2010-INBOX
2) Mostly every email software allow sorting by date. In INBOX, you
select from 1 January to 31 December 2010 messages with the shift key.
(in mutt, use ~d)
3) Cut/paste in INBOX.2010-INBOX
b) With imapsync:
-----------------
imapsync ... \
--search 'SENTSINCE 1-Jan-2010 SENTBEFORE 31-Dec-2010'
--regextrans2 's/^INBOX$/INBOX.2010-INBOX/' \
--folder INBOX
=======================================================================

25
FAQ.d/FAQ.Gmail.txt
View file

@ -1,5 +1,5 @@
#!/bin/cat
$Id: FAQ.Gmail.txt,v 1.1 2015/03/26 08:12:20 gilles Exp gilles $
$Id: FAQ.Gmail.txt,v 1.2 2015/05/11 01:11:40 gilles Exp gilles $
======================================================================
Imapsync with Gmail
@ -71,11 +71,12 @@ and trailing blank characters are problems with gmail, and
also successive blanks ending with the IMAP error
"NO [CANNOT] Folder contains excess whitespace (Failure)"
If you want to change only leading and trailing blank characters
then use instead:
--regextrans2 "s,(/|^) +,\$1,g" --regextrans2 "s, +(/|$),\$1,g"
then use the following instead
On Linux/Unix:
--regextrans2 "s,(/|^) +,\$1,g" --regextrans2 "s, +(/|$),\$1,g"
On Windows:
--regextrans2 "s,(/|^) +,$1,g" --regextrans2 "s, +(/|$),$1,g"
On Windows, in the previous example containing \$1 you have to
replace the two \$1 by $1 (remove the \ before $1).
--regextrans2 "s/[\^]/_/g" is mandatory. It converts, since
not accepted by gmail, character ^ to character _ underscore.
@ -185,6 +186,20 @@ label CanWait and only it.
--skipcrossduplicates, will only put in "[Gmail]/All Mail"
the messages that are not labeled at all.
=======================================================================
Q. I can't authenticate with Gmail via IMAP
Gmail says "Please log in via your web browser"
R1. See Coert Grobbelaar solution:
https://security.stackexchange.com/questions/86404/how-do-i-interact-with-google-to-import-email-via-imapsync
R2. I had the same issue one time (mars 2015) logging to Gmail
with imapsync.
The Gmail imap message error said "Please log in via your web browser"
so I logged for this account via a web browser,
it asked me to receive a code via a mobile, I said yes,
I entered the code and everything went ok.
=======================================================================
Q. Gmail does not really delete messages in folder [Gmail]/All Mail
What happens? What can I do?