Modifications to use the new mmlimitrate(3) API
[mmondor.git] / mmsoftware / mmmail / ChangeLog
Content-type: text/html Pulsar-Zone Public Git Repositories - mmondor.git/blame - mmsoftware/mmmail/ChangeLog

500 - Internal Server Error

Malformed UTF-8 character (fatal) at (eval 5) line 1, <$fd> line 1163.
da634739 1$Id: ChangeLog,v 1.32 2003/10/23 01:01:27 mmondor Exp $
5Release: mmmail 0.0.24 devl
6Date : July 11, 2003
7By : Matthew Mondor
9* New features
10 - mailbox-specific statistic counters maintained with mmstat(3) are now
11 unified under mmmail|* rather than separated into mmsmtpd|* and mmpop3d|*.
12 - domain-specific statistic counters are now also maintained which total
13 the number of in/out messages and bytes for all mailboxes of the domain.
14 This is very useful to know how much one can charge for a domain, and/or
15 to generate pie graphics of distribution of bandwidth among the domains.
16 - Added STATFAIL_LOGIN and STATFAIL_PASSWORD configuration file options to
17 mmpop3d(8) which allow to easily detect failed logins.
a7a3986f 18 - Added mmpop3d|total|logins mmstat(3) key
19* Performance enhancements
20 - The configuration file parser was rewritten to be more efficient and
21 to hold cleaner code than the previous one which was migrated from
22 one of my first C programs many years ago :) The configuration files
23 format is now more flexible as well, although the current files will
24 parse properly without modifications still.
25 - mmhash(3) is now used by the command parser for more speed. Any such
26 speed improvement is always welcome when using userspace threads.
27* Bug fixes
28 - mmstatd(8) and the mmstat(3) library had a bugfix.
29 - The message rate sanity checking host-based cache nodes expireing thread
30 would be awakened unnecessarily too fast, considering FLOOD_EXPIRES as a
31 number of seconds rather than as a number of minutes.
32* Other
33 - A minor code cleanup was done
37Release: mmmail 0.0.23 devl
bac02e9e 38Date : July 1, 2003
39By : Matthew Mondor
41* Bug fixes
42 - The IP-address based connection rate and DNS cache expire thread used to
43 sometimes waste CPU cycles looping more often that it should, sleeping for
44 0 seconds between loops. Fixed.
45 - The same was happening to the message rate host-based cache expire thread,
46 which also was fixed.
097ff059 47 - mmstatd(8) had a bugfix and was upgraded to 0.0.8
48* Important
49 - The mmstat(3) key names were modified to be '|' separated rather than
50 '.' separated. Although I have been trying to avoid such a change which
51 obviously requires fixing alot of scripts I am using, it was proven with
52 time that the '.' character was too widely used and that '|' was ideal
53 as a replacement. It is not hard to use a script to use the mmstat(8)
54 reset command and convert all old entries to the new type if necessary.
55 As a result, filenames which comport '.' characters for which counters
56 are maintained are much better to handle and to isolate. The same applies
57 to IP addresses.
58 - The maximum key name length now was bumped from 128 to 256 bytes. As
59 a result, a special log file synchronization will be required as was
60 the case for mmstatd 0.0.7 upgrade (and mmmail 0.0.22 upgrade). This also
61 requires mmstat clients using the mmstat(3) API to be recompiled.
62* New features
63 - Per-mailbox message/bytes statistics are now being maintained via mmstat(3)
64 in mmsmtpd for incomming messages.
65 - Per-user/box logins/message/bytes/delete statistics are now being
66 maintained via mmstat(3) by mmpop3d.
70Release: mmmail 0.0.22 devl
0ba2fea9 71Date : June 19, 2003
72By : Matthew Mondor
74* Important
75 - Important changes to mmstat(3) and mmstatd(8) were made which require a
76 database synchronization procedure to be performed before upgrading.
77 See mmstatd/ChangeLog for details.
78 - The GROUPS directive in mmsmtpd.conf(5), mmpop3d.conf(5) and
79 mmstatd.conf(5) now expect groups to be comma-separated instead of
80 space-separated. The quotes then of course also become optional if
81 multiple groups are specified. This was made because mmreadcfg(3) library
82 is also used by other of my projects for which comma-separated groups
83 were also required.
cddd431b 84* Various
7a56f31f 85 - Fixed ALIGN_CEIL alignment macros which used to always add bytes even
86 if the value was already aligned properly. As a result a few bytes of
87 memory were often wasted.
88 - Interface of various lower layer functions and macros were improved, and
89 mmftpd was migrated to use the new ones.
90 - When an alias points to an invalid address, an error is now logged about
91 it for the administrator to notice and delete the alias.
92 - mmstatd will now perform much better when used with a large number of
93 keys, and will now retain the real time of the operation if it had to
94 be killed before it can incorporate the recovery logs and sync.
95 - mmsmtpd now uses mmhash(3) for it's FLOOD_PROTECTION cache which enhances
96 performance when many hosts are in the cache, and yields cleaner code.
97 As the number of RCPTs allowed per post is generally small, it still uses
98 linked lists for those with the old 64-bit hash for faster comparition,
99 but sequencial searching is still performed when checking against
100 duplicates. This will change if I ever obtain a request about someone who
101 needs to use enough RCPTs in a post to use a hash table for performance.
102 - mmhash(3) is now used by mmserver(3) as well, which should enhance
103 connection validity and DNS hostname cache performance.
104* Bug fixes
105 - The SQL SELECT statement in mmpop3d's auth_pass() function was erroneous
106 and often caused POP3 authentication problems. Fixed.
110Release: mmmail 0.0.21 devl
1a5bbe01 111Date : January 9, 2003
112By : Matthew Mondor
42dd3825 114* Bug fixes (thanks to Jeroen Oostendorp for reporting)
46cf2cd5 115 - HELO would not accept IP addresses. It now will.
116 - The first matching entry in the aliases list would immediately map
117 an address, even if there existed closer matches. The new algorithm
7a2e3b41 118 will only map the address to the best match. (See important changes
119 section for more details).
120 - The RESOLVE_MX_HELO configuration file keyword was changed to RESOLVE_HELO
121 and it's behavior changed accordingly, to resolve hostname's A record
1a5bbe01 122 rather than the MX one. It was erroneous to expect SMTP clients to have
46cf2cd5 123 valid MX records.
124 - The check_nofrom() function, used to check for a match within the
125 nofrom database table, would only match on a hostname basis if resolving
126 client hostnames was enabled and the address could resolve successfully.
127 It now will also always check if the IP address also matches no matter
128 what.
7a2e3b41 129* Important changes
130 - When desired, it is important for frontend scripts to be able to generate
131 required password hashes. This has not been easy with the previous method,
132 which did not even use standard base64 encoding but a custom one on the
133 MD5 result. Calling external binaries such as mmpasswd is not ideal.
134 Moreover, I wanted to get rid of the mhash library dependancy.
135 The solution was simple: password hashes now consist of standard
136 crypt(3) generated strings. Both MD5 and DES modes are supported.
137 However, mmpasswd will only generate MD5 ones with crypt(3).
138 This also allows administrators to easily migrate from real system
139 users, as this is the way system user password hashes are stored.
140 An attempt was made to provide a utility to easily convert old hashes to
141 new ones; This however is impossible due to a bug in the previous method
142 which caused the last few bytes of the MD5 results to be lost during the
143 base64 conversion.
144 A recommendation is to generate random passwords for the users, notify
145 them via email and then activate the new version using the new passwords
146 say, a week afterwards. This password format will then remain unchanged
49769af2 147 for next versions.
47071c2b 148 It is important that people upgrading from 0.0.20 or older execute the
149 provided MySQL script upgrade-0.0.21.sql. Also, the user_passwd field of
150 the user table will need to be updated using the new mmpasswd results for
151 each user password.
152 - The alias table was modified because of the new best matching algorithm
153 which could be slower with large tables. To cope against this, an alias
154 column was added (the upgrade-0.0.21.sql script will also add it). This
155 means that you will have to add the new domains after upgrading, and alter
156 the patterns to no longer hold the domain.
157* Real asynchroneous functions support
158 - The Pth library has limits in that only one process is used for all
159 threads. The various pth_*() functions, and special care has to be taken
160 to prevent locking the whole process when a single thread needs to
161 perform operations, because of the non-preemptive nature of Pth.
162 Another side effect of using Pth is that SMP systems gain no performance
163 over single CPU systems. Moreover, some functions which can take a while
164 can lock the whole process when no pth_*() wrapper function exists to
165 do it. An obvious example is hostname resolving, which can lock the whole
166 process for long periods on slow networks.
167 - A solution was worked out to allow threads to execute real asynchroneous
168 functions without blocking the main process (and therefore allowing other
169 threads to remain responsive), and to even take advantage of
170 multi-processor systems where available.
171 - The technique uses an AmigaOS-like device task/thread which works over the
172 Pth thread-safe message passing mechanism, to serve the various
173 asynchroneous functions like a daemon would. This device internally uses
174 a pool of real asynchroneous processes to which it dispatches the requests
5eb34fba 175 in a distributed manner via unix datagram sockets.
176 - The new ASYNC_PROCESSES configuration file option was added as a result,
177 which allows to specify the number of slave asynchroneous processes to run.
178 - This facility is currently used by mmmail to generate password hashes and
179 to resolve hostnames and MX records (when enabled).
180* Miscelaneous new features
181 - If the daemons are compiled with -DNODETACH, the main process will not
182 fork(). Useful for debugging.
183 - If compiled with -DNODROPPRIVS, daemons will make sure they are not
184 started by the superuser to accept running, but will then not attempt
185 to perform any modifications on the current permissions. Useful for
186 non-privileged users.
187 - mmstatd can now be specified configuration file to use at startup
188 via command-line argument.
189 - mmstat library will now first check for MMSTATCONF environment variable
190 for the configuration file to use instead of the default
191 "/etc/mmstatd.conf". This can be useful to non-privileged users who want
192 to run mmstatd.
193 The default is to only accept to be started by the superuser and then
194 drop privileges.
195 - mmstat user client utility now supports hreport for more human readable
196 results (although less verbose).
197 - Configuration file option DELAY_ON_ERROR was added to allow to pause
198 at every user command error. Thanks to Jeroen Oostendorp for the idea.
199 - mmstat facility now allows wildcard pattern matching for UPDATE, RESET
200 and DELETE operations which operates atomically on all matching keys
201 for that uid.
202 - STATFAIL_EOF was added to mmsmtpd.conf which allows to maintain statistics
203 via the mmstat service about clients which disconnect improperly.
204* Other
205 - Optimized the command matching loop by using fast packed hashes
206 - Other optimizations were performed by moving some variables in
207 closer scope, usually resulting in compilers using registers for
208 appropriate variables. This is especially useful for GCC which
209 ignores register directives.
42dd3825 210 - Significant code cleanups
211 - Several environment variables can now be set to modify the behavior
212 of the script (eg: change installation prefix, default
213 group/user, etc).
214 - The memory pool allocation system was optimized even more, now keeping
215 statistics on the average number of pages in use, so that it scales
216 better. Previously used pages are now privileged as well, which causes
217 unix kernels to allow processes using it to perform more efficiently,
218 as well as the underlaying allocator.
219 - Better synchronization with mmstat service, now using persistant keys
220 and has possibility to delete old temporary keys at startup using
221 wildcard matching.
225Release: mmmail 0.0.20 devl
226Date : November 8, 2002
227By : Matthew Mondor
229* New features
231 STATFAIL_TIMEOUT boolean confguration file options were added, which
232 allow the administrator to view statistics using the mmstat(8) utility,
233 optionally taking action on abnormal functional issues.
234 See mmsmtpd.conf(5) man page for more information.
235 - Added new CHROOT_DIR configuration parameter to optionally allow the
236 server to run enclosed into a chroot(2) jail.
237 - When starting the server it is now possible to specify the configuration
238 file to read. This allows to start several copies using different
239 configurations.
240* Bug fixes
241 - When multiple groups feature was introduced in 0.0.16 devl, mmmail stopped
242 setting the initial group using setgid(2), assuming that setgroups(2)
243 would. This was not the case, and mmmail would then remain part of the
244 wheel group. It will now set the real and effective primary group, the
245 secondary groups, and then the real and effective user. The first group
246 of the GROUPS configuration file parameter is used for the primary group.
247 Although mmmail does not access files, this bug did not hurt it, but
248 let's not leave anything out considering security.
249 - The pop3d needed to replace any "\n.\n" to "\n..\n" while sending out
250 to of course avoid the client to consider the message completed when it
251 is not the case. This is now done. As a result it cannot send the
252 whole message using a single fdbrwrite() call, it has to perform several
253 ones as needed.
254 - Removed the annoying LOOP: debugging which used to flood syslog.
255* Other
256 - mmsmtpd now has less overhead when creating new client contexts
257 - Now uses getnameinfo(3) instead of gethostbyaddr(3) to resolve hostnames
258 if RESOLVE_HOSTS is TRUE. This is more suitable with threads, since
259 gethostbyaddr(3) uses static data and required a mutex to be thread-safe.
263Release: mmmail 0.0.19 devl
264Date : October 28, 2002
265By : Matthew Mondor
267* Important bug fix
268 - omission of an unlinknode() in the new rate limit feature was fixed.
269 The daemons could lock in a loop taking 100% CPU time. Fixed.
273Release: mmmail 0.0.18 devl
274Date : October 26, 2002
275By : Matthew Mondor
277* New features
278 - Anti-DoS connection rate limit feature was added in mmserver library,
279 consequently new CONNECTION_RATE and CONNECTION_PERIOD configuration
280 file options were added. These are on a per-IP address basis.
281 - The bytes statistic counters were bumped to 64-bit.
282* Bug fixes
283 - Yet again a build/install script bug; Permissions would not be applied
284 properly to newly installed files, the problem started to occur when
285 automatic directory creation was added. This was now fixed. Users should
286 consider upgrading as fast as possible, or to make sure that their
287 configuration files in /etc/ have safe permissions.
288 See the following man pages: mmsmtpd.conf(5), mmpop3d.conf(5).
289 The install script will force safe permissions on the configuration files
290 while preserving them if they already exist.
291 - The staff group would not be created by if it did not exist.
292 Thanks to Jeroen Oostendorp for reporting this.
293 - The bandwidth shaping system was not accurate enough, as it used to
294 throttle by 4096 bytes blocks, requireing too much CPU time in syscalls
295 for high limits. It was rewritten using a new design which now only sleeps
296 if required after the maximum number of bytes allowed into a second were
297 transfered. So it now uses a second's worth of data resolution rather than
298 a 4096 bytes resolution. Some parts of mmfd library internals and API
299 were modified as a result (See mmfd(3) man page for details). Another
300 side effect is that it takes far less CPU time with high limits.
301 Thanks to Darren Price for the extensive testing and reports,
302 to Eric Weisenhaus and agrajag for reporting these earlier.
303 - There existed a potential race condition when hostname resolving was
304 enabled, which could cause a memory leak. This was fixed.
305 Thanks to Jeroen Oostendorp for reporting it.
306* Other
307 - When mmmail needs to write out blocks of data it no longer uses mmfd(3)
308 library fdbwrite() buffering call but fdbrwrite() instead to save CPU
309 cycles and avoid buffer size splits.
310 - Some source code auditing and cleaning was made, several optimizations
311 were performed, some buffers were aligned to favorize word-copy/move
312 operations as opposed to single-byte moves, in mmstring library).
313 - I thank everyone who are testing this software, and am again requesting
314 that it be evaluated, as I intend to eventually release mmmail v1 stable.
315 Support for more features will mostly be added into future mmmail devl
316 software, for an eventual mmmail v2. The current design drafts for it
317 can be found at under the mmmail
318 TODO entry. For mmmail v1 release to happen, it has to be free of any
319 issues, it is therefore important to report any bug, to
320 It also would be nice if I could release with it a list of systems it is
321 known to work on, so if betatesters find some time to describe their setup
322 to me via email it would be appreciated. I yet have to test it again on
323 ultrasparc with Linux and Solaris for instance.
327Release: mmmail 0.0.17 devl
328Date : October 11, 2002
329By : Matthew Mondor
331* Bug fixes
332 - The build scripts did not behave as expected on systems which are using
333 bash for /bin/sh. Some script sections needed modifications to work
334 on both. Now seems to build fine on linux systems.
335 - Various new GCC versions warning issues were fixed.
336 - When installing, some directories were assumed to exist, they will
337 now be created when missing.
338 - A bug in mm_memmov() was fixed.
339 - Some systems, including linux, required libresolv to be linked with
340 the daemons. Fixed.
344Release: mmmail 0.0.16 devl
345Date : October 6, 2002
346By : Matthew Mondor
348* New features
349 - Now uses mmstatd(8) for persistant statistics storage and volatile who
350 database. This includes mmstat(8) administration facility.
351 This daemon also comports crash recovery using internal logging.
352 See mmstatd(8) man page for more information.
353 - Processes may now be part of several groups.
354 - It is now possible to set global maximum download and upload limits for
355 the whole server.
356 - Improved installation/upgrade and scripts, and use
357 of /bin/sh rather than make because of GNU/BSD make inconsistencies.
358 - mmmail(8), mmsmtpd(8), mmpop3d(8), mmsmtpd.conf(5), mmpop3d.conf(5),
359 mmpasswd(8), mmstatd(8), mmstat(8), mmstat(3), mmstatd.conf(5)
360 man pages were written.
361* Bug Fixes
362 - Hostname sections starting with alpha characters only used to be accepted,
363 it now will accept alpha-numeric instead for the starting hostname section
364 character.
365 - The "Received:" headers were wrong about the "for" part. The previous
366 way do_data() worked prevented modifying the message header for each
367 RCPT, which was required. do_data() was now rewritten around mmfd(3)
368 fdbreadbuf() and "for" should be adequate as it's a re-implementation.
369 Thanks to Jeroen Oostendorp for reminding me of this bug.
370 - On various systems, including OpenBSD, the NetBSD-style __RCSID and
371 __COPYRIGHT, although useful, caused problems. They were now replaced
372 by custom macros, which should fix compiling issues.
373 Thanks to Dinos Costanti for reporting the issue.
374* Other
375 - The code used to be compatible with older C compilers, it now was converted
376 to ANSI C. Use of __P() macro is no longer made.
377 - mmsmtpd now reads in messages faster than it used to, do_data() was
378 re-written in a way to avoid two previous memory copying passes on the
379 whole message. It now uses mmfd(3)'s new fdbreadbuf() function. Previously
380 it would call fdbgets() on a buffer, copy the string into a linked list,
381 using strdup(3), and after message reception another buffer would be
382 created by copying all lines from the list sequencially again.
383 These two passes can now be avoided, as fdbreadbuf() causes fdbgets() to
384 write it's lines directly into a single buffer. Of course we still can't
385 avoid the buffer copy/translation pass which occurs to produce the mysql
386 query, however.
387 - Some mmlist node manipulation functions were replaced by macros.
388 mmlist(3) now privileges pre-allocated nodes when inserting them back
389 in the free nodes queue, making a better useage of memory.
393Release: mmmail 0.0.15 devl
394Date : July 27, 2002
395By : Matthew Mondor
397* Bug fixes
398 - Fixed bandwidth shaping library, recent versions would totally ignore
399 bandwidth rate specified, because of erroneous 0 initialization.
400 Also fixed a bug where bandwidth would always be lower than requested.
401 Thanks to Eric Weisenhaus, Darren Price and agrajag for reporting these.
402 - Configuration option LOG_LEVEL would not accept values below 1
403 - Fixed mm_strnicmp() which would sometimes cause various problems
404 - Sometimes fdbgets() status would not be logged in case of error,
405 occuring during mail receipt, not disclosing why 520 Error was sent.
406 - The 520 Error sent when an error occurs receiving mail data was changed
407 to a 452 code error.
408 - Added LOG_LEVEL 4 for extreme verbosity (message DATA lines are logged).
412Release: mmmail 0.0.14 devl
413Date : June 4, 2002
414By : Matthew Mondor
416* Bug fixes
417 - The FROM:<> string was compared case-sensitively (thanks to Jeroen
418 Oostendorp)
419 - Various optimizations
420 - A pretty serious bug was fixed (which only affected glibc-based systems),
421 where syslog() would potentially be called with user supplied parts,