Thread: "End-to-end" paper
For those of you who have missed it, here http://www.google.com/search?q=cache:web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf+clark+end+to+end&hl=en is the paper some of us mention, "END-TO-END ARGUMENTS IN SYSTEM DESIGN" by Saltzer, Reed, and Clark. The abstract is: This paper presents a design principle that helps guide placement of functions among the modules of a distributed computersystem. The principle, called the end-to-end argument, suggests that functions placed at low levels of a systemmay be redundant or of little value when compared with the cost of providing them at that low level. Examples discussedin the paper include bit error recovery, security using encryption, duplicate message suppression, recovery from system crashes, and delivery acknowledgement. Low level mechanisms to support these functions are justified onlyas performance enhancements. It was written in 1981 and is undiminished by the subsequent decades. Nathan Myers ncm@zembu.com
At 12:24 AM 17-05-2001 -0700, Nathan Myers wrote: > >For those of you who have missed it, here > >http://www.google.com/search?q=cache:web.mit.edu/Saltzer/www/publications/e ndtoend/endtoend.pdf+clark+end+to+end&hl=en > >is the paper some of us mention, "END-TO-END ARGUMENTS IN SYSTEM DESIGN" >by Saltzer, Reed, and Clark. > >The abstract is: > > This paper presents a design principle that helps guide placement of > functions among the modules of a distributed computer system. The > principle, called the end-to-end argument, suggests that functions > placed at low levels of a system may be redundant or of little value > when compared with the cost of providing them at that low level. > Examples discussed in the paper include bit error recovery, security > using encryption, duplicate message suppression, recovery from > system crashes, and delivery acknowledgement. Low level mechanisms > to support these functions are justified only as performance > enhancements. > >It was written in 1981 and is undiminished by the subsequent decades. > Maybe I don't understand the paper. The end-to-end argument might be true if taking the monolithic approach. I find more useful ideas gleaned from the RFCs, TCP/IP and the OSI 7 layer model: modularity, "useful standard interfaces", "Be liberal in what you accept, and conservative in what you send" and so on. Within a module I figure the end to end argument might hold, but the author keeps talking about networks and networking. SSL and TCP are useful. The various CRC checks down the IP stack to the datalink layer have their uses too. By splitting stuff up at appropriate points, adding or substituting objects at various layers becomes so much easier. People can download Postgresql over token ring, Gigabit ethernet, X.25 and so on. Splitting stuff up does mean that the bits and pieces now do have a certain responsibility. If those responsibilities involve some redundancies in error checking or encryption or whatever, so be it, because if done well people can use those bits and pieces in interesting ways never dreamed of initially. For example SSL over TCP over IPSEC over encrypted WAP works (even though IPSEC is way too complicated :)). There's so much redundancy there, but at the same time it's not a far fetched scenario - just someone ordering online on a notebook pc. But if a low level module never bothered with error correction/detection/handling or whatever and was optimized for an application specific purpose, it's harder to use it for other purposes. And if you do, some chap could post an article to Bugtraq on it, mentioning exploit, DoS or buffer overflow. Cheerio, Link.
On Thu, May 17, 2001 at 06:04:54PM +0800, Lincoln Yeoh wrote: > At 12:24 AM 17-05-2001 -0700, Nathan Myers wrote: > > > >For those of you who have missed it, here > > > >http://www.google.com/search?q=cache:web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf+clark+end+to+end&hl=en > > > >is the paper some of us mention, "END-TO-END ARGUMENTS IN SYSTEM DESIGN" > >by Saltzer, Reed, and Clark. > > > >The abstract is: > > > > This paper presents a design principle that helps guide placement > > of functions among the modules of a distributed computer system. > > The principle, called the end-to-end argument, suggests that > > functions placed at low levels of a system may be redundant or > > of little value when compared with the cost of providing them > > at that low level. Examples discussed in the paper include > > bit error recovery, security using encryption, duplicate > > message suppression, recovery from system crashes, and delivery > > acknowledgement. Low level mechanisms to support these functions > > are justified only as performance enhancements. > > > >It was written in 1981 and is undiminished by the subsequent decades. > > Maybe I don't understand the paper. Yes. It bears re-reading. > The end-to-end argument might be true if taking the monolithic > approach. I find more useful ideas gleaned from the RFCs, TCP/IP and > the OSI 7 layer model: modularity, "useful standard interfaces", "Be > liberal in what you accept, and conservative in what you send" and so > on. The end-to-end principle has had profound effects on the design of Internet protocols, perhaps most importantly in keeping them simpler than OSI's. > Within a module I figure the end to end argument might hold, The end-to-end principle isn't particularly applicable within a module. It's a system-design principle. Its prescription for individual modules is: don't imagine that anybody else gets much value from your complex error recovery shenanigans; they have to do their own error recovery anyway. You provide more value by making a good effort. > but the author keeps talking about networks and networking. Of course networking is just an example, but it's a particularly good example. Data storage (e.g. disk) is another good example; in the context of the paper it may be thought of as a mechanism for communicating with other (later) times. The point there is that the CRCs and ECC performed by the disk are not sufficient to ensure reliability for the system (e.g. database service); for that, end-to-end measures such as hot-failover, backups, redo logs, and block- or record-level CRCs are needed. The purpose of the disk CRCs is not reliability, a job they cannot do alone, but performance: they help make the need to use the backups and redo logs infrequent enough to be tolerable. > SSL and TCP are useful. The various CRC checks down the IP stack to > the datalink layer have their uses too. Yes, of course they are useful. The authors say so in the paper, and they say precisely how (and how not). > By splitting stuff up at appropriate points, adding or substituting > objects at various layers becomes so much easier. People can download > Postgresql over token ring, Gigabit ethernet, X.25 and so on. As noted in the paper, the principle is most useful in helping to decide what goes in each layer. > Splitting stuff up does mean that the bits and pieces now do have > a certain responsibility. If those responsibilities involve some > redundancies in error checking or encryption or whatever, so be > it, because if done well people can use those bits and pieces in > interesting ways never dreamed of initially. > > For example SSL over TCP over IPSEC over encrypted WAP works (even > though IPSEC is way too complicated :)). There's so much redundancy > there, but at the same time it's not a far fetched scenario - just > someone ordering online on a notebook pc. The authors quote a similar example in the paper, even though it was written twenty years ago. > But if a low level module never bothered with error > correction/detection/handling or whatever and was optimized for > an application specific purpose, it's harder to use it for other > purposes. And if you do, some chap could post an article to Bugtraq on > it, mentioning exploit, DoS or buffer overflow. The point is that leaving that stuff _out_ is how you keep low-level mechanisms useful for a variety of purposes. Putting in complicated error-recovery stuff might suit it better for a particular application, but make it less suitable for others. This is why, at the IP layer, packets get tossed at the first sign of congestion. It's why TCP connections often get dropped at the first sign of a data-format violation. This is a very deep principle; understanding it thoroughly will make you a much better system designer. Nathan Myers ncm@zembu.com
On OBSD from cvs source, clean checkout: gcc -O2 -pipe -Wall -Wmissing-prototypes -Wmissing-declarations -I../../../../src/include -DLIBDIR=\"/usr/local/pgsql/lib\ " -DDLSUFFIX=\".so\" -c -o dfmgr.o dfmgr.c dfmgr.c: In function `load_external_function': dfmgr.c:118: `RTLD_GLOBAL' undeclared (first use in this function) dfmgr.c:118: (Each undeclared identifier is reported only once dfmgr.c:118: for each function it appears in.) gmake[4]: *** [dfmgr.o] Error 1 gmake[4]: Leaving directory `/home/bpalmer/APPS/pgsql/src/backend/utils/fmgr' ?? RTLD_GLOBAL problems? - b b. palmer, bpalmer@crimelabs.net pgp: www.crimelabs.net/bpalmer.pgp5
This seems to have been broken for a few days ~monday night - brandon b. palmer, bpalmer@crimelabs.net pgp: www.crimelabs.net/bpalmer.pgp5
On Sat, May 19, 2001 at 08:03:50PM -0400, bpalmer wrote: > On OBSD from cvs source, clean checkout: > > gcc -O2 -pipe -Wall -Wmissing-prototypes -Wmissing-declarations > -I../../../../src/include -DLIBDIR=\"/usr/local/pgsql/lib\ > " -DDLSUFFIX=\".so\" -c -o dfmgr.o dfmgr.c > dfmgr.c: In function `load_external_function': > dfmgr.c:118: `RTLD_GLOBAL' undeclared (first use in this function) > dfmgr.c:118: (Each undeclared identifier is reported only once > dfmgr.c:118: for each function it appears in.) > gmake[4]: *** [dfmgr.o] Error 1 > gmake[4]: Leaving directory > `/home/bpalmer/APPS/pgsql/src/backend/utils/fmgr' > > > ?? RTLD_GLOBAL problems? Not a solution, but a few data points: I had a successful build from cvs of May 19 16:21 GMT under NetBSD/i386, and for me RTLD_GLOBAL is defined in /usr/include/dlfcn.h ie., system header file, not postgresql. Hope that helps, Patrick
This was added by Peter E to allow PL/Perl to compile. > On Sat, May 19, 2001 at 08:03:50PM -0400, bpalmer wrote: > > On OBSD from cvs source, clean checkout: > > > > gcc -O2 -pipe -Wall -Wmissing-prototypes -Wmissing-declarations > > -I../../../../src/include -DLIBDIR=\"/usr/local/pgsql/lib\ > > " -DDLSUFFIX=\".so\" -c -o dfmgr.o dfmgr.c > > dfmgr.c: In function `load_external_function': > > dfmgr.c:118: `RTLD_GLOBAL' undeclared (first use in this function) > > dfmgr.c:118: (Each undeclared identifier is reported only once > > dfmgr.c:118: for each function it appears in.) > > gmake[4]: *** [dfmgr.o] Error 1 > > gmake[4]: Leaving directory > > `/home/bpalmer/APPS/pgsql/src/backend/utils/fmgr' > > > > > > ?? RTLD_GLOBAL problems? > > Not a solution, but a few data points: I had a successful build from cvs of > May 19 16:21 GMT under NetBSD/i386, and for me RTLD_GLOBAL is defined in > /usr/include/dlfcn.h ie., system header file, not postgresql. > > Hope that helps, > > Patrick > > ---------------------------(end of broadcast)--------------------------- > TIP 4: Don't 'kill -9' the postmaster > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026