Thread: SQL query never ends
I'm experimenting a random problem on my PC. Sometimes a sql query executed with Statement never returns. The code is blocked at a line like this one: rs = stmt.executeQuery("select true"); The query is not important: it fails with any type of query, at random, even with the one above. I never get an exception. Now the logs. I've enable logging at the database level and I have set "statement_timeout = 3000". I have searched for the query in postgresql's log but I cannot find the query which is blocking in my java code so I guess that it means that the query never reached postgresql. I've enabled jdbc logging with "loglevel=2" and this is what I get: simple execute, handler=org.postgresql.jdbc2.AbstractJdbc2Statement$StatementResultHandler@996cca, maxRows=0, fetchSize=0, flags=17 FE=> Parse(stmt=null,query="i've remove the select query",oids={}) FE=> Bind(stmt=null,portal=null) FE=> Describe(portal=null) FE=> Execute(portal=null,limit=0) FE=> Sync .. and nothing else. No "<=BE ..." follow the " FE=> Sync" log entry. I've also tried to put "stmt.setQueryTimeout(3);" but it doesn't seem to do anything. I'm using postgresql 8.1.4 on Windows 2000 Pro SP4 and the driver is "8.1-407 JDBC 3". I'm also using java 1.5.0_06-b05. I guess that there is a bug in the driver. If I'm wrong, please tell me how to solve it. If I'm right, could you try to fix it? _________________________________________________________________ Log p� MSN Messenger direkte p� nettet: http://webmessenger.msn.com
This sort of behavior often indicates some kind of unsafe concurrent usage of the JDBC Connection. Is this running in a multi-threaded environment? If so, are there measures in place to ensure that only one thread uses the connection at a time? It would also be useful if you could do a thread dump while it's hanging and send a stack trace of the affected thread. -- Mark On Wed, 2006-08-09 at 10:21 +0000, James Im wrote: > I'm experimenting a random problem on my PC. Sometimes a sql query > executed with Statement never returns. The code is blocked at a line > like this one: > > rs = stmt.executeQuery("select true"); > > The query is not important: it fails with any type of query, at random, > even with the one above. > > I never get an exception. > > Now the logs. I've enable logging at the database level and I have set > "statement_timeout = 3000". I have searched for the query in > postgresql's log but I cannot find the query which is blocking in my > java code so I guess that it means that the query never reached postgresql. > > I've enabled jdbc logging with "loglevel=2" and this is what I get: > > simple execute, > handler=org.postgresql.jdbc2.AbstractJdbc2Statement$StatementResultHandler@996cca, > maxRows=0, fetchSize=0, flags=17 > FE=> Parse(stmt=null,query="i've remove the select query",oids={}) > FE=> Bind(stmt=null,portal=null) > FE=> Describe(portal=null) > FE=> Execute(portal=null,limit=0) > FE=> Sync > > .. and nothing else. No "<=BE ..." follow the " FE=> Sync" log entry. > > > I've also tried to put "stmt.setQueryTimeout(3);" but it doesn't seem to > do anything. > > I'm using postgresql 8.1.4 on Windows 2000 Pro SP4 and the driver is > "8.1-407 JDBC 3". I'm also using java 1.5.0_06-b05. > > > I guess that there is a bug in the driver. If I'm wrong, please tell me > how to solve it. If I'm right, could you try to fix it? > > _________________________________________________________________ > Log p MSN Messenger direkte p nettet: http://webmessenger.msn.com > > > ---------------------------(end of broadcast)--------------------------- > TIP 1: if posting/reading through Usenet, please send an appropriate > subscribe-nomail command to majordomo@postgresql.org so that your > message can get through to the mailing list cleanly
James, We will need more information in order to help you here. As Mark said, are you using this concurrently ? Dave On 9-Aug-06, at 6:21 AM, James Im wrote: > I'm experimenting a random problem on my PC. Sometimes a sql query > executed with Statement never returns. The code is blocked at a line > like this one: > > rs = stmt.executeQuery("select true"); > > The query is not important: it fails with any type of query, at > random, > even with the one above. > > I never get an exception. > > Now the logs. I've enable logging at the database level and I have set > "statement_timeout = 3000". I have searched for the query in > postgresql's log but I cannot find the query which is blocking in my > java code so I guess that it means that the query never reached > postgresql. > > I've enabled jdbc logging with "loglevel=2" and this is what I get: > > simple execute, > handler=org.postgresql.jdbc2.AbstractJdbc2Statement > $StatementResultHandler@996cca, > maxRows=0, fetchSize=0, flags=17 > FE=> Parse(stmt=null,query="i've remove the select query",oids={}) > FE=> Bind(stmt=null,portal=null) > FE=> Describe(portal=null) > FE=> Execute(portal=null,limit=0) > FE=> Sync > > .. and nothing else. No "<=BE ..." follow the " FE=> Sync" log entry. > > > I've also tried to put "stmt.setQueryTimeout(3);" but it doesn't > seem to > do anything. > > I'm using postgresql 8.1.4 on Windows 2000 Pro SP4 and the driver is > "8.1-407 JDBC 3". I'm also using java 1.5.0_06-b05. > > > I guess that there is a bug in the driver. If I'm wrong, please > tell me > how to solve it. If I'm right, could you try to fix it? > > _________________________________________________________________ > Log på MSN Messenger direkte på nettet: http://webmessenger.msn.com > > > ---------------------------(end of > broadcast)--------------------------- > TIP 1: if posting/reading through Usenet, please send an appropriate > subscribe-nomail command to majordomo@postgresql.org so that > your > message can get through to the mailing list cleanly >
>Is this running in a multi-threaded environment? Yes it is. >If so, are there measures in place to ensure that only one thread uses the connection at a time? I'm using a thread pool so normally one thread takes one connection out of the pool and operates on it. From what I see the borrowObject() appears to contains a synchronized block to that 2 threads could not take a same connection at the same time. However, just to be sure, I added synchronization to the methods of the GenericObjectPool (you can see the -locked below). If it ever was a synchronization problem I think that now there is none. >It would also be useful if you could do a thread dump while it's hanging >and send a stack trace of the affected thread. Here it is. This time, the sql statement that blocked is "select true" which I do before returning the connection from the pool to make sure that the connection is ok. None of the other threads where in the org.postgresql package when I did the thread dump. "Thread-11" prio=6 tid=0x25499dd0 nid=0x868 runnable [0x2801f000..0x2801fa64] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:235) - locked <0x115fcf88> (a java.io.BufferedInputStream) at org.postgresql.core.PGStream.ReceiveIntegerR(PGStream.java:275) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1171) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:188) - locked <0x115f8210> (a org.postgresql.core.v3.QueryExecutorImpl) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:452) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:340) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:239) at services.jdbc.StatementWrapper.executeQuery(Unknown Source) at services.jdbc.ConnectionWrapper.validateObject(Unknown Source) at services.pool.GenericObjectPool.testObject(Unknown Source) at services.pool.GenericObjectPool.borrowObject(Unknown Source) - locked <0x115875c0> (a services.pool.GenericObjectPool) at services.jdbc.DataSource.getConnection(Unknown Source) at services.jdbc.GlobalJdbc.getConnection(Unknown Source) at common.BuildTreeListingCommon.rebuildListing(Unknown Source) at common.BuildTreeListingCommon.rebuildListing(Unknown Source) at client.BuildTreeListing.getPage(Unknown Source) at common.WebDispatcher.start(Unknown Source) at java.lang.Thread.run(Thread.java:595) _________________________________________________________________ Download din yndlingsmusik p� MSN Music: http://www.msn.dk/music - det er nemt og billigt
James Im wrote: > I'm experimenting a random problem on my PC. Sometimes a sql query > executed with Statement never returns. > simple execute, > handler=org.postgresql.jdbc2.AbstractJdbc2Statement$StatementResultHandler@996cca, > > maxRows=0, fetchSize=0, flags=17 > FE=> Parse(stmt=null,query="i've remove the select query",oids={}) > FE=> Bind(stmt=null,portal=null) > FE=> Describe(portal=null) > FE=> Execute(portal=null,limit=0) > FE=> Sync > > .. and nothing else. No "<=BE ..." follow the " FE=> Sync" log entry. This seems normal enough. What is logged immediately before before this? Can you capture the network traffic between the server and JDBC driver? If it's a concurrency issue we should see something at the connection level. I'm not familiar with what windows tools are available to take network captures (in the unix world I'd use tcpdump/snoop/ethereal), but if you manage to get something in a format that ethereal will read, send me the raw capture off-list and I'll take a look. > I've also tried to put "stmt.setQueryTimeout(3);" but it doesn't seem to > do anything. setQueryTimeout() is a no-op in the current driver. -O
I've done some more tests and I think that I can say that a same connection is not used by several thread at the same time. However different connections are being used by several threads at the same time. >Can you capture the network traffic between the server and JDBC driver? I don't know how to do that on Windows. Wireshark cannot listen to the windows loopback interface. Have you seen the thread dump? Does it helps you at all? _________________________________________________________________ Ta' p� udsalg �ret rundt p� MSN Shopping: http://shopping.msn.dk - her finder du altid de bedste priser
James, Change the connection so that it connects to the ethernet device on the box, then Wireshark can listen to it. --dc-- On 10-Aug-06, at 6:45 AM, James Im wrote: > I've done some more tests and I think that I can say that a same > connection is not used by several thread at the same time. > However different connections are being used by several threads at the > same time. > >> Can you capture the network traffic between the server and JDBC >> driver? > I don't know how to do that on Windows. Wireshark cannot listen to the > windows loopback interface. > > Have you seen the thread dump? Does it helps you at all? > > _________________________________________________________________ > Ta' på udsalg året rundt på MSN Shopping: http://shopping.msn.dk > - her finder du altid de bedste priser > > > ---------------------------(end of > broadcast)--------------------------- > TIP 3: Have you checked our extensive FAQ? > > http://www.postgresql.org/docs/faq >
Sorry it doesn't work. I connect to 192.168.0.8 but Wireshark doesn't caputre anything on the database port. I'll try to reproduced the problem on linux. Dave Cramer wrote: >James, > >Change the connection so that it connects to the ethernet device on >the box, then Wireshark can listen to it. > >--dc-- >On 10-Aug-06, at 6:45 AM, James Im wrote: > >>I've done some more tests and I think that I can say that a same >>connection is not used by several thread at the same time. >>However different connections are being used by several threads at the >>same time. >> >>>Can you capture the network traffic between the server and JDBC driver? >>I don't know how to do that on Windows. Wireshark cannot listen to the >>windows loopback interface. >> >>Have you seen the thread dump? Does it helps you at all? >> >>_________________________________________________________________ >>Ta' p� udsalg �ret rundt p� MSN Shopping: http://shopping.msn.dk - >>her finder du altid de bedste priser >> >> >>---------------------------(end of broadcast)--------------------------- >>TIP 3: Have you checked our extensive FAQ? >> >> http://www.postgresql.org/docs/faq >> > > >---------------------------(end of broadcast)--------------------------- >TIP 4: Have you searched our list archives? > > http://archives.postgresql.org > _________________________________________________________________ Find dine dokumenter lettere med MSN Toolbar med Windows-pc-s�gning: http://toolbar.msn.dk
Yeah, Windows special-cases even non-loopback local network traffic and bypasses the packet sniffer. I once had to do some iptables magic with a Linux box to bounce local traffic off a remote server and back in so I could sniff it. -- Mark On Thu, 2006-08-10 at 14:55 +0000, James Im wrote: > Sorry it doesn't work. I connect to 192.168.0.8 but Wireshark doesn't > caputre anything on the database port. > > I'll try to reproduced the problem on linux. > > Dave Cramer wrote: > >James, > > > >Change the connection so that it connects to the ethernet device on > >the box, then Wireshark can listen to it. > > > >--dc-- > >On 10-Aug-06, at 6:45 AM, James Im wrote: > > > >>I've done some more tests and I think that I can say that a same > >>connection is not used by several thread at the same time. > >>However different connections are being used by several threads at the > >>same time. > >> > >>>Can you capture the network traffic between the server and JDBC driver? > >>I don't know how to do that on Windows. Wireshark cannot listen to the > >>windows loopback interface. > >> > >>Have you seen the thread dump? Does it helps you at all? > >> > >>_________________________________________________________________ > >>Ta' p udsalg ret rundt p MSN Shopping: http://shopping.msn.dk - > >>her finder du altid de bedste priser > >> > >> > >>---------------------------(end of broadcast)--------------------------- > >>TIP 3: Have you checked our extensive FAQ? > >> > >> http://www.postgresql.org/docs/faq > >> > > > > > >---------------------------(end of broadcast)--------------------------- > >TIP 4: Have you searched our list archives? > > > > http://archives.postgresql.org > > > > _________________________________________________________________ > Find dine dokumenter lettere med MSN Toolbar med Windows-pc-sgning: > http://toolbar.msn.dk > > > ---------------------------(end of broadcast)--------------------------- > TIP 1: if posting/reading through Usenet, please send an appropriate > subscribe-nomail command to majordomo@postgresql.org so that your > message can get through to the mailing list cleanly
I'm executing 'SELECT CURRENT_TIMESTAMP' from a statement object with auto commit off and prepare threshold set to 1. I'm finding that the result comes back the same for after multiple calls and instantiations of the statement object(The same happens for a prepared statement). However this does not occur when I perform a commit before or after every call. Is this expected behaviour?
Nicholas, Yes this is expected behaviour. The time is the time when you started the transaction. It has nothing to do with JDBC. Dave On 10-Aug-06, at 8:37 PM, Nicholas E. Wakefield wrote: > I'm executing 'SELECT CURRENT_TIMESTAMP' from a statement object with > auto commit off and prepare threshold set to 1. I'm finding that the > result comes back the same for after multiple calls and instantiations > of the statement object(The same happens for a prepared statement). > However this does not occur when I perform a commit before or after > every call. > > Is this expected behaviour? > > ---------------------------(end of > broadcast)--------------------------- > TIP 6: explain analyze is your friend >
James Im wrote: > Have you seen the thread dump? Does it helps you at all? Unfortunately the thread dump just says what the logging says -- that the driver is waiting for a message from the server that never comes. -O
Hi, Nicolas, Nicholas E. Wakefield wrote: > I'm executing 'SELECT CURRENT_TIMESTAMP' from a statement object with > auto commit off and prepare threshold set to 1. I'm finding that the > result comes back the same for after multiple calls and instantiations > of the statement object(The same happens for a prepared statement). > However this does not occur when I perform a commit before or after > every call. > > Is this expected behaviour? Yes, it's expected, intended and documented in the PostgreSQL docs: http://www.postgresql.org/docs/8.1/static/functions-datetime.html SELECT timeofday(); returns a clock that advances even during transactions. As it returns a textual representation, you may need to cast it to a timestamp in some query contexts, though: select timeofday(); -- returns text select timeofday()::timestamp; -- returns timestamp - postgresql variant select CAST(timeofday() AS timestamp) ; -- returns timestamp in sql99 HTH, Markus -- Markus Schaber | Logical Tracking&Tracing International AG Dipl. Inf. | Software Development GIS Fight against software patents in EU! www.ffii.org www.nosoftwarepatents.org
The file commview.zip contains traces of the postgresql port captured with Commview (demo). error4.cap should contains the error but may be the beggining is missing. In error5.cap contains all packets exchanged during the error. I'm 100% sure about this capture session. The error should be near the end of the file I guess. The thread dump for the bolcking error that occured in error5.cap is the following: "Thread-23" prio=6 tid=0x256a0b70 nid=0x744 runnable [0x2829f000..0x2829fb64] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:235) - locked <0x116082b0> (a java.io.BufferedInputStream) at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:256) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1163) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:188) - locked <0x11603398> (a org.postgresql.core.v3.QueryExecutorImpl) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:452) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:340) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:239) at com.kumaru.services.jdbc.StatementWrapper.executeQuery(Unknown Source) at com.kumaru.servers.http.HttpUtil.storeUrlMp(Unknown Source) at com.kumaru.servers.http.HttpUtil$1.run(Unknown Source) at com.kumaru.services.pool.ThreadRunnable.run(Unknown Source) Commview makes it possible to listen to the windows loopback but with some limitations (see: http://www.tamos.com/htmlhelp/commview/loopback.htm). That's why I've also done another test. The file wireshark.zip contains 3 files with traces of the postgresql port captured with Wireshark. I could not reproduce exactly the same problem as the setup is not 100% the same. Instead of having server DB in the same windows 2000 box I moved the server into a vmware windows xp box (on the same machine). The difference (from Java point of view) is that the code (in org.postgresql... ) throws a IO exception instead of blocking. The traces are big because the error happens randomly. I hope you've got enough info because I don't think that I can do more! [attachements removed : ask me if you which to receive them] _________________________________________________________________ F� de bedste s�geresultater med MSN Search: http://search.msn.dk
I've tried to reproduce the bug on a linux box but couldn't. I don't know what to think. May be I haven't tried hard enough or may be this is a windows bug or a bug in postgresql on windows. As I said, I don't know. I think that one way to find out would be to get enough information about what's happening from the java side (something like loglevel=10 ?). The driver (or a special version of the driver just for this purpose) could contain a lot more traces so that: 1) any problem of synchronization could be discarded. 2) you could know what java has sent through the socket and 3) what it has received until the moment the code will block. I've never really looked at the driver's code so for me it would be hard to do this but I think that if one of you know the driver's code well it should not be that hard. As I said above: I could understand that you don't want to add many traces in the driver and call it version 408 but you could send me a 'custom' version of the driver with all these traces enabled, just to find out what's happening. Oliver Jowett wrote: >James Im wrote: >>The file commview.zip contains traces of the postgresql port captured >>with Commview (demo). error4.cap should contains the error but may be >>the beggining is missing. In error5.cap contains all packets exchanged >>during the error. I'm 100% sure about this capture session. The error >>should be near the end of the file I guess. > >Well.. this has me confused. > >The wireshark version has the driver sending a normal query and >getting no response from the server .. some time later, it sends a >Terminate (presumably in response to a close() call) and the other end >resets the connection. Looks like what I'd expect if there was a >stateful firewall losing connection state somehow, but the connection >hasn't been idle, so it's not just a simple timeout. > >The commview version just has a normal connection as far as I can see. >I don't see anything strange at all there.. > >About the only explanation I can come up with is if something >somewhere is silently eating responses from the PG server at random >intervals! > >-O > _________________________________________________________________ V�lg selv hvordan du vil kommunikere - skrift, tale, video eller billeder med MSN Messenger: http://messenger.msn.dk/ - her kan du det hele
Hi, James, James Im wrote: > I've tried to reproduce the bug on a linux box but couldn't. I don't > know what to think. May be I haven't tried hard enough or may be this is > a windows bug or a bug in postgresql on windows. As I said, I don't know. Did you try to reproduce the bug with only the jdbc part running on linux, against the windows server, or vice versa? HTH, Markus -- Markus Schaber | Logical Tracking&Tracing International AG Dipl. Inf. | Software Development GIS Fight against software patents in EU! www.ffii.org www.nosoftwarepatents.org
On Saturday 12 August 2006 04:40, James Im wrote: > I've tried to reproduce the bug on a linux box but couldn't. I don't > know what to think. May be I haven't tried hard enough or may be this is > a windows bug or a bug in postgresql on windows. As I said, I don't know. Are you running on a dual processor with Win2003 SP2? If so, I have had problems with hangs as well. They appear to be in the server core, not in the driver. Search the archives in -performance and -hackers for my name. jan -- -------------------------------------------------------------- Jan de Visser jdevisser@digitalfairway.com Baruk Khazad! Khazad ai-menu! --------------------------------------------------------------
Jan de Visser wrote: >Are you running on a dual processor with Win2003 SP2? No. Windows 2000 SP4 on a regular P4 2.53Ghz _________________________________________________________________ Opret en personlig blog og del dine billeder p� MSN Spaces: http://spaces.msn.com/
James Im wrote: > I think that one way to find out would be to get enough information > about what's happening from the java side (something like loglevel=10 > ?). The driver (or a special version of the driver just for this > purpose) could contain a lot more traces so that: 1) any problem of > synchronization could be discarded. 2) you could know what java has sent > through the socket and 3) what it has received until the moment the > code will block. This is essentially what loglevel=2 already does already .. -O
>>I think that one way to find out would be to get enough information >>about what's happening from the java side (something like loglevel=10 >>?). The driver (or a special version of the driver just for this >>purpose) could contain a lot more traces so that: 1) any problem of >>synchronization could be discarded. 2) you could know what java has sent >>through the socket and 3) what it has received until the moment the >>code will block. > >This is essentially what loglevel=2 already does already .. > I don't know. I've been asked a few times if my app had synchronization problems. If the driver traces contained enough info then these questions were superfluous I guess. If you knew precisely what network traffic was going in and out through the socket why had I to capture the network traffic with wireshark and co? Anyhow. You know better than me. _________________________________________________________________ Find dine dokumenter lettere med MSN Toolbar med Windows-pc-s�gning: http://toolbar.msn.dk
James Im wrote: > I don't know. > I've been asked a few times if my app had synchronization problems. If > the driver traces contained enough info then these questions were > superfluous I guess. There is protection against concurrent access down in the code that handles the protocol layer, but if there was a bug with that and you were accessing a connection from multiple threads then it could explain the behavior you see. But I would expect that to show up in the network traces if that were the case, which it doesn't. loglevel=2 logs as protocol-level messages are sent and received; you could certainly log more detail, but I haven't managed to think of a failure mode where more detail there would help diagnose it. If you're prepared to do the work to add more detailed logging, go for it, but I don't have time to do it myself. > If you knew precisely what network traffic was going in and out through > the socket why had I to capture the network traffic with wireshark and co? It was possible that the driver was formatting things incorrectly at the protocol level; since the logging at loglevel=2 does not log the exact data sent/received I needed to see what was actually hitting the network. (I am not convinced that commview is giving an accurate picture of what is going on -- it looks suspiciously like it is hooking in at an API level and then faking a network trace in libpcap format from that -- the TCP data was all over the place..) > Anyhow. You know better than me. Ideally if you can get it happening between two machines where you can get a good capture of the network traffic then I might be able to make progress.. As it is, I don't think you've managed to do that, the wireshark case seems to be something different (that probably needs diagnosis on the server side to work out why the server is not responding). It might be interesting to compare captures on the server and client side in that case, since the connection gets abruptly reset. For the moment I am going to have to shelve investigating this further due to insufficient time .. It's certainly not an obvious problem, I can't see anything wrong in the driver that would trigger it, and I am beginning to suspect OS- or JVM- level weirdness since it is so sensitive to your system setup. I know that there are antivirus/firewall products that like to get their fingers into the windows network API, and they've been known to cause strange bugs in the past .. Maybe that's a possibility here? -O
>I can't see anything wrong in the driver that would trigger it, and I >am beginning to suspect OS- or JVM- level weirdness since it is so >sensitive to your system setup. I know that there are >antivirus/firewall products that like to get their fingers into the >windows network API, and they've been known to cause strange bugs in >the past .. Maybe that's a possibility here? > I'm using ZoneAlarm Pro 3.5. As I was having problems today I recalled your comment and shutdown ZoneAlarm. After that I did not experience any problem so this might be the culprit. If I have any problem while I'm NOT using ZoneAlarm, I'll tell you. Thank you for your help. _________________________________________________________________ Ta' p� udsalg �ret rundt p� MSN Shopping: http://shopping.msn.dk - her finder du altid de bedste priser