Thread: NIC to NIC connection
We are upgrading our servers and have run into an interesting situation. It has been proposed that we have a direct connection from the web servers to the postgres server via extra NICs. Has anyone done this before and how big a project would it be to alter ASP and Java applications to make use of such a connection?
Before we even waste time installing the NIC's I would like a sense of how hard it is to get postgres to use that kind of a connection vs over the Internet. We are looking to increase communication speed between the web servers and database server as much as possible.
Thanks
Kent Anderson
On Tue, Oct 19, 2004 at 12:05:05PM -0400, Kent Anderson wrote:
> We are upgrading our servers and have run into an interesting situation. It
> has been proposed that we have a direct connection from the web servers to
> the postgres server via extra NICs.
> Before we even waste time installing the NIC's I would like a sense of how
> hard it is to get postgres to use that kind of a connection vs over the
> Internet.
I don't understand. What interface other than a NIC are you now
using to go "over the Internet"? Or do you mean that you want an
_extra_ IP address so that the web servers don't have to compete for
the bandwidth? In that case, sure, you'll probably get a minor
performance increase, depending on your current network traffic.
A
--
Andrew Sullivan | ajs@crankycanuck.ca
In the future this spectacle of the middle classes shocking the avant-
garde will probably become the textbook definition of Postmodernism.
--Brad Holland
You would assign a different subnet to the connection, and then tell the servers to connect to the PG server's address on that subnet. No other changes required. Very odd setup though. If you want a 'private' connection then use a switch, rather than needing umpty NICs in the PG server.
-----Original Message-----
From: pgsql-admin-owner@postgresql.org [mailto:pgsql-admin-owner@postgresql.org] On Behalf Of Kent Anderson
Sent: 19 October 2004 17:05
To: Pgsql-Admin@Postgresql. Org
Subject: [ADMIN] NIC to NIC connectionWe are upgrading our servers and have run into an interesting situation. It has been proposed that we have a direct connection from the web servers to the postgres server via extra NICs. Has anyone done this before and how big a project would it be to alter ASP and Java applications to make use of such a connection?Before we even waste time installing the NIC's I would like a sense of how hard it is to get postgres to use that kind of a connection vs over the Internet. We are looking to increase communication speed between the web servers and database server as much as possible.ThanksKent Anderson
This is more a networking issue than a PostgreSQL one. What you'll like want to do is to set up a nic in each box and use a "rolled" cable directly between them. Assign an IP to each machine like 10.0.0.1 and 10.0.0.2 and edit your route tables to make each machine use those IPs respectively to talk to each other. I'm a unix guy, so explaining exactly how to do this in windows is a mystery to me. I'm guessing windows because of the ASP you mention. The basic theory stays the same though. On Tue, 2004-10-19 at 10:05, Kent Anderson wrote: > We are upgrading our servers and have run into an interesting > situation. It has been proposed that we have a direct connection from > the web servers to the postgres server via extra NICs. Has anyone done > this before and how big a project would it be to alter ASP and Java > applications to make use of such a connection? > > Before we even waste time installing the NIC's I would like a sense of > how hard it is to get postgres to use that kind of a connection vs > over the Internet. We are looking to increase communication speed > between the web servers and database server as much as possible. > > Thanks > Kent Anderson
Depends on the NIC. If your running 100MBS Ether and put in 1GBS Ether or
fiber, it would make a difference.
Rick
Andrew Sullivan
<ajs@crankycanuck.ca> To: "Pgsql-Admin@Postgresql. Org" <pgsql-admin@postgresql.org>
Sent by: cc:
pgsql-admin-owner@pos Subject: Re: [ADMIN] NIC to NIC connection
tgresql.org
10/19/2004 11:28 AM
On Tue, Oct 19, 2004 at 12:05:05PM -0400, Kent Anderson wrote:
> We are upgrading our servers and have run into an interesting situation.
It
> has been proposed that we have a direct connection from the web servers
to
> the postgres server via extra NICs.
> Before we even waste time installing the NIC's I would like a sense of
how
> hard it is to get postgres to use that kind of a connection vs over the
> Internet.
I don't understand. What interface other than a NIC are you now
using to go "over the Internet"? Or do you mean that you want an
_extra_ IP address so that the web servers don't have to compete for
the bandwidth? In that case, sure, you'll probably get a minor
performance increase, depending on your current network traffic.
A
--
Andrew Sullivan | ajs@crankycanuck.ca
In the future this spectacle of the middle classes shocking the avant-
garde will probably become the textbook definition of Postmodernism.
--Brad Holland
---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faqs/FAQ.html
On Tue, Oct 19, 2004 at 11:02:31AM -0600, Scott Marlowe scratched on the wall:
> This is more a networking issue than a PostgreSQL one. What you'll like
> want to do is to set up a nic in each box and use a "rolled" cable
> directly between them
Actually, you want a "cross-over" cable, not a "roll" cable. There is
a difference.
With many newer, high quality NICs, you don't even need that. Many
modern NICs do auto MDI/MDI-X detection, so any standard cable will do.
-j
--
Jay A. Kreibich | Comm. Technologies, R&D
jak@uiuc.edu | Campus IT & Edu. Svcs.
<http://www.uiuc.edu/~jak> | University of Illinois at U/C
On Tue, Oct 19, 2004 at 12:11:20PM -0500, Richard_D_Levine@raytheon.com wrote:
>
> Depends on the NIC. If your running 100MBS Ether and put in 1GBS Ether or
> fiber, it would make a difference.
Not if you're not using 100 megs in the first place :)
A
--
Andrew Sullivan | ajs@crankycanuck.ca
I remember when computers were frustrating because they *did* exactly what
you told them to. That actually seems sort of quaint now.
--J.D. Baldwin
My 2 pence. I think it's not a big task. If the servers are located nearby, all you need is. Connect the two machines: =========================== - get a crossover patch cable (Should be crossover - you can buy or make one by googling instructions.) - install one extra NIC on each machine and connect the two NICs using the crossover cable - give a private address to each NIC so that they share the same internal network, like 192.168.0.3 and 192.168.0.4. Change connection string: =========================== - change the database host in your ASP or JSP database connection file(s) to this internal address. (For example, jdbc:postgresql://db.some.com/mydb to jdbc:postgresql://192.168.0.4/mydb.) If you are using dreamweaver, etc., if you will need to change the connection setting there too. In my network a bulk transfer speed through the internal link is not much better than that of the external link. But the former is significantly better during a busy day. (But then, the database sessions are more likely to be interactive type of traffic...) You also get security as a side benefit. (Data can't be snooped.) Regards, Ben Kim Database Developer/Systems Administrator College of Education Texas A&M University On Tue, 19 Oct 2004, Kent Anderson wrote: > We are upgrading our servers and have run into an interesting situation. It > has been proposed that we have a direct connection from the web servers to > the postgres server via extra NICs. Has anyone done this before and how big > a project would it be to alter ASP and Java applications to make use of such > a connection? > > Before we even waste time installing the NIC's I would like a sense of how > hard it is to get postgres to use that kind of a connection vs over the > Internet. We are looking to increase communication speed between the web > servers and database server as much as possible. > > Thanks > Kent Anderson >
True. I can't imagine a modern server room, even running windows, that
doesn't have a gigabit or fiber backbone for the servers.
Rick
Andrew Sullivan
<ajs@crankycanuck.ca> To: "Pgsql-Admin@Postgresql. Org"
<pgsql-admin@postgresql.org>,
Sent by: pgsql-admin-owner@postgresql.org
pgsql-admin-owner@pos cc:
tgresql.org Subject: Re: [ADMIN] NIC to NIC connection
10/19/2004 12:58 PM
On Tue, Oct 19, 2004 at 12:11:20PM -0500, Richard_D_Levine@raytheon.com
wrote:
>
> Depends on the NIC. If your running 100MBS Ether and put in 1GBS Ether
or
> fiber, it would make a difference.
Not if you're not using 100 megs in the first place :)
A
--
Andrew Sullivan | ajs@crankycanuck.ca
I remember when computers were frustrating because they *did* exactly what
you told them to. That actually seems sort of quaint now.
--J.D. Baldwin
---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend
On Tue, Oct 19, 2004 at 17:37:43 +0100, Matt Clark <matt@ymogen.net> wrote: > You would assign a different subnet to the connection, and then tell the > servers to connect to the PG server's address on that subnet. No other > changes required. Very odd setup though. If you want a 'private' > connection then use a switch, rather than needing umpty NICs in the PG > server. Switches are not security devices. While it is harder to sniff packets on switches, you can't count on them to prevent hostile machines on the switch from playing games with the arp protocol. Also I believe that if a switch doesn't remember where a particular mac address is it will send the packet to all of the attached ports.
>Switches are not security devices. While it is harder to sniff packets on >switches, you can't count on them to prevent hostile machines on the >switch from playing games with the arp protocol. Also I believe that if >a switch doesn't remember where a particular mac address is it will send >the packet to all of the attached ports. > > If you have 6 app servers it's just daft to stick 6 NICs in your DB server. If absolute privacy is a concern (not mentioned by the OP), then use a dedicated switch (or switches) for the 'private' subnet. Even better, use SSH. But all this is over the top for 99.9% of uses anyway. A VLAN is as private as anything else, so you can just create a VLAN on your current switch fabric and use that. No kind of traffic on a VLAN will hit any other VLAN. Unless of course someone has hacked your switch, set up a mirror port, attached a sniffer or other hacked machine to it, and is assiduously reading your traffic, in which case you have bigger problems.... M
On Tue, Oct 19, 2004 at 23:13:44 +0100, Matt Clark <matt@ymogen.net> wrote: > > If you have 6 app servers it's just daft to stick 6 NICs in your DB > server. While there might be some cases where that makes sense most likely it isn't something you would want to do. I believe the original motivation was to solve bandwidth congestion rather than security. Most likely the systems were already connected to switches. If the nic to the web server(s) were really swamped (which seems unlikely, but could be the case), then adding a second nic and connecting to another port on the switch they are already using would probably work as well as running cables directly from the web server(s) to the database. They would need to adjust the routing tables so that some traffic went over each nic. An easy change would be to specify that traffic destined for the database server IP address is to go over the new nic and all other traffic is to go over the old nic.
There have been many comments about this already Kent. My $.02: The most common practice I am aware of is to install 2 NIC's in each appserver - one to your load balancer, and one to your private network (192.168.*) where your database server sites. In fact, ideally your database machine has no publically addressable nic at all. I have personally dealt with such a setup in several installs, some doing of millions of page views per *day*, and it has always been very reliable, secure, and fast. Use gigabit everywhere on your 192.168 "database" network. If you are concernced about bandwidth, wire up http://www.mrtg.org and look at the traffic for yourself. getting postgres to use this setup should be a piece of cake. Just make sure your settings in pg_hba.conf are setup right. Good luck. On Tue, 19 Oct 2004, Kent Anderson wrote: > We are upgrading our servers and have run into an interesting situation. It > has been proposed that we have a direct connection from the web servers to > the postgres server via extra NICs. Has anyone done this before and how big > a project would it be to alter ASP and Java applications to make use of such > a connection? > > Before we even waste time installing the NIC's I would like a sense of how > hard it is to get postgres to use that kind of a connection vs over the > Internet. We are looking to increase communication speed between the web > servers and database server as much as possible. > > Thanks > Kent Anderson > [ \ / [ >X< Christian Fowler | spider@viovio.com [ / \ http://www.viovio.com | http://www.tikipro.org
If the NICs are saturated then channel bonding is the answer. Any modern switch and OS will let you bind together multiple NICs so they appear as one. You can get an 4Gbps link this way.If you have 6 app servers it's just daft to stick 6 NICs in your DB server.While there might be some cases where that makes sense most likely it isn't something you would want to do. I believe the original motivation was to solve bandwidth congestion rather than security.
Bruno Wolff III wrote: > Also I believe that if > a switch doesn't remember where a particular mac address is it will send > the packet to all of the attached ports. I don't think so, I guess the switch perform a sort of arpping in order to detect who have a macaddress assigned, even the multicast is not sent to all ports but only to that ports where "someone" sent an arp packet saying the he was registered to a multicast address. However I don't think exist a standard. Regards Gaetano Mendola
On Thu, Oct 21, 2004 at 10:07:33 +0200, Gaetano Mendola <mendola@bigfoot.com> wrote: > Bruno Wolff III wrote: > >Also I believe that if > >a switch doesn't remember where a particular mac address is it will send > >the packet to all of the attached ports. > > I don't think so, I guess the switch perform a sort of arpping in order to > detect who have a macaddress assigned, even the multicast is not sent > to all ports but only to that ports where "someone" sent an arp packet > saying > the he was registered to a multicast address. Switches don't do arps; they are a layer 2 device (and hence don't know about IP addresses. They cache mac address port bindings. If their cache expires before the cached IP mac address bindings in some attached device then they can get requests to send to mac addresses for which they don't have a cached port. In that case they are supposed to do a port flood of the packet. I actually had a problem with a switch that was misconfigured and didn't do a port flood in these circumstances. That combined with a linux box that wasn't chatty enough to stay in the switch's cache and that the router cached IP address - mac address bindings for hours while the switch kept mac address - port bindings for minutes caused my machine to be unreachable to other hosts for periods of time.
On Thu, Oct 21, 2004 at 10:07:33AM +0200, Gaetano Mendola scratched on the wall:
> Bruno Wolff III wrote:
> >Also I believe that if
> >a switch doesn't remember where a particular mac address is it will send
> >the packet to all of the attached ports.
>
> I don't think so, I guess the switch perform a sort of arpping in order to
> detect who have a macaddress assigned,
No, he's right. If the MAC to port mapping has not been learned by
the switch, the packet is flooded to all ports or (for really bad
switches) dropped. A switch is a pure layer-two device and ARP
involves layer-three addresses and concepts.
The only part of a switch that will ARP is the management system,
assuming it has an IP address of its own.
This is actually a standard test we run when evaluating switches for
deployment on campus. We hook about ten devices up to a switch and
send them tons of traffic. The systems count incoming packets but
they do not generate any traffic, so the switch is forced to flood
all the traffic. The vast majority of low-cost switches roll over and
die at that point.
> even the multicast is not sent
> to all ports but only to that ports where "someone" sent an arp packet
> saying the he was registered to a multicast address.
Not ARP, IGMP (Internet Group Management Protocol). It's part of the
IP suite, and is much higher up in the protocol stack.
And again, this isn't true unless the switch does IGMP snooping or
has multicast assist from the router. IGMP snooping is becoming more
common on higher-end switches, but there are some companies that
prefer the router assist design. The problem with that is that the
switches and routers need to be from the same company.
IGMP snooping is also really tricky to do right, and there are still
some situations where you are forced to flood traffic.
-j
--
Jay A. Kreibich | Comm. Technologies, R&D
jak@uiuc.edu | Campus IT & Edu. Svcs.
<http://www.uiuc.edu/~jak> | University of Illinois at U/C
On Thu, Oct 21, 2004 at 07:05:40PM +0200, Gaetano Mendola scratched on the wall:
> We have some switches that are able to do ip routing too... :-(
So called "layer-three switches" are a whole different game.
Essentualy, they are just routers where the "routing" interfaces and
the physical interfaces are abstracted from each other with a layer-two
switching engine. The term "layer-three swtich" is really more of a
marketing term, however, and is not a very accurate description of
what the device is or how it works.
> I have to admint that I'm not a switch specialist but given the ability
> to do routing I was imagine the arpping trich...
Yes, but such systems are generally routers first and switches second.
-j
--
Jay A. Kreibich | Comm. Technologies, R&D
jak@uiuc.edu | Campus IT & Edu. Svcs.
<http://www.uiuc.edu/~jak> | University of Illinois at U/C
Jay A. Kreibich wrote: > On Thu, Oct 21, 2004 at 10:07:33AM +0200, Gaetano Mendola scratched on the wall: > >>Bruno Wolff III wrote: >> >>>Also I believe that if >>>a switch doesn't remember where a particular mac address is it will send >>>the packet to all of the attached ports. >> >>I don't think so, I guess the switch perform a sort of arpping in order to >>detect who have a macaddress assigned, > > > No, he's right. If the MAC to port mapping has not been learned by > the switch, the packet is flooded to all ports or (for really bad > switches) dropped. A switch is a pure layer-two device and ARP > involves layer-three addresses and concepts. We have some switches that are able to do ip routing too... :-( I have to admint that I'm not a switch specialist but given the ability to do routing I was imagine the arpping trich... Regards Gaetano Mendola
Jay A. Kreibich wrote: > On Thu, Oct 21, 2004 at 07:05:40PM +0200, Gaetano Mendola scratched on the wall: > > >>We have some switches that are able to do ip routing too... :-( > > > So called "layer-three switches" are a whole different game. Ok that's explain all, I was able to create two different VLAN's with a cable between two ports ( in order to simulate two different switches), each port was appartaining to a different VLAN and I was seeing ARPREQUEST passing by the cable in order to detect the address in the other side, just for fun I replaced the cable with a traffic shaper and all was working very fine... However I wasn't curious enough to see who was the arp request source, may be was not the "router/switcher" but a client that was doing it. Regards Gaetano Mendola