Thread: NIC to NIC connection

NIC to NIC connection

From
"Kent Anderson"
Date:
We are upgrading our servers and have run into an interesting situation.  It has been proposed that we have a direct connection from the web servers to the postgres server via extra NICs. Has anyone done this before and how big a project would it be to alter ASP and Java applications to make use of such a connection?
 
Before we even waste time installing the NIC's I would like a sense of how hard it is to get postgres to use that kind of a connection vs over the Internet. We are looking to increase communication speed between the web servers and database server as much as possible.
 
Thanks
Kent Anderson

Re: NIC to NIC connection

From
Andrew Sullivan
Date:
On Tue, Oct 19, 2004 at 12:05:05PM -0400, Kent Anderson wrote:
> We are upgrading our servers and have run into an interesting situation.  It
> has been proposed that we have a direct connection from the web servers to
> the postgres server via extra NICs.

> Before we even waste time installing the NIC's I would like a sense of how
> hard it is to get postgres to use that kind of a connection vs over the
> Internet.

I don't understand.  What interface other than a NIC are you now
using to go "over the Internet"?  Or do you mean that you want an
_extra_ IP address so that the web servers don't have to compete for
the bandwidth?  In that case, sure, you'll probably get a minor
performance increase, depending on your current network traffic.

A

--
Andrew Sullivan  | ajs@crankycanuck.ca
In the future this spectacle of the middle classes shocking the avant-
garde will probably become the textbook definition of Postmodernism.
                --Brad Holland

Re: NIC to NIC connection

From
"Matt Clark"
Date:
You would assign a different subnet to the connection, and then tell the servers to connect to the PG server's address on that subnet.  No other changes required.  Very odd setup though.  If you want a 'private' connection then use a switch, rather than needing umpty NICs in the PG server.
-----Original Message-----
From: pgsql-admin-owner@postgresql.org [mailto:pgsql-admin-owner@postgresql.org] On Behalf Of Kent Anderson
Sent: 19 October 2004 17:05
To: Pgsql-Admin@Postgresql. Org
Subject: [ADMIN] NIC to NIC connection

We are upgrading our servers and have run into an interesting situation.  It has been proposed that we have a direct connection from the web servers to the postgres server via extra NICs. Has anyone done this before and how big a project would it be to alter ASP and Java applications to make use of such a connection?
 
Before we even waste time installing the NIC's I would like a sense of how hard it is to get postgres to use that kind of a connection vs over the Internet. We are looking to increase communication speed between the web servers and database server as much as possible.
 
Thanks
Kent Anderson

Re: NIC to NIC connection

From
"Scott Marlowe"
Date:
This is more a networking issue than a PostgreSQL one.  What you'll like
want to do is to set up a nic in each box and use a "rolled" cable
directly between them.  Assign an IP to each machine like 10.0.0.1 and
10.0.0.2 and edit your route tables to make each machine use those IPs
respectively to talk to each other.  I'm a unix guy, so explaining
exactly how to do this in windows is a mystery to me.  I'm guessing
windows because of the ASP you mention.  The basic theory stays the same
though.

On Tue, 2004-10-19 at 10:05, Kent Anderson wrote:
> We are upgrading our servers and have run into an interesting
> situation.  It has been proposed that we have a direct connection from
> the web servers to the postgres server via extra NICs. Has anyone done
> this before and how big a project would it be to alter ASP and Java
> applications to make use of such a connection?
>
> Before we even waste time installing the NIC's I would like a sense of
> how hard it is to get postgres to use that kind of a connection vs
> over the Internet. We are looking to increase communication speed
> between the web servers and database server as much as possible.
>
> Thanks
> Kent Anderson


Re: NIC to NIC connection

From
Richard_D_Levine@raytheon.com
Date:
Depends on the NIC.  If your running 100MBS Ether and put in 1GBS Ether or
fiber, it would make a difference.

Rick



 
                      Andrew Sullivan
 
                      <ajs@crankycanuck.ca>        To:       "Pgsql-Admin@Postgresql. Org" <pgsql-admin@postgresql.org>
 
                      Sent by:                     cc:
 
                      pgsql-admin-owner@pos        Subject:  Re: [ADMIN] NIC to NIC connection
 
                      tgresql.org
 

 

 
                      10/19/2004 11:28 AM
 

 

 




On Tue, Oct 19, 2004 at 12:05:05PM -0400, Kent Anderson wrote:
> We are upgrading our servers and have run into an interesting situation.
It
> has been proposed that we have a direct connection from the web servers
to
> the postgres server via extra NICs.

> Before we even waste time installing the NIC's I would like a sense of
how
> hard it is to get postgres to use that kind of a connection vs over the
> Internet.

I don't understand.  What interface other than a NIC are you now
using to go "over the Internet"?  Or do you mean that you want an
_extra_ IP address so that the web servers don't have to compete for
the bandwidth?  In that case, sure, you'll probably get a minor
performance increase, depending on your current network traffic.

A

--
Andrew Sullivan  | ajs@crankycanuck.ca
In the future this spectacle of the middle classes shocking the avant-
garde will probably become the textbook definition of Postmodernism.
                --Brad Holland

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

               http://www.postgresql.org/docs/faqs/FAQ.html






Re: NIC to NIC connection

From
"Jay A. Kreibich"
Date:
On Tue, Oct 19, 2004 at 11:02:31AM -0600, Scott Marlowe scratched on the wall:
> This is more a networking issue than a PostgreSQL one.  What you'll like
> want to do is to set up a nic in each box and use a "rolled" cable
> directly between them

  Actually, you want a "cross-over" cable, not a "roll" cable.  There is
  a difference.

  With many newer, high quality NICs, you don't even need that.  Many
  modern NICs do auto MDI/MDI-X detection, so any standard cable will do.


   -j

--
                     Jay A. Kreibich | Comm. Technologies, R&D
                        jak@uiuc.edu | Campus IT & Edu. Svcs.
          <http://www.uiuc.edu/~jak> | University of Illinois at U/C

Re: NIC to NIC connection

From
Andrew Sullivan
Date:
On Tue, Oct 19, 2004 at 12:11:20PM -0500, Richard_D_Levine@raytheon.com wrote:
>
> Depends on the NIC.  If your running 100MBS Ether and put in 1GBS Ether or
> fiber, it would make a difference.

Not if you're not using 100 megs in the first place :)

A

--
Andrew Sullivan  | ajs@crankycanuck.ca
I remember when computers were frustrating because they *did* exactly what
you told them to.  That actually seems sort of quaint now.
        --J.D. Baldwin

Re: NIC to NIC connection

From
Ben Kim
Date:
My 2 pence.

I think it's not a big task. If the servers are located nearby, all you
need is.

Connect the two machines:
===========================

- get a crossover patch cable (Should be crossover - you can buy or make
one by googling instructions.)

- install one extra NIC on each machine and connect the two NICs using the
crossover cable

- give a private address to each NIC so that they share the same internal
network, like 192.168.0.3 and 192.168.0.4.

Change connection string:
===========================

- change the database host in your ASP or JSP database connection file(s)
to this internal address. (For example, jdbc:postgresql://db.some.com/mydb
to jdbc:postgresql://192.168.0.4/mydb.)

If you are using dreamweaver, etc., if you will need to change the
connection setting there too.

In my network a bulk transfer speed through the internal link is not much
better than that of the external link. But the former is significantly
better during a busy day. (But then, the database sessions are more likely
to be interactive type of traffic...)

You also get security as a side benefit. (Data can't be snooped.)


Regards,

Ben Kim
Database Developer/Systems Administrator
College of Education
Texas A&M University

On Tue, 19 Oct 2004, Kent Anderson wrote:

> We are upgrading our servers and have run into an interesting situation.  It
> has been proposed that we have a direct connection from the web servers to
> the postgres server via extra NICs. Has anyone done this before and how big
> a project would it be to alter ASP and Java applications to make use of such
> a connection?
>
> Before we even waste time installing the NIC's I would like a sense of how
> hard it is to get postgres to use that kind of a connection vs over the
> Internet. We are looking to increase communication speed between the web
> servers and database server as much as possible.
>
> Thanks
> Kent Anderson
>


Re: NIC to NIC connection

From
Richard_D_Levine@Raytheon.com
Date:
True.  I can't imagine a modern server room, even running windows, that
doesn't have a gigabit or fiber backbone for the servers.

Rick



 
                      Andrew Sullivan
 
                      <ajs@crankycanuck.ca>        To:       "Pgsql-Admin@Postgresql. Org"
<pgsql-admin@postgresql.org>, 
                      Sent by:                      pgsql-admin-owner@postgresql.org
 
                      pgsql-admin-owner@pos        cc:
 
                      tgresql.org                  Subject:  Re: [ADMIN] NIC to NIC connection
 

 

 
                      10/19/2004 12:58 PM
 

 

 




On Tue, Oct 19, 2004 at 12:11:20PM -0500, Richard_D_Levine@raytheon.com
wrote:
>
> Depends on the NIC.  If your running 100MBS Ether and put in 1GBS Ether
or
> fiber, it would make a difference.

Not if you're not using 100 megs in the first place :)

A

--
Andrew Sullivan  | ajs@crankycanuck.ca
I remember when computers were frustrating because they *did* exactly what
you told them to.  That actually seems sort of quaint now.
                         --J.D. Baldwin

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend






Re: NIC to NIC connection

From
Bruno Wolff III
Date:
On Tue, Oct 19, 2004 at 17:37:43 +0100,
  Matt Clark <matt@ymogen.net> wrote:
> You would assign a different subnet to the connection, and then tell the
> servers to connect to the PG server's address on that subnet.  No other
> changes required.  Very odd setup though.  If you want a 'private'
> connection then use a switch, rather than needing umpty NICs in the PG
> server.

Switches are not security devices. While it is harder to sniff packets on
switches, you can't count on them to prevent hostile machines on the
switch from playing games with the arp protocol. Also I believe that if
a switch doesn't remember where a particular mac address is it will send
the packet to all of the attached ports.

Re: NIC to NIC connection

From
Matt Clark
Date:
>Switches are not security devices. While it is harder to sniff packets on
>switches, you can't count on them to prevent hostile machines on the
>switch from playing games with the arp protocol. Also I believe that if
>a switch doesn't remember where a particular mac address is it will send
>the packet to all of the attached ports.
>
>
If you have 6 app servers it's just daft to stick 6 NICs in your DB
server.   If absolute privacy is a concern (not mentioned by the OP),
then use a dedicated switch (or switches) for the 'private' subnet.
Even better, use SSH.  But all this is over the top for 99.9% of uses
anyway.  A VLAN is as private as anything else, so you can just create a
VLAN on your current switch fabric and use that.  No kind of traffic on
a VLAN will hit any other VLAN.  Unless of course someone has hacked
your switch, set up a mirror port, attached a sniffer or other hacked
machine to it, and is assiduously reading your traffic, in which case
you have bigger problems....


M

Re: NIC to NIC connection

From
Bruno Wolff III
Date:
On Tue, Oct 19, 2004 at 23:13:44 +0100,
  Matt Clark <matt@ymogen.net> wrote:
>
> If you have 6 app servers it's just daft to stick 6 NICs in your DB
> server.

While there might be some cases where that makes sense most likely it
isn't something you would want to do. I believe the original motivation
was to solve bandwidth congestion rather than security.
Most likely the systems were already connected to switches.
If the nic to the web server(s) were really swamped (which seems unlikely,
but could be the case), then adding a second nic and connecting to another
port on the switch they are already using would probably work as well
as running cables directly from the web server(s) to the database.
They would need to adjust the routing tables so that some traffic went
over each nic. An easy change would be to specify that traffic destined
for the database server IP address is to go over the new nic and all other
traffic is to go over the old nic.

Re: NIC to NIC connection

From
Christian Fowler
Date:
There have been many comments about this already Kent. My $.02:

The most common practice I am aware of is to install 2 NIC's in each
appserver - one to your load balancer, and one to your private network
(192.168.*) where your database server sites. In fact, ideally your
database machine has no publically addressable nic at all. I have
personally dealt with such a setup in several installs, some doing of
millions of page views per *day*, and it has always been very reliable,
secure, and fast.

Use gigabit everywhere on your 192.168 "database" network. If you are
concernced about bandwidth, wire up http://www.mrtg.org and look at the
traffic for yourself.

getting postgres to use this setup should be a piece of cake. Just make
sure your settings in pg_hba.conf are setup right.

Good luck.

On Tue, 19 Oct 2004, Kent Anderson wrote:

> We are upgrading our servers and have run into an interesting situation.  It
> has been proposed that we have a direct connection from the web servers to
> the postgres server via extra NICs. Has anyone done this before and how big
> a project would it be to alter ASP and Java applications to make use of such
> a connection?
>
> Before we even waste time installing the NIC's I would like a sense of how
> hard it is to get postgres to use that kind of a connection vs over the
> Internet. We are looking to increase communication speed between the web
> servers and database server as much as possible.
>
> Thanks
> Kent Anderson
>

[ \ /
[ >X<   Christian Fowler      | spider@viovio.com
[ / \   http://www.viovio.com | http://www.tikipro.org

Re: NIC to NIC connection

From
Matt Clark
Date:

If you have 6 app servers it's just daft to stick 6 NICs in your DB 
server.   
While there might be some cases where that makes sense most likely it
isn't something you would want to do. I believe the original motivation
was to solve bandwidth congestion rather than security.
 
If the NICs are saturated then channel bonding is the answer.  Any modern switch and OS will let you bind together multiple NICs so they appear as one.  You can get an 4Gbps link this way.

Re: NIC to NIC connection

From
Gaetano Mendola
Date:
Bruno Wolff III wrote:
> Also I believe that if
> a switch doesn't remember where a particular mac address is it will send
> the packet to all of the attached ports.

I don't think so, I guess the switch perform a sort of arpping in order to
detect who have a macaddress assigned, even the multicast is not sent
to all ports but only to that ports where "someone" sent an arp packet saying
the he was registered to a multicast address.

However I don't think exist a standard.


Regards
Gaetano Mendola




Re: NIC to NIC connection

From
Bruno Wolff III
Date:
On Thu, Oct 21, 2004 at 10:07:33 +0200,
  Gaetano Mendola <mendola@bigfoot.com> wrote:
> Bruno Wolff III wrote:
> >Also I believe that if
> >a switch doesn't remember where a particular mac address is it will send
> >the packet to all of the attached ports.
>
> I don't think so, I guess the switch perform a sort of arpping in order to
> detect who have a macaddress assigned, even the multicast is not sent
> to all ports but only to that ports where "someone" sent an arp packet
> saying
> the he was registered to a multicast address.

Switches don't do arps; they are a layer 2 device (and hence don't know
about IP addresses. They cache mac address port bindings. If their cache
expires before the cached IP mac address bindings in some attached device
then they can get requests to send to mac addresses for which they don't
have a cached port. In that case they are supposed to do a port flood of
the packet.

I actually had a problem with a switch that was misconfigured and didn't
do a port flood in these circumstances. That combined with a linux box
that wasn't chatty enough to stay in the switch's cache and that the
router cached IP address - mac address bindings for hours while the switch
kept mac address - port bindings for minutes caused my machine to be
unreachable to other hosts for periods of time.

Re: NIC to NIC connection

From
"Jay A. Kreibich"
Date:
On Thu, Oct 21, 2004 at 10:07:33AM +0200, Gaetano Mendola scratched on the wall:
> Bruno Wolff III wrote:
> >Also I believe that if
> >a switch doesn't remember where a particular mac address is it will send
> >the packet to all of the attached ports.
>
> I don't think so, I guess the switch perform a sort of arpping in order to
> detect who have a macaddress assigned,

  No, he's right. If the MAC to port mapping has not been learned by
  the switch, the packet is flooded to all ports or (for really bad
  switches) dropped.  A switch is a pure layer-two device and ARP
  involves layer-three addresses and concepts.

  The only part of a switch that will ARP is the management system,
  assuming it has an IP address of its own.

  This is actually a standard test we run when evaluating switches for
  deployment on campus.  We hook about ten devices up to a switch and
  send them tons of traffic. The systems count incoming packets but
  they do not generate any traffic, so the switch is forced to flood
  all the traffic.  The vast majority of low-cost switches roll over and
  die at that point.

> even the multicast is not sent
> to all ports but only to that ports where "someone" sent an arp packet
> saying the he was registered to a multicast address.

  Not ARP, IGMP (Internet Group Management Protocol).  It's part of the
  IP suite, and is much higher up in the protocol stack.

  And again, this isn't true unless the switch does IGMP snooping or
  has multicast assist from the router.  IGMP snooping is becoming more
  common on higher-end switches, but there are some companies that
  prefer the router assist design.  The problem with that is that the
  switches and routers need to be from the same company.

  IGMP snooping is also really tricky to do right, and there are still
  some situations where you are forced to flood traffic.

   -j

--
                     Jay A. Kreibich | Comm. Technologies, R&D
                        jak@uiuc.edu | Campus IT & Edu. Svcs.
          <http://www.uiuc.edu/~jak> | University of Illinois at U/C

Re: NIC to NIC connection

From
"Jay A. Kreibich"
Date:
On Thu, Oct 21, 2004 at 07:05:40PM +0200, Gaetano Mendola scratched on the wall:

> We have some switches that are able to do ip routing too... :-(

  So called "layer-three switches" are a whole different game.

  Essentualy, they are just routers where the "routing" interfaces and
  the physical interfaces are abstracted from each other with a layer-two
  switching engine.  The term "layer-three swtich" is really more of a
  marketing term, however, and is not a very accurate description of
  what the device is or how it works.

> I have to admint that I'm not a switch specialist but given the ability
> to do routing I was imagine the arpping trich...

  Yes, but such systems are generally routers first and switches second.

   -j

--
                     Jay A. Kreibich | Comm. Technologies, R&D
                        jak@uiuc.edu | Campus IT & Edu. Svcs.
          <http://www.uiuc.edu/~jak> | University of Illinois at U/C

Re: NIC to NIC connection

From
Gaetano Mendola
Date:
Jay A. Kreibich wrote:
> On Thu, Oct 21, 2004 at 10:07:33AM +0200, Gaetano Mendola scratched on the wall:
>
>>Bruno Wolff III wrote:
>>
>>>Also I believe that if
>>>a switch doesn't remember where a particular mac address is it will send
>>>the packet to all of the attached ports.
>>
>>I don't think so, I guess the switch perform a sort of arpping in order to
>>detect who have a macaddress assigned,
>
>
>   No, he's right. If the MAC to port mapping has not been learned by
>   the switch, the packet is flooded to all ports or (for really bad
>   switches) dropped.  A switch is a pure layer-two device and ARP
>   involves layer-three addresses and concepts.

We have some switches that are able to do ip routing too... :-(
I have to admint that I'm not a switch specialist but given the ability to do
routing I was imagine the arpping trich...


Regards
Gaetano Mendola





Re: NIC to NIC connection

From
Gaetano Mendola
Date:
Jay A. Kreibich wrote:
 > On Thu, Oct 21, 2004 at 07:05:40PM +0200, Gaetano Mendola scratched on the wall:
 >
 >
 >>We have some switches that are able to do ip routing too... :-(
 >
 >
 >   So called "layer-three switches" are a whole different game.

Ok that's explain all, I was able to create two different VLAN's with a
cable between two ports ( in order to simulate two different switches), each port was
appartaining to a different VLAN and I was seeing ARPREQUEST passing by the cable in
order to detect the address in the other side, just for fun I replaced the cable with
a traffic shaper and all was working very fine...
However I wasn't curious enough to see who was the arp request source, may be was not
the "router/switcher" but a client that was doing it.


Regards
Gaetano Mendola