Re: how to identify outliers - Mailing list pgsql-general

From Sam Mason
Subject Re: how to identify outliers
Date
Msg-id 20091028110039.GQ5407@samason.me.uk
Whole thread Raw
In response to Re: how to identify outliers  (John R Pierce <pierce@hogranch.com>)
Responses Re: how to identify outliers
List pgsql-general
> Rhys A.D. Stewart wrote:
> >I would like to remove the outliers in distance

As others have said; an "outlier" is normally a human call and not
something that's generally valid to do automatically.  The operator
would probably want to go in and look to see why it's that far out and
either fix the typo or do whatever else it takes to "fix" the problem.

On Tue, Oct 27, 2009 at 04:58:23PM -0700, John R Pierce wrote:
> you could probably do something with a standard deviation that is more
> accurate for large sets than just tossing the 2 outliers.

I'd agree, stddev is probably best and the following should do something
reasonable for what the OP was asking:

  SELECT d.*
  FROM data d, (
    SELECT avg(distance), stddev(distance) FROM data) x
  WHERE abs(d.distance - x.avg) < x.stddev * 2;

--
  Sam  http://samason.me.uk/

pgsql-general by date:

Previous
From: Alban Hertroys
Date:
Subject: Re: Slow running query with views...how to increase efficiency? with index?
Next
From: "Chris Spotts"
Date:
Subject: Re: how to identify outliers