Re: Corrupted Data ? - Mailing list pgsql-general

On 08/12/2016 08:30 AM, Ioana Danes wrote:

On Fri, Aug 12, 2016 at 11:26 AM, Adrian Klaver
<adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>> wrote:

    On 08/12/2016 08:10 AM, Ioana Danes wrote:



        On Fri, Aug 12, 2016 at 10:47 AM, Francisco Olarte
        <folarte@peoplecall.com <mailto:folarte@peoplecall.com>
        <mailto:folarte@peoplecall.com <mailto:folarte@peoplecall.com>>>
        wrote:

            CCing to the list...

        Thanks


            On Fri, Aug 12, 2016 at 4:10 PM, Ioana Danes
        <ioanadanes@gmail.com <mailto:ioanadanes@gmail.com>
            <mailto:ioanadanes@gmail.com <mailto:ioanadanes@gmail.com>>>
        wrote:
            >> given 318220 and 318216 are just a bit away ( 4db08/4db0c
        ), and it
            >> repeats sporadically, have you ruled out ( by having page
            checksums or
            >> other mechanism ) a potential disk read/write error ?
            >>
            >>
            >> > Also the index is correct on db3 as the record in case
        (with
            drawid =
            >> > 318216) is retrieved if I filter by drawid = 318220
            >>
            >> Specially if this happens, you may have some slightly bad
        disks/ram/
            >> leading to this kind of problems.
            >>
            >
            > Could be. I also had some issues with an rsync between db3 and
            drdb a week
            > ago that did not complete for bigger files (> 200MB) and
        gave me some
            > corruption messages. Then the system was revbooted and
        everything
            seemed
            > fine but apparently it is not.
            > I am planning to drop & create the table from a good
        backup and if
            that does
            > not fix the issue then I will rebuild the server.

            I would check whatever logs you can ( syslog or eventlog,
        smart log,
            etc.. ) hunting for disk errors ( sometimes they are
        reported ). This
            kind of problems, with programs as tested as postgres and
        rsync, tend
            to indicate controller/RAM/disk going bad ( in your case it
        could be
            caused by a single bit getting flipped in a sector for the data
            portion of the table, and not being propagated either because it
            happened after your sync of drdb or because it was synced
        from the WAL
            and not the table, or because it was read from the disk cache ).

        I agree, unfortunately I did not find any clues about corruption
        or any
        anomalies in the logs.
        I will work tonight to rebuild that table and see where I go
        from there.


    The db3 database is on a different machine from all the other
    databases you set up, correct?

Yes, they are all different vms first 3 dbs are on the same cluster but
drdb is a remote machine,

Aah, another player in the mix.

What virtualization technology are you using?

Thank you



        Thanks,
        ioana

            Francisco Olarte.




    --
    Adrian Klaver
    adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>

--
Adrian Klaver
adrian.klaver@aklaver.com