RE: RHEL 7 (systemd) reboot - Mailing list pgsql-general

Sorry, I wasn't clear in the prior posts.   

The stop script is running during reboot. The problem is the database is not reachable when the stop script runs.  The
ctmdistserver shut down is as follows:
 
   Stop control-m application
   Stop control-m configuration agent
   Stop database

As you can see the intent is for the database to be shut down after the product. 

But as you noticed from /var/log/message the stop_ctmlinux_server.sh  script is running but unable to execute the
updatequery.
 

I created the following Service definition and scripts that follow -- note there are 2 datacenters (ctmdist, ctmlinux)
thathave comparable scripts so I have only included one set:
 

[root@kccontrolmt01 ~]# cat ControlM_Shutdown.service
[Unit]
Description=Run mycommand at shutdown
Requires=network.target CTM_Postgre.service
DefaultDependencies=no
Before=shutdown.target reboot.target

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/bin/true
ExecStop=/root/scripts/control-m_shutdown.sh

[Install]
WantedBy=multi-user.target


[root@kccontrolmt01 ~]# cat /root/scripts/control-m_shutdown.sh
#!/bin/sh
  # Shutdown any running Control-M services
    STATUS=$(/usr/bin/systemctl is-active CTMLinux_Server.service)
    if [ ${STATUS} == "active" ]; then
      /usr/bin/systemctl stop CTMLinux_Server.service 
    fi

    STATUS=$(/usr/bin/systemctl is-active CTMDist_Server.service)
    if [ ${STATUS} == "active" ]; then
      /usr/bin/systemctl stop CTMDist_Server.service 
    fi

    STATUS=$(/usr/bin/systemctl is-active EnterpriseManager.service)
    if [ ${STATUS} == "active" ]; then
      /usr/bin/systemctl stop EnterpriseManager.service
    fi
exit 0


#!/bin/bash

# stop CONTROL-M
if [ -f /data00/ctmlinux/ctm_server/scripts/shut_ctm ]; then
  echo "Stopping CONTROL-M application"
  /data00/ctmlinux/ctm_server/scripts/shut_ctm
fi

# stop CONTROL-M Configuration Agent
if [ -f /data00/ctmlinux/ctm_server/scripts/shut_ca ]; then
  echo "Stopping CONTROL-M Server Configuration Agent"
  /data00/ctmlinux/ctm_server/scripts/shut_ca
fi

# stop database
/data00/ctmlinux/ctm_server/scripts/dbversion
if [ $? -ne 0 ] ; then
  echo "SQL Server is already stopped "
else
  if [ -f /data00/ctmlinux/ctm_server/scripts/shutdb ]; then
    echo "Stopping SQL server for CONTROL-M"
    /data00/ctmlinux/ctm_server/scripts/shutdb
  fi
fi

exit 0

-----Original Message-----
From: Adrian Klaver <adrian.klaver@aklaver.com> 
Sent: Wednesday, October 10, 2018 8:25 AM
To: Bryce Pepper <BPepper@KCSouthern.com>; pgsql-general@lists.postgresql.org
Subject: Re: RHEL 7 (systemd) reboot

This email originated from outside the company. Please use caution when opening attachments or clicking on links. If
yoususpect this to be a phishing attempt, please report via PhishAlarm.
 
________________________________

On 10/10/18 5:32 AM, Bryce Pepper wrote:
> Adrian,
> Thanks for the inquiry.  The function (db_execute_sql) is coming from a vendor (BMC) product called Control-M. It is
ascheduling product.
 
> The tmp file is deleted before I can see its contents but I believe it is trying to update some columns in the
CMS_SYSPRMtable.
 
> I also think the postgresql instance is already stopped and hence why the db_execute fails.  I will try to modify the
vendorfunction to save off the contents of the query.
 

Alright, I'm confused. In your earlier post you said the stop script is not running. Yet here it is, just not at the
righttime. I think a more detailed explanation is needed:
 

1) The stop script you are concerned about is a systemd  script, one that you created or system provided?

2) What is the shutdown service you refer to?

3) Is there a separate shutdown script for the Control-M product?

4) What do you expect to happen vs what is happening?

>
> Bryce
>
> p.s. Do you know of any verbose logging that could be turned on to catch when pgsql is being terminated?
>
>
> -----Original Message-----
> From: Adrian Klaver <adrian.klaver@aklaver.com>
> Sent: Tuesday, October 09, 2018 7:39 PM
> To: Bryce Pepper <BPepper@KCSouthern.com>; 
> pgsql-general@lists.postgresql.org
> Subject: Re: RHEL 7 (systemd) reboot
>
> This email originated from outside the company. Please use caution when opening attachments or clicking on links. If
yoususpect this to be a phishing attempt, please report via PhishAlarm.
 
> ________________________________
>
> On 10/9/18 11:06 AM, Bryce Pepper wrote:
>> I am running three instances (under different users) on a RHEL 7 
>> server to support a vendor product.
>>
>> In the defined services, the start & stop scripts work fine when 
>> invoked with systemctl {start|stop} whatever.service  but we have 
>> automated monthly patching which does a reboot.
>>
>> Looking in /var/log/messages and the stop scripts do not get invoked 
>> on reboot, therefore I created a new shutdown service as described 
>> here <https://unix.stackexchange.com/questions/211924/effect-of-reboot-signal-on-systemd-service-state>.
>>
>> It appears that PostGreSQL is receiving a signal from somewhere prior 
>> to my script running.
>>
>
>>
>> The database must be available for the product to shut down in a 
>> consistent state.
>>
>> I am open to suggestions.
>
> What is the below doing or coming from?:
>
> db_execute_sql failed while processing 
> /data00/ctmlinux/ctm_server/tmp/upd_CMS_SYSPRM_29448.
>
>>
>> Thanks,
>>
>> Bryce
>>
>> *Bryce Pepper*
>>
>> Sr. Unix Applications Systems Engineer
>>
>> *The Kansas City Southern Railway Company *
>>
>> 114 West 11^th Street  |  Kansas City,  MO 64105
>>
>> Office:  816.983.1512
>>
>> Email: bpepper@kcsouthern.com <mailto:bpepper@kcsouthern.com>
>>
>
>
> --
> Adrian Klaver
> adrian.klaver@aklaver.com
>


--
Adrian Klaver
adrian.klaver@aklaver.com