Symantec Backup Exec RALUS crashes on CentOS 7

We just deployed our first CentOS 7 machine and are trying to back it up using Backup Exec 2010 R3 and the RALUS agent.

After installing the missing compatibility libraries needed for the RALUS:

yum install compat-libstdc++-33.i686 compat-libstdc++-33.x86_64

the agent installs and starts but once the Media Server connects to it the agent crashes.

Some log digging came up with this from /var/log/messages:

Aug  7 16:09:41 localhost kernel: beremote[10898]: segfault at fffffffffffffffc ip 00007f7543fbc8cc sp 00007f75420c29d8 error 5 in libc-2.17.so[7f7543f3c000+1b6000]
Aug  7 16:09:41 localhost abrt-hook-ccpp: Saved core dump of pid 10894 (/opt/VRTSralus/bin/beremote) to /var/tmp/abrt/ccpp-2014-08-07-16:09:41-10894 (48660480 bytes)
Aug  7 16:09:41 localhost abrt-server: Package 'VRTSralus' isn't signed with proper key
Aug  7 16:09:41 localhost abrt-server: 'post-create' on '/var/tmp/abrt/ccpp-2014-08-07-16:09:41-10894' exited with 1
Aug  7 16:09:41 localhost abrt-server: Deleting problem directory '/var/tmp/abrt/ccpp-2014-08-07-16:09:41-10894'

and running the agent in debug mode shows this:

[root@localhost bin]# ./beremote --log-console
f8a1b740 Thu Aug  7 16:25:38 2014 : Starting BE Remote Agent
f8a1b740 Thu Aug  7 16:25:38 2014 : Requested no generation of log file
f8a1b740 Thu Aug  7 16:25:38 2014 : No configuration file specified.  Using default.
f8a1b740 Thu Aug  7 16:25:38 2014 : Log to console: enabled
f8a1b740 Thu Aug  7 16:25:38 2014 : Successfully set the supplementary groups of the process
f8a1b740 Thu Aug  7 16:25:38 2014 : Initialized locks for SSL callbacks
f8a1b740 Thu Aug  7 16:25:38 2014 : Starting NDMP processor
f8a1b740 Thu Aug  7 16:25:38 2014 : NDMPDMainThreadFunc spawned: grpid=1, tid=-231061760
f23a4700 Thu Aug  7 16:25:38 2014 : FS_InitFileSys
f23a4700 Thu Aug  7 16:25:38 2014 :   libbedsnt5.so could not be loaded: 0x       2 (2)
f23a4700 Thu Aug  7 16:25:38 2014 :   libbedssql2.so could not be loaded: 0x       2 (2)
f23a4700 Thu Aug  7 16:25:38 2014 :   libbedsxchg.so could not be loaded: 0x       2 (2)
f23a4700 Thu Aug  7 16:25:38 2014 :   libbedsxese.so could not be loaded: 0x       2 (2)
f23a4700 Thu Aug  7 16:25:38 2014 :   libbedsmbox.so could not be loaded: 0x       2 (2)
f23a4700 Thu Aug  7 16:25:38 2014 :   libbedspush.so could not be loaded: 0x       2 (2)
f23a4700 Thu Aug  7 16:25:38 2014 :   libbedsnote.so could not be loaded: 0x       2 (2)
f23a4700 Thu Aug  7 16:25:38 2014 :   libbedsmdoc.so could not be loaded: 0x       2 (2)
f23a4700 Thu Aug  7 16:25:38 2014 :   libbedssps2.so could not be loaded: 0x       2 (2)
f23a4700 Thu Aug  7 16:25:38 2014 :   libbedssps3.so could not be loaded: 0x       2 (2)
f23a4700 Thu Aug  7 16:25:38 2014 :   libbedsupfs.so could not be loaded: 0x       2 (2)
f23a4700 Thu Aug  7 16:25:38 2014 :   libbedsshadow.so could not be loaded: 0x       2 (2)
f23a4700 Thu Aug  7 16:25:38 2014 :   libbedsoffhost.so could not be loaded: 0x       2 (2)
f23a4700 Thu Aug  7 16:25:38 2014 :   loaded libbedsvx.so
f23a4700 Thu Aug  7 16:25:38 2014 :   loaded libbedsrman.so
f23a4700 Thu Aug  7 16:25:38 2014 :   loaded libbedssms.so
f23a4700 Thu Aug  7 16:25:38 2014 :   loaded libbedssmsp.so
f23a4700 Thu Aug  7 16:25:38 2014 :   libbedsra.so could not be loaded: 0x       2 (2)
f23a4700 Thu Aug  7 16:25:38 2014 :   libbedsdb2.so could not be loaded: 0x       2 (2)
f23a4700 Thu Aug  7 16:25:38 2014 :   loaded libbedsedir.so
f23a4700 Thu Aug  7 16:25:38 2014 :   libbedsvmesx.so could not be loaded: 0x       2 (2)
f23a4700 Thu Aug  7 16:25:38 2014 : Initializing FSs
f23a4700 Thu Aug  7 16:25:38 2014 : FS 1 failed to initialize: 0xE000FE46
f23a4700 Thu Aug  7 16:25:38 2014 : Function called: RMAN_InitFileSys
f23a4700 Thu Aug  7 16:25:38 2014 : Using 'UTF-8' Encoding.
f23a4700 Thu Aug  7 16:25:38 2014 : Using vfm path /opt/VRTSralus/VRTSvxms from config.
f23a4700 Thu Aug  7 16:25:38 2014 : Sucessfully set VFM_PRIVATE_ROOT env to /opt/VRTSralus/VRTSvxms.
f23a4700 Thu Aug  7 16:25:38 2014 : VFM_PRIVATE_ROOT was set with value /opt/VRTSralus/VRTSvxms
f23a4700 Thu Aug  7 16:25:38 2014 :      VXMS Initialization OK.
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <rootfs> mounted at </>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <proc> mounted at </proc>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <sysfs> mounted at </sys>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <devtmpfs> mounted at </dev>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <securityfs> mounted at </sys/kernel/security>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <tmpfs> mounted at </dev/shm>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <devpts> mounted at </dev/pts>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <tmpfs> mounted at </run>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <tmpfs> mounted at </sys/fs/cgroup>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <cgroup> mounted at </sys/fs/cgroup/systemd>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <pstore> mounted at </sys/fs/pstore>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <cgroup> mounted at </sys/fs/cgroup/cpuset>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <cgroup> mounted at </sys/fs/cgroup/cpu,cpuacct>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <cgroup> mounted at </sys/fs/cgroup/memory>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <cgroup> mounted at </sys/fs/cgroup/devices>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <cgroup> mounted at </sys/fs/cgroup/freezer>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <cgroup> mounted at </sys/fs/cgroup/net_cls>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <cgroup> mounted at </sys/fs/cgroup/blkio>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <cgroup> mounted at </sys/fs/cgroup/perf_event>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <cgroup> mounted at </sys/fs/cgroup/hugetlb>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <configfs> mounted at </sys/kernel/config>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <xfs> mounted at </>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <autofs> mounted at </proc/sys/fs/binfmt_misc>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <debugfs> mounted at </sys/kernel/debug>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <mqueue> mounted at </dev/mqueue>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <hugetlbfs> mounted at </dev/hugepages>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <nfsd> mounted at </proc/fs/nfsd>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <xfs> mounted at </boot>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <xfs> mounted at </var>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <rpc_pipefs> mounted at </var/lib/nfs/rpc_pipefs>
f23a4700 Thu Aug  7 16:25:38 2014 : Detected Mounted Filesystem: type <binfmt_misc> mounted at </proc/sys/fs/binfmt_misc>
f23a4700 Thu Aug  7 16:25:38 2014 : INFORMATIONAL: Zero value found for 'DisableRMAL' from ralus.cfg, allowing RMAL to initialize
f23a4700 Thu Aug  7 16:25:38 2014 : Successfully resolved the "ndmp" service to port: 10000 (host order)
f23a4700 Thu Aug  7 16:25:38 2014 : BETCPListener successfully installed a signal handler for SIGTERM
f23a4700 Thu Aug  7 16:25:38 2014 : BETCPListener::BETCPListener: This system appears to be a Dual IP system
f23a4700 Thu Aug  7 16:25:38 2014 : BETCPListener::BETCPListener: Successfully set the IPV6_V6ONLY option, this listener may behave as Dual Stack listener
f23a4700 Thu Aug  7 16:25:38 2014 : Started NDMP Listener on port 10000
f0dfd700 Thu Aug  7 16:25:48 2014 : NrdsAdvertiserThread: advertisement cycle started.
f0dfd700 Thu Aug  7 16:25:48 2014 : RMAN_EnumSelfDLE: AgentConfig GetOracleDBNames returned error. If Oracle Agent is installed, please run AgentConfig.
f0dfd700 Thu Aug  7 16:25:48 2014 : NrdsAdvertiserThread: EnumSelfDLE for file system 14 returned 0(0x0) and 0 DLEs
GetIfAddrs(LINUX): failed err = 11
GetAdaptersAddresses: error = 1, ret=-1
f0dfd700 Thu Aug  7 16:25:48 2014 : VX_RemoveDLE: DestroyDLE()
f0dfd700 Thu Aug  7 16:25:48 2014 : NrdsAdvertiserThread: EnumSelfDLE for file system 22 returned -1(0xFFFFFFFF) and 0 DLEs
f0dfd700 Thu Aug  7 16:25:48 2014 : NrdsAdvertiserThread: Security is enabled!!!
f0dfd700 Thu Aug  7 16:25:48 2014 : This instance of BETCPListener was not requested to install a signal handler and hence will not install one!
GetIfAddrs(LINUX): failed err = 11
GetAdaptersAddresses: error = 1, ret=-1
f0dfd700 Thu Aug  7 16:25:48 2014 : NrdsAdvertiserThread: connect to target=mediaserver.mydomain port=6101 failed
f0dfd700 Thu Aug  7 16:25:48 2014 : NrdsAdvertiserThread: Retrying in 60 seconds
e5ad8700 Thu Aug  7 16:25:53 2014 : NrdsAdvertiserThread: negative (purge) advertisement cycle started.
e5ad8700 Thu Aug  7 16:25:53 2014 : NrdsAdvertiserThread: no purge is pending.
e5ad8700 Thu Aug  7 16:25:53 2014 : NrdsAdvertiserThread: negative (purge) advertisement cycle complete.  Waiting 240 minutes before advertising again.
f0dfd700 Thu Aug  7 16:26:48 2014 : NrdsAdvertiserThread: Security is enabled!!!
f0dfd700 Thu Aug  7 16:26:48 2014 : This instance of BETCPListener was not requested to install a signal handler and hence will not install one!
GetIfAddrs(LINUX): failed err = 11
Segmentation fault (core dumped)

 

I’ve opened a case with Symantec and their answer was that CentOS isn’t supported and neither is RHEL7.

Anyone else running into this? Have you fixed it?

I found a blog post where someone suggested hex editing the beremote binary which I’d rather not do. Plus our version of the agent is newer than the one he describes in his post: http://blog.redweb.at/2012/08/howto-backupexec-2012-linux-agent-and-kernel-3-0-debian/

6 thoughts on “Symantec Backup Exec RALUS crashes on CentOS 7”

  1. Did you ever find a solution to this?  Have the same issue on a new box and came across your post with the same exact issue we have.

    Reply
    • No not yet. We’re relying on Snapshots of the VM taken via vSphere and our NetApp Storage.

      We did just recently upgrade to Backup Exec 2014. My co-worker told me the RALUS hasn’t been updated since 2012 though so I suspect the same problem exists.

      Reply
  2. I would try the libbesocket.so patch in that blog post you linked (now dead, try http://preview.tinyurl.com/j2le9ke instead) — the process for finding the place to change in newer versions of the library file is well-described in both the article and the comments there (i.e. find ‘$0x8938’ within the hex dump, find the offset of the nearest ‘jns’ op after that.

    If you’ve already got an alternative (i.e. your snapshots) then it shouldn’t present too much risk, and its probably a lot easier to manage than the patched kernel recommended by https://www.veritas.com/support/en_US/article.TECH201095

    Reply
    • I should have updated my post. The latest Linux Agent from BE 2014 works fine on CentOS 7.

      I’ll double check at work tomorrow but I’m pretty sure that’s what we ended up doing.

      Reply

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.