Showing posts with label Oracle Cloud Control 12c. Show all posts
Showing posts with label Oracle Cloud Control 12c. Show all posts

Saturday 8 November 2014

EM 12.1.0.4: Agents Failed to Upload with EMD upload error:full upload has failed: Upload Recent Sending Statistics

Problem

After upgrading the OEM to 12.1.0.4, some of the agents failed to upload the files.
$ emctl upload agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload error:full upload has failed: Upload Recent Sending Statistics
________________________________

Avg send time=0.0 milliseconds
Backoff Event List
________________________________
Backoff event : channel=Metadata expiration at 2014-09-02 12:16:19 CDT
Backoff event : channel=Metadata expiration at 2014-09-02 12:16:21 CDT
Backoff event : channel=Metadata expiration at 2014-09-02 12:16:15 CDT
Backoff event : channel=Metadata expiration at 2014-09-02 12:16:17 CDT
Backoff event : channel=Metadata expiration at 2014-09-02 12:16:42 CDT
Backoff event : channel=Metadata expiration at 2014-09-02 12:17:17 CDT
Upload Failure List
_______________________________
Connection Event List
_____________________________
Upload timed out before completion.
Number of files to upload before the uploadNow call: 53, total size (MB): 0.061219215
Remaining number of files to upload: 61, total size (MB): 0.06959915 (TIMEOUT)

Cause

Issue is due to a bug on the OMS side.

BUG 19458672 - High contention from metaver upgrade callback if default metric keys don't exis

Due to this bug, repository database is heavily loaded and wont accept files from the Agents.
 

Solution

Apply Patch 19458672 on OMS Home as per the readme instructions.
 

References

Metalink Doc : Doc ID 1932205.1

EMD upload error:full upload has failed: Upload Recent Sending Statistics

Problem:

bash-3.2$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 12.1.0.3.0
OMS Version : 12.1.0.3.0
Protocol Version : 12.1.0.1.0
Agent Home : /u01/app/oem12cagent/agent_inst
Agent Binaries : /u01/app/oem12cagent/core/12.1.0.3.0
Agent Process ID : 6635
Parent Process ID : 1855
Agent URL : https://flex-bip-prod-dc-01:3872/emd/main/
Repository URL : https://osb-prod-dc-1:1159/empbs/upload
Started at : 2014-10-29 01:53:57
Started by user : oracle
Last Reload : (none)
Last successful upload : 2014-11-06 15:55:40
Last attempted upload : 2014-11-09 10:10:55
Total Megabytes of XML files uploaded so far : 20.22
Number of XML files pending upload : 4,925
Size of XML files pending upload(MB) : 2.9
Available disk space on upload filesystem : 75.90%
Collection Status : [COLLECTIONS_HALTED(
UPLOAD SYSTEM Threshold - unable to purge files in upload system)]
Backoff Expiration : 2014-11-09 10:13:36
Heartbeat Status : Ok
Last attempted heartbeat to OMS : 2014-11-09 10:11:25
Last successful heartbeat to OMS : 2014-11-09 10:11:25
Next scheduled heartbeat to OMS : 2014-11-09 10:12:25

---------------------------------------------------------------
Agent is Running and Ready
==========
bash-3.2$ ./emctl upload agent
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload error:full upload has failed: Upload Recent Sending Statistics
________________________________

Avg send time=0.0 milliseconds
Backoff Event List
________________________________
Backoff event : channel=Response Severity expiration at 2014-11-09 12:33:08 BDT
Backoff event : channel=Response Severity expiration at 2014-11-09 12:33:07 BDT
Backoff event : channel=Response Severity expiration at 2014-11-09 12:33:09 BDT
Upload Failure List
_______________________________
Connection Event List
_____________________________
Upload timed out before completion.
Number of files to upload before the uploadNow call: 4975, total size (MB): 2.9422493
Remaining number of files to upload: 4975, total size (MB): 2.9422493 (TIMEOUT)
bash-3.2$


SOLUTION:

bash-3.2$ ./emctl secure agent
Oracle Enterprise Manager Cloud Control 12c Release 2  
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Agent successfully stopped...   Done.
Securing agent...   Started.
Enter Agent Registration Password : 
Agent successfully restarted...   Done.
EMD gensudoprops completed successfully
Securing agent...   Successful.

bash-3.2$./emctl upload
Oracle Enterprise Manager Cloud Control 12c Release 2  
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

Sunday 22 September 2013

Step by Step Oracle Enterprise Manager Cloud Control 12c Release 3 Installation on Oracle Linux 5.9 and 6.4

Oracle Enterprise Manager Cloud Control 12c Release 3 Installation on Oracle Linux 5.9 and 6.4

Enterprise Manager Cloud Control is the new name for what was previously know as Enterprise Manager Grid Control. Depending on the source of the material, you may still see the product described as Grid Control. This article describes the installation of Oracle Enterprise Manager Cloud Control 12c Release 3 on Oracle Linux 5.9 and 6.4. (x86_64). This is the first Cloud Control release capable of monitoring the Oracle 12c database.
Note. Oracle Database 12c is not currently a supported release for use as a management repository database. For this reason, this article uses Oracle 11gR2 (11.2.0.3) as the repository database.
Related articles.

Software

Download the following software:
Note. Previous versions of the database can be used to hold the repository, but anything earlier than 11.2.0.2 will require additional patches.

OS Installation

Install Oracle Linux (OL) 5.9 or 6.4 in the same way you would for a regular Oracle Database installation. You can see examples of these types of installations below. Remember to check the relevant database installation article (listed below) for the specific package group selection.
During this installation I used a virtual machine with 6G RAM and 50G disk space. The swap size was set at 8G, the firewall was disabled and SELinux was set to permissive.

Database Installation

Use one of the following articles to install the 11.2.0.3 database.
The installation documentation says the following packages are necessary for the cloud control installation. If you have performed the database installation as described in one of the above articles, most of these prerequisites will already have been met.
# OL 5.x and 6.x
yum install make-3* -y
yum install binutils-2* -y
yum install gcc-4* -y
yum install libaio-0* -y
yum install glibc-common-2* -y
yum install libstdc++-4* -y
yum install sysstat-* -y
yum install glibc-devel -y

# OL 6.x only
yum install glibc-devel.i686 -y (not installed already)

# OL 5.x only
yum install glibc-devel.i386 -y
yum install setarch -y
yum install rng-utils-2* -y
yum install libXtst-1* -y
yum install xorg-x11-utils -y
If you have performed a default database installation you will need to deconfigure Enterprise Manager Database Control. Run the following command as the “oracle” user.
$ emca -deconfig dbcontrol db -repos drop -SYS_PWD  -SYSMAN_PWD 
Make the following initialization parameter changes and restart the instance.
sqlplus / AS SYSDBA

ALTER SYSTEM SET processes=300 SCOPE=SPFILE;
ALTER SYSTEM SET session_cached_cursors=200 SCOPE=SPFILE;
ALTER SYSTEM SET sga_target=2G SCOPE=SPFILE;
ALTER SYSTEM SET shared_pool_size=600M SCOPE=SPFILE;
ALTER SYSTEM SET pga_aggregate_target=1G SCOPE=SPFILE;
ALTER SYSTEM SET job_queue_processes=20 SCOPE=SPFILE;

-- May be required if using older versions of DB.
--ALTER SYSTEM SET log_buffer=10485760 SCOPE=SPFILE;
--ALTER SYSTEM SET open_cursors=300 SCOPE=SPFILE;

-- Restart the instance.
SHUTDOWN IMMEDIATE
STARTUP
If you have done a default installation as described here, the UNDO tablespace will be autoextensible. If you have done a custom installation make sure the UNDO tablespace is at least 200M in size. Also, make sure you have 3 redo logs of at least 300M to prevent a warning during the installation.
The database installation is now complete.

Cloud Control 12c Installation

Make the following directories to hold the management server and agent.
$ mkdir -p /u01/app/oracle/oms12cr3
$ mkdir -p /u01/app/oracle/agent12cr3
Unzip the Cloud Control media, the start the installation by running the “runInstller” script.
$ mkdir em12cr3
$ unzip -d em12cr3 em12103_linux64_disk1.zip
$ unzip -d em12cr3 em12103_linux64_disk2.zip
$ unzip -d em12cr3 em12103_linux64_disk3.zip
$ cd em12cr3

$ ./runInstaller
If you wish to receive support information, enter the required details, or uncheck the security updates checkbox and click the “Next” button. Click the “Yes” button the subsequent warning dialog.
My Oracle Support Details
If you wish to check for updates, enter the required details, or check the “Skip” option and click the “Next” button.
Software Updates
If you have performed the prerequisites as described, the installation should pass all prerequisite checks. Click the “Next” button.
Prerequisite Checks
Select the “Create a new Enterprise Manager System” and “Simple” options, then click the “Next” button.
Install Types
Enter the middleware and agent locations, then click the “Next” button.
Installation Details
Enter the administrator password and database repository details, then click the “Next” button.
Configuration Details
On the first warning dialog, click the “Yes” button to disable the stats gathering job.
Repository Warning 1
If you have any additional warnings, check they don’t look like show-stoppers, then click the “OK” button to continue.
If you are happy with the review information, click the “Install” button.
Review
Wait while the installation and configuration take place.
Installation Progress Details
When prompted, run the root scripts, then click the “OK” button.
Root Scripts
Make note of the URLs, then click the “Close” button to exit the installer. A copy of this information is available in the “/u01/app/oracle/Middleware/oms/install/setupinfo.txt” file.
Finish
The login screen is available from a browser using the URL provided in the previous screen (“https://ol6-em12cr3.localdomain:7803/em”). Log in with the username “sysman” and the password you specified during your installation.
Login
Once logged in, you are presented with a with the “License Agreement” screen. Click the “I Accept” button and you are presented with the homepage selector screen. On the right side of the screen it lists the post-installation setup tasks you need to work through. I have these documented in a separate article. Select the desired homepage (I chose Summary).
Homepage Selector
You are presented with the selected screen as the console homepage.
Console

Startup/Shutdown

Use the following commands to turn on all components installed by this article.
#!/bin/bash
export ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/db_1
export OMS_HOME=/u01/app/oracle/oms12cr3/oms
export AGENT_HOME=/u01/app/oracle/agent12cr3/core/12.1.0.3.0

# Start everything
$ORACLE_HOME/bin/dbstart $ORACLE_HOME

$OMS_HOME/bin/emctl start oms

$AGENT_HOME/bin/emctl start agent
Use the following commands to turn off all components installed by this article.
#!/bin/bash
export ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/db_1
export OMS_HOME=/u01/app/oracle/oms12cr3/oms
export AGENT_HOME=/u01/app/oracle/agent12cr3/core/12.1.0.3.0

# Stop everything
$OMS_HOME/bin/emctl stop oms -all

$AGENT_HOME/bin/emctl stop agent

$ORACLE_HOME/bin/dbshut $ORACLE_HOME

Oracle Enterprise Manager Cloud Control 12c Post-Installation Setup Tasks

Setup Software Library

  • Create a directory to use as the software library.
    $ mkdir -p /u01/app/oracle/oms12cr2/software_library
  • Navigate to the “Software Library: Administration” screen using the menu at the top-right of the screen (Setup > Provisioning and Patching > Software Library).
  • Select the storage type of “OMS Agent Filesystem”.
  • Click the “Add +” button.
  • Enter a name and use the lookup buttons to select the local agent and location on the file system for the software library. Once you’ve selected the appropriate values, click the “OK” button.
  • The software library is now configured.

Set My Oracle Support (MOS) Credentials

  • Navigate to the “My Oracle Support Preferred Credentials” screen using the menu at the top-right of the screen (Setup > My Oracle Support > Set Credentials…).
  • Enter the credentials and click the “Apply” button.

Download Additional Agents

  • Navigate to the “Self Update” screen using the menu at the top-right of the screen (Setup > Extensibility > Self Update).
  • Click on the “Check Updates” button and “OK” on the subsequent message dialog.
  • Click on the “Agent Software” link.
  • Highlight the agent of interest and click the “Download” button. Select the download schedule and click the “Select” button. Click the “OK” button on the confirmation dialog.
  • Click the refresh button on the top-right of the screen until the download is complete and the status changes to “Downloaded”.
  • Highlight the newly downloaded software and click the “Apply” button, followed by the “OK” button on the two following message dialogs.
  • When the status changes to “Applied”, the agent software is ready for installation on a target.

Install an Agent on a Target Host

  • Navigate to the “Add Targets Manually” screen using the menu at the top-right of the screen (Setup > Add Target > Add Targets Manually).
  • Accept the “Add Host Targets” option by clicking the “Add Host…” button.
  • Click the “Add +” button.
  • Enter the host and platform, then click the “Next” button.
  • Enter the installation details and click the “Next” button.
    Installation Base Directory  : /u01/app/oracle/agent12cr2
    Instance Directory           : /u01/app/oracle/agent12cr2/agent_inst (default)
    Named Credential	     : (click the "+" button and add the credentials of the "oracle" user)
    Privileged Delegation Setting: (leave blank)
    Port                         : 3872
    If you are installing the agent on a HP Service Guard package, remember to set the “Additional Parameters” to point at the package-specific inventory location and override the machine name with the package name. For example.
    INVENTORY_LOCATION=/u07/app/oraInventory ORACLE_HOSTNAME=my-package.example.com
  • Check the information on the review screen and click the “Deploy Agent” button.
  • Wait while the installation takes place. The “Add Host Status” page refreshes every 30 seconds.
  • When the installation completes, run the specified “root.sh” script and click the “Done” button.
  • The host will now be visible on the “Targets > Hosts” screen.

Discover Targets on Host

  • Navigate to the “Add Targets Manually” screen using the menu at the top-right of the screen (Setup > Add Target > Add Targets Manually).
  • Select the “Add Non-Host Targets Using Guided Process (Also Adds Related Targets)” option, select the target types to be discovered (eg. Oracle Database, Listener and Automatic Storage Manager) and click the “Add Using guided Discovery…” button.
  • Enter the host name and click the “Continue” button.
  • Click the “Configure” icon for any discovered targets and enter the required details. If you are using HP Service Guard, remember to only select and configure targets belonging to the package. By default, the agent will discover all targets on the physical machine.
  • When all the configuration steps are complete, click the “Finish” button, followed by the “Save” button, then finally the “OK” button.
  • The targets will now be listed on the relevant target screen (Targets > Databases).

Add Administrator Users

  • Navigate to the “Administrators” screen using the menu at the top-right of the screen (Setup > Security > Administrators).
  • Click the “Create” button.
  • Enter the username/password details and check the “Super Administrator” checkbox, then click the “Review” button.
  • Click the “Finish” button.

Notifications

Setup and maintenance of notifications has changed considerably in Cloud Control 12c compared to previous versions of Grid Control. There are several areas to consider when configuring and diagnosing notification issues.

Changing the Management Agent Time Zone

You can reset the time zone used by the Management Agent using the following command:
cd $AGENT_HOME/bin

[oracle@testsrv bin]$ ./emctl resetTZ agent
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Updating /u01/app/oracle/agent12cr2/agent_inst/sysman/config/emd.properties...
Successfully updated /u01/app/oracle/agent12cr2/agent_inst/sysman/config/emd.properties.
Login as the em repository user and run the  script:
exec mgmt_target.set_agent_tzrgn('testsrv:3872','Asia/Dacca')
and commit the changes
This can be done for example by logging into sqlplus and doing
SQL> exec mgmt_target.set_agent_tzrgn('testsrv:3872','Asia/Dacca')
SQL> commit
[oracle@testsrv bin]$

To see status of agent

[oracle@testsrv bin]$ ./emctl config agent getTZ
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Asia/Dacca

UnInstall Oracle 12c Cloud Agent

go to 12c AGENT_HOME../u01/app/oracle/agent12cr2/core/12.1.0.3.0/oui/bin

execute the runInstaller.sh and select the installed software one by one and uninstall these.