Aug 31

ORA-31617: unable to open dump file “/export/data.dmp” for write

This morning a user came to me because they were getting what appeared to be a Permission error

I checked the directory listed, /nfs/unix/export. It is mounted on both nodes of the cluster.

Checked file permissions, using touch I created a file as the oracle user, and deleted it, no permissions issues there.

It turns out it was the directory object itself! I looked at the job definition, and it was using the directory /export

Export is a link to the /nfs/unix/export directory. The second node had link, but not the first! What threw me off was the directory object itself was only pointing to /export, but the error message was pointing to the full NFS path. Dropping and re-creating the directory object in the database pointing to /nfs/unix/export/ resolve the issue.

Aug 31

RMAN-08137: WARNING: archived log not deleted, needed for standby or upstream capture process

On a non-production database I was getting the following error when trying to delete archive logs with RMAN.

This database does not have any golden gate processes associated with it, so why doesn’t Oracle want to delete the  archive log?

The database was rebuilt with a copy of a production database that did have golden gate running on it.

We can see the name of the extract that is still registered with the database.

Since the database was a copy of our production database that did have golden gate running, it till thinks there is a valid extract. Connecting to the database and unregistered the extract allows the archive log to be deleted.

 

Aug 31

ORA-02396: exceeded maximum idle time, please connect again

I was doing a schema import last night, over a database link. Lack of sleep had set in and I was getting the following error

Now if you aren’t tired this should be a pretty obvious fix, the idle_time value on the remote server was set to 60 and was disconnecting me during the index build section of the import. I altered the profile to give my user/profile unlimited idle_time. Problem solve, import complete.

Aug 07

RMAN-20036: invalid record order

Trying to register a database with my rman catalog I was getting the following error

Poking around metalink I found note 113325.1

Basicly it says there is a mismatch in the number of standby redo logs, but this database does not have a standby. However it was cloned from a database that had a standby, and I apparently forgot to remove the standby redo logs after I cloned it.

So cleanup the standby logfiles

I dropped and re-created the catalog

 

Mar 28

Now on SSL

I have moved this site over to a secure connection. LetsEncrypt offers free SSL certificates for your website! The setup was a bit of a pain because I have a non-standard configuration for this server, but after a little poking and prodding, we are now running on https.

Feb 20

Targets.xml was rejected: loaded with a wrong agent token

I noticed an agent on one of my database servers was down this morning. Agents crash from time to time, no big deal. I will just restart it

Well that sucks. Something is out of sync. Usually this will happen when your file system is filled up and a change cannot be written to the targets.xml. I wasn’t out of space though. I’m not quite sure what caused this, but I know how to fix it.

make a backup of the targets.xml file, blank out the file, and add the opening and closign targets tag

Then re-synchronize the agent from OMS, this will re-populate the targets.xml

it takes about another 10 minutes to resynchronize but while it is running the agent will look like this

After a little waiting around the agent is back to normal

 

Feb 03

Oracle OEM 13c LDAP error

My new OEM 13c system has been having lots of issues, I have multiple bugs registered with oracle. This one is particularly odd, trying to create a new user I get an LDAP error

The user actually does get created, but when I try to assign additional privileges to the user I get the error again, and this time it doesn’t actually add them.

The kicker is, I am not using LDAP, the user is created as an internal user, no single sign on!

I am trying to give a user access to the performance page, so off to emcli!!

I need to find the privileges related to the performance tab. It’s better to copy and paste these since they are case sensitive.

I want to give myUser DB_PERFORMANCE_VIEW and DB_PERFORMANCE_HOME_VIEW. I have my databases assigned to groups, so lets do that grant

I looped through my dev/test/prod groups

Checking inside OEM, the grants are now in place

I don’t understand what OEM is doing though the web interface that is difference from using emcli. I had another issue deploying plugins using the web interface. I was able to deploy them with emcli. I will have a post on that soon.

Jan 06

ohasd permissions

I was working kind of late last night, trying to get a new install working. I ran into a bug where the permissions on the ohasd were incorrect after patching GI. I went out to a working server to see what the permissions should be, build my chown and chmod statements and pasted them into my terminal window. Unfortunately I got them in the wrong terminal, and had managed to copy the wrong permissions. I changed the ownership on ohasd on the first node of my production RAC Cluster. Apparently the permissions are really important because the whole node went down.

A little bit of panic set in and I wasn’t sure what I had done. I didn’t realize i had pasted the permission statements into the wrong window, and the error messages weren’t very helpful.

I was getting nothing in the logs.

I figured it must be a permission issue, but I wasn’t quite sure what to reset them to.

Apparently I am not the first person to do this since Oracle has a document for fixing this!

How to check and fix file permissions on Grid Infrastructure environment (Doc ID 1931142.1)

I ran


rebooted the node, and all was right with the world, except for my ego being kind of damaged from making such a silly mistake.

Sep 30

myMYSQL Nightmare Continues

My fight this week with mySQL continues. I started getting really strange errors in the error log, and users couldn’t connect, not even root@localhost.

The error log showed

Followed by logins failing

What I think had happened, mySQL was upgraded on the OS by the linux admins. It doesn’t look like anything inside the database was upgrade.

The new version of mySQL uses the innodb engine (v5.6.15)

Whereas the older version of mySQL uses MyISAM (from another server v5.5.50)

mysql was expecting INNODB and found MyISAM

This should be a simple fix, just run the mysql_upgrade script. However the database is either down, or I can’t log in when the database is up.

So I had to shutdown the database, start it up skipping permissions, and change my root password

in another session I was now able to connect to the database

and update the root password

then killed off my instance, and started it back up

run the mysql_upgrade script

and restart once more

I don’t know what wrong with the passwords at this point, but none of the users could log in, so I changed all of their passwords to their current passwords. Just rewriting the password seemed to do the trick.

Everything seems to be back to normal…

 

 

Sep 28

Open Source Rant

%98 of my time I spend as an Oracle DBA. I am not limited just to Oracle though. I have used mySQL and SQL server extensively. Something that always drives me crazy about mySQL is the documentation. It is horrible. The fact that it is done as an open forum where people post and try and provide information is a big red flag to me. Just trying to figure out how a simple command works is incredibly frustrating. It assumes you know what you are doing. I do know what I am doing, but can still get tripped up when they don’t provide command flow diagrams. I can’t imagine how frustrating trying to use mysql must be for a more average user.

Older posts «