Sep 30

myMYSQL Nightmare Continues

My fight this week with mySQL continues. I started getting really strange errors in the error log, and users couldn’t connect, not even root@localhost.

The error log showed

Followed by logins failing

What I think had happened, mySQL was upgraded on the OS by the linux admins. It doesn’t look like anything inside the database was upgrade.

The new version of mySQL uses the innodb engine (v5.6.15)

Whereas the older version of mySQL uses MyISAM (from another server v5.5.50)

mysql was expecting INNODB and found MyISAM

This should be a simple fix, just run the mysql_upgrade script. However the database is either down, or I can’t log in when the database is up.

So I had to shutdown the database, start it up skipping permissions, and change my root password

in another session I was now able to connect to the database

and update the root password

then killed off my instance, and started it back up

run the mysql_upgrade script

and restart once more

I don’t know what wrong with the passwords at this point, but none of the users could log in, so I changed all of their passwords to their current passwords. Just rewriting the password seemed to do the trick.

Everything seems to be back to normal…



Sep 28

Open Source Rant

%98 of my time I spend as an Oracle DBA. I am not limited just to Oracle though. I have used mySQL and SQL server extensively. Something that always drives me crazy about mySQL is the documentation. It is horrible. The fact that it is done as an open forum where people post and try and provide information is a big red flag to me. Just trying to figure out how a simple command works is incredibly frustrating. It assumes you know what you are doing. I do know what I am doing, but can still get tripped up when they don’t provide command flow diagrams. I can’t imagine how frustrating trying to use mysql must be for a more average user.

Jun 17

Mind your spaces

I am trying to export a database, everything except for a single table. I need to send a copy of the database to the vendor, but they don’t need a table that takes up %90 of the space in the database due to it storing a massive number of BLOBs and CLOBs

This should be really simple

That syntax looks correct to me! Maybe I copy/pasted in a special character or something…. I type out the full command by hand and get the same error. I am stumped!


I worked the googles on the internet machine, and see a bunch of examples that look pretty much identical to mine. Then I notice it. I have a space after the “IN” key word.

Thwarted by spaces again! It seems trival, and kind of incorrect since the “IN” in a sql statement needs that space there.

Jun 08

Windows 10 – When You Can’t Be Trusted

Unfortunately some things require windows. Old interfaces that ensure you are running IE6, old proprietary administration software, and other things I don’t use on a daily basis. Windows 10 is one of the most frustrating versions of windows I have ever run. It pretty much assumes you are too stupid to have any idea what you are doing. I just spend an hour fighting with it to let me download a file that builds boot disks. It decided it wasn’t safe. I don’t know what that means since I have been using this product for years! In fact Edge assumes you are so stupid, I can’t get on most websites because it doesn’t assume they are safe! It also doesn’t think it is safe for me to install chrome! How anyone could use the pile of crap for day to day work baffles me. I am glad my work laptop is still Windows 7, where I can get a virus if I am stupid, but doesn’t try to baby proof everything!

Jun 08

Rushing = Stupid Mistakes

I am currently being asked to refresh a database. The end users are pissed because they asked for the database to be refreshed at noon yesterday, this was the first they had asked for it, and they were doing “really important things” and by 3pm were pissed that I hadn’t responded to them, let alone completed the work. The thing was, I took a vacation. I took a whole two days off to spend some time with my girlfriend and wasn’t checking my email. Never mind that we have a whole team of DBAs that can refresh a database, they had emailed me. What they should have done was open a proper ticket, and someone would have helped out even though they hadn’t planned and were now in a panic.

So After escalating it to their boss because I wasn’t providing them with “good service”, who in turn talked to my boss, who informed him how they had dropped the ball, and I would deal with it today, that they were being completely unprofessional. (My boss is a good guy, and sticks up for me).

So here is the technical part of this post. Ever DBA has cloned a database, thousands and thousands of databases, but when you are rushing you can make stupid errors. I just fought this error for 20 minutes

This should have been a no brainier, but in my rush I wasn’t fully paying attention to the error.

I was just issuing the command

My defaults for RMAN are using sbt ie tape, so for some reason the tape channels were being loaded. This causes an error. The solution is simple and should have been fairly immediate, just allocate disk channels.

Even simple things become very hard when you are rushing.

Jun 02


I run mainly on Macs, which means OS X. Unfortunately that means I don’t have access to the GUI of vmware. As of 6.0 U2, vmware has a very slick web UI built into it. Just look at the following article, all I had to do was ssh to my ESXi server and run:


Web Interface for ESXi without vCenter

Mar 29

ASM du -s

one of my biggest complaints about ASM is the lack of the ability to do the equivalent of “du -hs *” to view the size of all the directories. I found a little piece of shell script that does just this. It is extremely slow but it is very helpful!

Just change the first argment “dg=RECO” to the actual disc group you want to examine.

Mar 08

HOTSOS In Texas!

I am currently in Texas at the HOTSOS Symposium. I look forward to this conference every year. There are some brilliant people here, and it is a great place to learn new things. Kerry Osborn did a great live presentation yesterday, can’t wait to get home and try some of this stuff out. Also saw a great presentation by Tanel Poder.

The food down here is amazing too! We went to the Gas Monkey Bar and Grill last night and the food was amazing. I love traveling for work. I mean it is more fun to travel just for fun, but when I can learn something new, and have someone else pickup the majority of the cost, that is pretty amazing!

Feb 08


Currently I am changing passwords in several hundred databases, not that uncommon of a task for any DBA. Unfortunately it is time to rotate the dbsnmp password. Usually I would log into OEM after changing the password and manually update the password in the monitoring configuration for each and every database. This is fine if there are only a few passwords, but a huge pain when you are dealing with hundreds. The documentation oracle provides isn’t completely clear [1592390.1], at least it wasn’t to me at first.

First I would connect to the database as the sys user and change the password for dbsnmp

Then you connect to EM through emcli and update the monitoring credentials. For this post lets assume my Oracle RAC cluster has the name “MyRAC” which contains two nodes “MyRAC1” and “MyRAC2”. We don’t really need to know the individual instance names since the dbsnmp user is shared between both nodes.

Connect to the repository

Now you can change the password via emcli

Where I ran into trouble was the last argument, if you look at the monitoring configuration page for a database, it becomes clear what is being asked for


The role is the same role you would select on the monitoring setup page, Normal or SYSDBA.

The syntax is slightly different if you are doing a single instance, but basically the same idea

So I am going to modify the change password proceedure to spool out these commands and just run them against OEM when I have completed changing the passwords on the databases.

Dec 07


I would just like to point out that I am a guy, and not a very pretty one! Sexy DBA refers to the job, and the technology not myself! I would be one of the ugliest women alive! I am 6’6″, bald, and have a beard!

Being a DBA is very far from sexy. It was supposed to be ironic, I guess that wasn’t completely clear!


Older posts «