OraFAQ Forum: Server Administration » Logs to Check When Db Gets Completely Crashed

Home » RDBMS Server » Server Administration » Logs to Check When Db Gets Completely Crashed (Oracle 10g)

Show: Today's Messages :: Polls :: Message Navigator
E-mail to friend

Logs to Check When Db Gets Completely Crashed [message #346547]

Mon, 08 September 2008 17:23

rpp_rp
Messages: 2
Registered: September 2006

Junior Member

Hi Everybody,

About two weeks back, our production Database was down and it took whole DBA Team around 4 hours to bring it up.

Can anyone suggests which are the logs we need to Check ( I am not pure DBA ..but, rather Mgmt Side and needs to provide the independent third party report on this ) ?

I know that

- Automatic Db Diagnostic Monitor ( ADDM) and STATPACK Report would provide useful info on the current state of the DB

- However, what I need to know is what happened at the time of the Db Breakdown and how we can prevent it further
based on Updates in Standard DB Admin Policies.

Any / ALL Inputs are highly Appreciated.

Thanks,

Regards,

- Ann

Report message to a moderator

Re: Logs to Check When Db Gets Completely Crashed [message #346548 is a reply to message #346547]

Mon, 08 September 2008 17:35

BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal

Senior Member

http://www.orafaq.com/forum/t/88153/0/
Please read & follow posting guidelines as stated in URL above

What OS name & version?
Which version of Oracle to 4 decimal places?

Besides the Oracle DB, what else runs on this system?

How long had the DB been up & stable prior to the outage?

> how we can prevent it further
If you have no clue as to what triggered the outage,
then most likely it could happen again, again.

>it took whole DBA Team around 4 hours to bring it up.
Why?
What did they say they did?
What was (or was not happening) during that 4 hours?
Was it 3 hours 59 minute looking for the correct command?

Post the contents of alert_SID.log file from 30 minutes before the start until 30 minute after the resolution.

Report message to a moderator

Re: Logs to Check When Db Gets Completely Crashed [message #346628 is a reply to message #346548]

Tue, 09 September 2008 03:12

Mohammad Taj
Messages: 2412
Registered: September 2006
Location: Dubai, UAE

Senior Member

...Was it 3 hours 59 minute looking for the correct command?...

Nice Joke Smile

but may be it is true...

Report message to a moderator

Re: Logs to Check When Db Gets Completely Crashed [message #346696 is a reply to message #346547]

Tue, 09 September 2008 07:24

varu123
Messages: 754
Registered: October 2007

Senior Member

>>Was it 3 hours 59 minute looking for the correct command?

Laughing

That is too much Ana.

The OP must have scared.

[Updated on: Tue, 09 September 2008 07:26]

Report message to a moderator

Re: Logs to Check When Db Gets Completely Crashed [message #346837 is a reply to message #346548]

Tue, 09 September 2008 17:46

rpp_rp
Messages: 2
Registered: September 2006

Junior Member

Thanks !

#1 SUN Solaris 4.0 , Oracle 10.2.0.2
#2 Its SAP CRM 4.0 on this Oracle DB
#3 DB was up and running well since 10 months untill this
Crash happened

#4 Ofcourse, DB Team must have some idea at least, what caused
this outage - Based upon the available trace and log files
I need to give independent assessment to Sr Management wit
hout any interaction with DBA Team ..They have already put
their defense at the Sr Management Hearing!!

#5 Well ..what the DBA Team were doing till 3hr 59Min is the
Good Q and that's where again I come in to picture to provide
the analytical report that what took so long to restore it -
Lack of Technical Competency was the Core Factor ( yes, it
is logical to say that such long time to restore the Prod
Db is not acceptable ) or the
Overall Poor DB Admin Policy ( or more precisely - the lack
of such Policy ) was also major contributor to it.

#6 My Task is at the very initial phase and as said I am on IT
Mgmnt Side and that's why I needed some specific trace files
and Log Files before I move to hands on level. Under this
back drop I don't have as of now any trace / log file.

I hope I provided enough details/ background of My Q.

Thanks for your input regarding trace file : The alert_SID.log ..

Any other log file , its' location - is highly appreciated.

Thanks,
- Ann

Report message to a moderator

Re: Logs to Check When Db Gets Completely Crashed [message #346839 is a reply to message #346547]

Tue, 09 September 2008 18:02

BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal

Senior Member

>SUN Solaris 4.0
Please check & double check this because AFAIK not a reasonable value.
post the output from command line below

uname -a

I can't/won't even begin to guess without some actual evidence.

Again the Oracle alert log from before the failure began until after DB was available.
If during that time frame any "trace files" *trc file were created,
then they may contain additional evidence.
My Solaris memory is somewhat faded, but the OS "messages" file where Solaris logs its activity for the same time frame as above.

What make/model of disk storage is under the DB? (NAS,SAN, etc.?)
what flavor of file system held Oracle data files?
Does oracle RDBMS run in archive mode?
In nice round numbers, about how big is this DB (total disk space consumed)?

I can pull the plug on Linux server, plug it back in the receptacle
& Oracle will be available in under 5 minutes

After you post the logs, then there could be more info requested.

Report message to a moderator

Re: Logs to Check When Db Gets Completely Crashed [message #346840 is a reply to message #346837]

Tue, 09 September 2008 18:08

Mahesh Rajendran
Messages: 10707
Registered: March 2002
Location: oracleDocoVille

Senior Member
Account Moderator

Initial post said
>>our production Database was down and it took whole DBA Team around 4 hours to bring it up.
Recent post says
>>time to restore the Prod
>>Db is not acceptable
Can I assume that the database was restored from backup?

>>Lack of Technical Competency was the Core Factor
That in most case is just a typical over statement from the "Mgmnt Side".
Lack of technical competency could have easily identified during normal business operations.
And to find the whole team is "lacking" is pure management disaster Smile

.
Well, ultimately the job was done. Right?
I would first refer back to
the SLA, which should have some time estimate to get the database "restored"
which should be supported by a documented and tested procedure.

Report message to a moderator

Re: Logs to Check When Db Gets Completely Crashed [message #346842 is a reply to message #346547]

Tue, 09 September 2008 18:31

BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal

Senior Member

Was there are hardware failure?
Was the database restored from backups?
If so, a 4 hour restoration process is not unreasonable; depending upon amount of data & whether tape drives were involved.
Every site should have documented & tested recovery procedures.
Does the type of failure you most recently experienced, correspond to any scenario in YOUR Recovery Procedures?
Each different failure scenario has a different Mean Time To Recover (MTTR).
In some cases MTTR should be measured in minutes.
In other cases MTTR could be measured in days.

More than likely there is more than enough blame to be shared by all involved.

From 1 perspective, you should be glad the database is back online.
Politics will decide whether 4 hours is reasonable or not.

Keep in mind, that nothing is impossible for the person who does NOT have to do it!

[Updated on: Tue, 09 September 2008 20:36] by Moderator

Report message to a moderator

Previous Topic:	Patch Apply on Prod with Standby
Next Topic:	Stored Outline

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Mon Jul 08 00:21:21 CDT 2024