| AIX random IO error [message #394850] |
Mon, 30 March 2009 09:57  |
blake.hall
Messages: 3
Registered: March 2009
Location: Baytown, Texas
|
Junior Member |
|
|
I have a production AIX database server with 8 production 7X24 instances. On only one of the 8 instances, randomly a scheduled procedure fails with these errors:
ORA-01115: IO error reading block from file 10 (block # 94006)
ORA-01110: data file 10: '/u05/oracle/bupurp/aap_data.dbf'
ORA-27091: skgfqio: unable to queue I/O
ORA-27072: skgfdisp: I/O error
IBM AIX RISC System/6000 Error: 5: I/O error
The procedure populates a table with data from 2 local tables and a linked database table.
Oracle recommends upgrading the AIX to the newest patch level -we are at the most current patch level.... and applying an Oracle database patch {5496862}. Unfortunately all 8 production instances will have to be patched, since all are on the same Oracle home.
Not that I don't want to do the work, but I am looking for alternatives to applying this patch to these high activity instances, to minimize downtime.
Has anyone had a similar situation and found another approach? On one forum I saw a similar post and the recommended solution was to move the tablespace file to a different drive, on a different controller.
Thanks in advance for your help!
|
|
|
|
| Re: AIX random IO error [message #394856 is a reply to message #394850] |
Mon, 30 March 2009 10:12  |
ThomasG
Messages: 3211
Registered: April 2005
Location: Heilbronn, Germany
|
Senior Member |
|
|
Have you checked on the OS side if there are any errors, or any limits placed on the Oracle user account?
(ulimit)
Have you checked if there are any errors in the system log or STALE PPs in the disk volume group?
(errpt / lsvg)
|
|
|
|
| Re: AIX random IO error [message #394885 is a reply to message #394850] |
Mon, 30 March 2009 13:17 |
blake.hall
Messages: 3
Registered: March 2009
Location: Baytown, Texas
|
Junior Member |
|
|
No limitations on the Oracle user in the OS. [response was unlimited]
No obvious errors on the error report... response below:
/home/eyaba> errpt / lsvg
IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
DCB47997 0330134709 T H hdisk14 DISK OPERATION ERROR
F7FA22C9 0320032809 I O SYSJ2 UNABLE TO ALLOCATE SPACE IN FILE SYSTEM
F7FA22C9 0319070309 I O SYSJ2 UNABLE TO ALLOCATE SPACE IN FILE SYSTEM
F7FA22C9 0308101009 I O SYSJ2 UNABLE TO ALLOCATE SPACE IN FILE SYSTEM
C69F5C9B 0308030009 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
1FB4FD21 0302054209 I S fscsi0 CONFIGURATION MISMATCH
1FB4FD21 0302054209 I S fscsi0 CONFIGURATION MISMATCH
1FB4FD21 0302054209 I S fscsi1 CONFIGURATION MISMATCH
1FB4FD21 0302054209 I S fscsi1 CONFIGURATION MISMATCH
/home/eyaba>
The recent dates and times that the procedure failed were:
03-25-2009 6:18am
03-22-2009 11:43pm
03-20-2009 5:36am
03-18-2009 2:27am
Thanks for your help!
|
|
|
|
|
|
|
|
|
|
|
|
| Re: AIX random IO error [message #395237 is a reply to message #395236] |
Tue, 31 March 2009 16:41 |
ThomasG
Messages: 3211
Registered: April 2005
Location: Heilbronn, Germany
|
Senior Member |
|
|
In the smitty console under "system storage management" somewhere. Where exactly would depend on how your storage is set up. But since the file is already >2GB it should be enabled.
You should have used "ulimit -a" by the way to see all user limits.
Also, errpt und lsvg are to separate commands, too, one to search the system error log, the other to list volume groups.
Both have different options (in case of lsvg also different options depending on how your storage is set up), too, to see different things.
But since the hardware looks definitely broken, you should call your hardware support anyway.
1) Fix the hardware
2) Fix the volume groups / filesystems
and only then you have any chance of doing.
3) Fix the database
|
|
|
|
] 
Blog
Search
Help
Members
Register
Login
Home
