DataGuard数据库服务器硬盘故障处理一则

　　昨天一台PC Server上的数据库又出问题，同样是硬盘故障。

　　这两台服务器用的都是联志的国产低端PC Server，这些服务器的质量实在是差，上次一台备机的硬盘损坏，然后又有一台因为电源模块的问题反复重起，现在这一台服务器的硬盘再次出现问题。

　　Nov 24 10:27:48 wapcom1 kernel: attempt to access beyond end of device

　　Nov 24 10:27:48 wapcom1 kernel: 08:08: rw=0, want=1564747716, limit=5245191

　　Nov 24 10:27:48 wapcom1 kernel: EXT3-fs error (device sd(8,8)): ext3_readdir:

　　directory #128110 contains a hole at offset 2011258880

　　Nov 24 10:27:49 wapcom1 kernel: attempt to access beyond end of device

　　Nov 24 10:27:49 wapcom1 kernel: 08:08: rw=0, want=1564747716, limit=5245191

　　Nov 24 10:27:50 wapcom1 kernel: EXT3-fs error (device sd(8,8)): ext3_readdir:

　　directory #128110 contains a hole at offset 2011262976

　　Nov 24 10:27:50 wapcom1 kernel: attempt to access beyond end of device

　　Nov 24 10:27:50 wapcom1 kernel: 08:08: rw=0, want=1564747716, limit=5245191

　　Nov 24 10:27:50 wapcom1 kernel: EXT3-fs error (device sd(8,8)): ext3_readdir:

　　directory #128110 contains a hole at offset 2011267072

　　Nov 24 10:27:50 wapcom1 kernel: attempt to access beyond end of device

　　Nov 24 10:27:50 wapcom1 kernel: 08:08: rw=0, want=1564747716, limit=5245191

　　Nov 24 10:27:50 wapcom1 kernel: EXT3-fs error (device sd(8,8)): ext3_readdir:

　　directory #128110 contains a hole at offset 2011271168

　　好在数据库通过DataGuard可以切换到另外一台，没有数据损失：

　　Thu Nov 23 18:46:18 2006

　　ARC0: Complete FAL archive (thread 1 sequence 6045 destination bmarksb)

　　ARC0: Begin FAL archive (thread 1 sequence 6047 destination bmarksb)

　　Creating archive destination LOG_ARCHIVE_DEST_2: 'bmarksb'

　　ARC0: Complete FAL archive (thread 1 sequence 6047 destination bmarksb)

　　ARC0: Begin FAL archive (thread 1 sequence 6048 destination bmarksb)

　　Creating archive destination LOG_ARCHIVE_DEST_2: 'bmarksb'

　　Thu Nov 23 18:46:18 2006

　　ARC1: Complete FAL archive (thread 1 sequence 6046 destination bmarksb)

　　ARC1: Begin FAL archive (thread 1 sequence 6049 destination bmarksb)

　　Creating archive destination LOG_ARCHIVE_DEST_2: 'bmarksb'

　　Thu Nov 23 18:46:18 2006

　　ARC0: Complete FAL archive (thread 1 sequence 6048 destination bmarksb)

　　Thu Nov 23 18:46:18 2006

　　ARC1: Complete FAL archive (thread 1 sequence 6049 destination bmarksb)

　　现在是主库所在的服务器出现问题:

　　SQL> select dbid,name,PROTECTION_MODE,DATABASE_ROLE,SWITCHOVER_STATUS from v$database;

　　DBID NAME PROTECTION_MODE DATABASE_ROLE SWITCHOVER_STATUS

　　---------- --------- -------------------- ---------------- ------------------

　　3520694939 BMARK MAXIMUM PERFORMANCE PRIMARY SESSIONS ACTIVE

　　备库现在一切正常:

　　SQL> select dbid,name,PROTECTION_MODE,DATABASE_ROLE,SWITCHOVER_STATUS from v$database;

　　DBID NAME PROTECTION_MODE DATABASE_ROLE SWITCHOVER_STATUS

　　---------- --------- -------------------- ---------------- ------------------

　　3520694939 BMARK MAXIMUM PERFORMANCE PHYSICAL STANDBY SESSIONS ACTIVE

　　现在需要的是一点停机时间进行切换。

　　切换日志:

　　Fri Nov 24 11:30:43 2006

　　alter database commit to switchover to physical standby with session shutdown

　　Fri Nov 24 11:30:43 2006

　　ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY

　　Fri Nov 24 11:30:43 2006

　　SMON: disabling tx recovery

　　Fri Nov 24 11:30:44 2006

　　Active process 26743 user 'oracle' program 'oracle@wapcom1.hawa.cn (CJQ0)'

　　Active process 9033 user 'oracle' program 'oracle@wapcom1.hawa.cn (TNS V1-V3)'

　　Active process 7655 user 'oracle' program 'oracle@wapcom1.hawa.cn (TNS V1-V3)'

　　...............

　　Active process 8944 user 'oracle' program 'oracle@wapcom1.hawa.cn (TNS V1-V3)'

　　Active process 29104 user 'oracle' program 'oracle@wapcom1.hawa.cn (TNS V1-V3)'

　　Active process 30750 user 'oracle' program 'oracle@wapcom1.hawa.cn (TNS V1-V3)'

　　Active process 9045 user 'oracle' program 'oracle@wapcom1.hawa.cn (TNS V1-V3)'

　　CLOSE: waiting for server sessions to complete.

　　Fri Nov 24 11:31:51 2006

　　CLOSE: all sessions shutdown successfully.

　　Fri Nov 24 11:32:09 2006

　　SMON: disabling cache recovery

　　Fri Nov 24 11:32:10 2006

　　Shutting down archive processes

　　Archiving is disabled

　　Fri Nov 24 11:32:10 2006

　　ARCH shutting down

　　Fri Nov 24 11:32:10 2006

　　ARCH shutting down

　　Fri Nov 24 11:32:10 2006

　　ARC0: Archival stopped

　　Fri Nov 24 11:32:10 2006

　　ARC1: Archival stopped

　　Fri Nov 24 11:32:10 2006

　　Thread 1 closed at log sequence 6076

　　Successful close of redo thread 1

　　Fri Nov 24 11:32:28 2006

　　ARCH: noswitch archival of thread 1, sequence 6076

　　ARCH: End-Of-Redo archival of thread 1 sequence 6076

　　ARCH: Evaluating archive log 3 thread 1 sequence 6076

　　ARCH: Beginning to archive log 3 thread 1 sequence 6076

　　Creating archive destination LOG_ARCHIVE_DEST_2: 'bmarksb'

　　Creating archive destination LOG_ARCHIVE_DEST_1: '/var/oradata/arch/1_6076.arc'

　　ARCH: Completed archiving log 3 thread 1 sequence 6076

　　ARCH: archiving is disabled due to current logfile archival

　　Clearing standby activation ID 3520937155 (0xd1dd3cc3)

　　The primary database controlfile was created using the

　　'MAXLOGFILES 5' clause.

　　The resulting standby controlfile will not have enough

　　available logfile entries to support an adequate number

　　of standby redo logfiles. Consider re-creating the

　　primary controlfile using 'MAXLOGFILES 8' (or larger).

　　Use the following SQL commands on the standby database to create

　　standby redo logfiles that match the primary database:

　　ALTER DATABASE ADD STANDBY LOGFILE 'srl1.f' SIZE 10485760;

　　ALTER DATABASE ADD STANDBY LOGFILE 'srl2.f' SIZE 10485760;

　　ALTER DATABASE ADD STANDBY LOGFILE 'srl3.f' SIZE 10485760;

　　ALTER DATABASE ADD STANDBY LOGFILE 'srl4.f' SIZE 10485760;

　　Archivelog for thread 1 sequence 6076 required for standby recovery

　　MRP0 started with pid=8

　　MRP0: Background Managed Standby Recovery process started

　　Media Recovery Log /var/oradata/arch/1_6076.arc

　　Identified end-of-REDO for thread 1 sequence 6076

　　Identified end-of-REDO for thread 1 sequence 6076

　　Media Recovery End-Of-Redo indicator encountered

　　Media Recovery Applied until change 194025715

　　MRP0: Media Recovery Complete: End-Of-REDO

　　Resetting standby activation ID 3520937155 (0xd1dd3cc3)

　　MRP0: Background Media Recovery process shutdown

　　Fri Nov 24 11:32:35 2006

　　Switchover: Complete - Database shutdown required

　　Completed: alter database commit to switchover to physical st

　　Fri Nov 24 11:32:53 2006

　　Shutting down instance: further logons disabled

　　Shutting down instance (immediate)

　　License high water mark = 140

　　Fri Nov 24 11:32:53 2006

　　ALTER DATABASE CLOSE NORMAL

　　ORA-1507 signalled during: ALTER DATABASE CLOSE NORMAL...

　　ARCH: Archiving is disabled

　　Shutting down archive processes

　　Archiving is disabled

　　Archive process shutdown avoided: 0 active

　　ARCH: Archiving is disabled

　　Shutting down archive processes

　　Archiving is disabled

　　Archive process shutdown avoided: 0 active

　　Fri Nov 24 11:33:14 2006

　　Starting ORACLE instance (normal)

　　LICENSE_MAX_SESSION = 0

　　LICENSE_SESSIONS_WARNING = 0

　　SCN scheme 2

　　Using log_archive_dest parameter default value

　　LICENSE_MAX_USERS = 0

　　SYS auditing is disabled

　　Starting up ORACLE RDBMS Version: 9.2.0.6.0.

　　System parameters with non-default values:

　　processes = 150

　　timed_statistics = TRUE

　　shared_pool_size = 83886080

　　large_pool_size = 33554432

　　standby_archive_dest = /var/oradata/arch

　　fal_server = bmarksb

　　fal_client = bmark

　　log_archive_format = %t_%s.arc

　　...........

　　CJQ0 started with pid=8

　　Fri Nov 24 11:33:15 2006

　　ARCH: STARTING ARCH PROCESSES

　　ARC0 started with pid=9

　　ARC0: Archival started

　　ARC1 started with pid=10

　　Fri Nov 24 11:33:15 2006

　　ARCH: STARTING ARCH PROCESSES COMPLETE

　　Fri Nov 24 11:33:15 2006

　　ARC1: Archival started

　　Fri Nov 24 11:33:15 2006

　　ARC0: Thread not mounted

　　Fri Nov 24 11:33:15 2006

　　ARC1: Thread not mounted

　　Fri Nov 24 11:33:22 2006

　　alter database mount standby database

　　Fri Nov 24 11:33:26 2006

　　Successful mount of redo thread 1, with mount id 3559140162

　　Fri Nov 24 11:33:26 2006

　　Standby Database mounted.

　　Completed: alter database mount standby database

　　Fri Nov 24 11:33:29 2006

　　ALTER DATABASE RECOVER managed standby database disconnect

　　Attempt to start background Managed Standby Recovery process

　　MRP0 started with pid=12

　　MRP0: Background Managed Standby Recovery process started

　　Fri Nov 24 11:33:34 2006

　　Completed: ALTER DATABASE RECOVER managed standby database d

　　Fri Nov 24 11:33:34 2006

　　Media Recovery Waiting for thread 1 seq# 6077

　　Media Recovery Log /var/oradata/arch/1_6077.arc

　　Media Recovery Waiting for thread 1 seq# 6078

　　Media Recovery Log /var/oradata/arch/1_6078.arc

　　Media Recovery Waiting for thread 1 seq# 6079

　　看来以后不能再采购联志服务器了。