Monday, 7 January 2013

VIOS2.2 UPGRADE

Table of Contents

1       Introduction.................................................................................................................................................................. 4
1.1       Assumptions............................................................................................................................................................. 4
1.2       Command Acronyms............................................................................................................................................. 4
2       Pre-implementation tasks..................................................................................................................................... 5
3       Upgrading to Fix Pack 24 for VIOS 2.2............................................................................................................. 7
The current level of the VIOS must be v2.1.2.1.0-FP22................................................................................... 7
4       Upgrading to VIOS 2.2 Fix Pack 24 to SP01..................................................................................................... 8
The current level of the VIOS must be 2.2.10-FP24........................................................................................... 8
5       Installing Interim Fix IZ91190 for VIOS 2.2.0.10-FP24 SP01.................................................................... 9
The current level of the VIOS must be 2.2.0.11-FP24-SP01........................................................................... 9
6       VIOS SDDPCM & MPIO Updates.............................................................................................................................. 10
7       Installing Interim SAN Fix IZ94290m03 for VIOS 2.2.0.10-FP24 SP01................................................ 11
The current level of the VIOS must be 2.2.10-FP24-SP01............................................................................. 11
8       VIOS Changes................................................................................................................................................................. 13
8.1       NTP (if you find this is already in place, please ignore)............................................................ 13
9       VIOS ITM Updates......................................................................................................................................................... 13
9.1       Check ITM file sets are at correct levels.......................................................................................... 13
9.2       VIOS ITM Premium Agent (DO only if missing base & premium).................................................. 15
     9.3      Converting the audit File system   from  JFS to JFS2
10         Post Updates Checks........................................................................................................................................... 16

1          Introduction


The following procedure details tasks required for upgrading VIO Servers to supported levels for POWER7.

 This document covers upgrade path for VIOS Servers running v2.1.2.1.0-FP22” to bring them in line with changes & software levels released in VIOS gold build image “VIOS_22011_FP24_SP01_mksysb”.

Upgrade software target levels are:-

·         VIOS v2.2.0.11, fixpack 24 with Service Pack 01 (VIOS v2.2.0.11-FP24-SP01)
·         Interim Fix IZ91190 for VIOS v2.2.0.10-FP24 SP01 (LACP on Etherchannel on SEA)
·         VIOS SDDPCM v2.6.03
·         VIOS host attachment script ( MPIO) v1.0.0.21
·         Apply  the SAN Efix IZ94290
·         VIOS environment changes (Locale & NTP)
·         VIOS ITM Updates


Additional configuration changes required:-

·         NTP Configuration
·         “PADMIN”  user locale setting

1.1        Assumptions


It is assumed that the implementer has the following skills required in order to complete these upgrades successfully:-

                VIOS and virtualisation skills
                Access to Unix Environments
                Knowledge of USPP Infrastructure
                Knowledge of CSM servers, HMC, NIM, VIO

1.2       Command Acronyms


Description of acronyms used throughout upgrade:

# - represents root shell command
$ - represents padmin shell command

oem_setup_env (exit “padmin restricted” shell to  aix shell)

<VIO>                                = represents padmin shell command
<VIO-ROOT>                = represents root shell in VIO Server
$ CSM                 = CSM Server which manages VIO server



2         Pre-implementation tasks


Minimum disc space requirement has changed for Fix Pack 24 (VIOS 2.2.0.10-FP-24
Fix Pack 24 (VIOS 2.2.0.10-FP-24). VIOS now requires a minimum space of 30GB in the rootvg volume group. Please ensure that your rootvg contains at least 30GB before you attempt to install or upgrade to Fix Pack 24.

                <VIO-ROOT>    #lsvg rootvg |grep 'FREE'

 Recommended Memory and Paging for VIO servers (mem0ry 2 GB ,paging 2 G  )
        
              <VIO-ROOT>      # lsattr -El mem0 -a goodsize |awk '{print $1,$2}';lsps -s |tail -1

goodsize 2048
                                                2048MB

1.            Verify HMC is accessible and console session for VIOs can be opened on previous working day and you have a valid “padmin” password.
               
                <CSM>                 # rconsole –rtn <vio-csm>

2.            Health checks all VIO clients and VIO Servers for any hardware/software errors

<VIO-ROOT>    # errpt ;lsvg –l rootvg; lsdev –Cc adapter; lspath,
# lsdev –Ccadapter |grep Virtual (make sure the virtual adapters are available)
  
3.            Run viosaudit3.sh script against each VIOS to capture and verify configuration from the CSM server.

<CSM>                 # /usr/local/bin/viosaudit3.sh <viohostname-csm>

4.            Create a mksysb backup of the VIOS from the CSM.

<CSM> #/usr/local/bin/vios_backup.ksh <viohostname-csm> <csmhostname-csm> -mksysb
                                               
5.            Verify Mksysb on the CSM

<CSM>                 # cd /viosbackups/<node-csm>
               <CSM> # cat <node-csm.mksysb.out>
<CSM>                 # listvgbackup –f < node-csm.mksysb>

6.            Take a copy of the VIO mapping information and configuration:
Set an alias to run lsmap from root shell in the VIO

                 <VIO-ROOT>   # mount-o soft $CSM:/NIM/vios/vio2.2 /mnt
                <VIO-ROOT>    # /mnt/script/ confsnap.ksh.fp24
                                                  
7.            Check and make sure the  Shared Ethernet adater  is in “auto” in both VIO servers in the Frame
               
<VIO>                  $ lsdev -type sea
                <VIO>                  $ entstat –all entX |grep –i high

8.   Put the Shared adapters in “standby” mode for the VIO server which the alt disk going to take.

    >>for each sea (normally 4)
<VIO>                  $ chdev -dev entX -attr ha_mode=standby


9.            Verify network connections on the VIO servers and those are set to standby.
                <VIO>                  $ uname -n
<VIO>                  $ lsdev -type sea
               <VIO>                   $ entstat –all entX |grep –i high

10.         From the VIO server run an alt_disk_install to the spare SCSI disks:

                to find spare disk  run;

<VIO-ROOT>   # lspv | grep root  ; lsdev -Cc disk | egrep -i ‘sas|scsi’

if there is already an alt disk present, remove it using

<VIO-ROOT>    # alt_disk_install -X <old_rootvg>


>>>>>Process if 2 spare disks exist:

<VIO-ROOT>    # alt_disk_install –C –B <hdiskX> <hdiskX>

>>>>>>>Process if 1 spare disk exists:

If only one spare disk exists, take a copy of the image.data file and edit all references of COPIES = 2 to 1:

                <VIO-ROOT>    # mkszfile
<VIO-ROOT>    # cd / ; cp –p image.data  image.data_bkup1

(change the “COPIES” value to COPIES = 1)

                <VIO>                  $ sed "s/COPIES= 2/COPIES= 1/g" image.data > image.data_copies1

Run alt_disk_install using an  “at job”

                <VIO-ROOT>    #  at now <enter>
                                alt_disk_install –C –B -i   /image.data_copies1 <hdiskX>
press ctrl D

11.   Amend bootlist in the VIO server to boot from alt disk created 

<VIO-ROOT>    # bootlist –m normal   <hdiskx> <hdisky>

12.          Verify alt_disk clone by running a reboot of the VIO to the alt_rootvg disks :

<VIO>                  $ uname -n
                <VIO>                  $ shutdown –restart

   13.  Check all the “disk paths” are online and the Virtual devices are ok

<VIO>                  $ lsdev –virtual
<VIO>                  $ lspath  |grep –v En

14.          Reboot back to the original disk after changing the bootlist.

15. Capture the output of the below command and store

                <VIO-ROOT>    # ps -eaf |grep aagent
          Expected:
root  340152       1   0   10 Feb      -  3:01 /opt/IBM/ITM/aix523/va/bin/kvaagent


3         Upgrading to Fix Pack 24 for VIOS 2.2


Do not attempt to combine FP24 and SP01 together which can potentially lead to unexpected results

The current level of the VIOS must be v2.1.2.1.0-FP22


  1. Check the locale is set to “en_US
<VIO-ROOT>    # locale

2.           If the command output shows “LANG” is not “en_US” edit “/etc/environment”  file using “vi” and change “LANG” to “LANG=en_US

<VIO-ROOT>    # cp -p /etc/environment /etc/environment.orig_fp24

<VIO >                 $ chlang -lang en_US

Ø       Logout and login

<VIO-ROOT>    # echo $LANG

Log out of the VIO and Log in again.

  1. Stop the Teamquest in the VIO server

        <VIO-ROOT>    # /usr/teamquest/manager/bin/tqmgr –shutdown
        <VIO-ROOT>    # /opt/IBM/ITM/bin/CandleAgent stop all

  1. Verify no processes are running when running below command

<VIO-ROOT>    # ps -eaf |grep aagent

  1. Mount the remote directory onto the Virtual I/O Server.

<VIO-ROOT>    # mount-o soft $CSM:/NIM/vios/vio2.2 /mnt

  1. Commit previous updates, by running the updateios command
<VIO>                  $ updateios -commit

 Apply VIOS v2.2 FP24 update by running the updateios command

<VIO>                  $ updateios -install -accept -dev /mnt/vio2.2_fp24

Initial the install will go through a “Pre-installtion Verification” before you are prompted with the following message to continue install:-

***************************************************************
End of installp PREVIEW.  No apply operation has actually occurred.
***************************************************************

Continue the installation [y|n]?

  1. Verify the compulsory preview install comes back as “success”, and then answer “Y” to continue. In total upgrade with take between 60 - 90 mins to complete

  1. Review “Installation Summary” and ensure file installations results are “Success”

  1. Switch to PAdmin RBAC role and enter “padmin” user password:
<VIO>                  $ swrole - PAdmin



  1. Unmount NFS mount

# umount /mnt

  1. Reboot VIOS as user padmin and answer “Y” to continue reboot and load all changes:
<VIO>                  $ shutdown -restart

  1. After VIOS has rebooted, you are required to accept the license.

<VIO>                  $ license -accept

  1. Verify that the update was successful by checking the results of the updateios command and by running the isolevel command, which should indicate that the ioslevel is now 2.2.0.11-FP-24.
<VIO>                  $ ioslevel

4         Upgrading to VIOS 2.2 Fix Pack 24 to SP01

The current level of the VIOS must be 2.2.10-FP24


  1. Check the VIO version
<VIO>                  $ ioslevel

To apply this Service Pack, the ioslevel must be:
2.2.0.10-FP-24
  1. Mount the remote directory onto the Virtual I/O Server.
<VIO-ROOT>    # mount-o soft $CSM:/NIM/vios/vio2.2 /mnt
  1. Commit any applied file set
<VIO>                  $ updateios -commit
  1. Apply VIOS v2.2 FP24 SP01 update by running the updateios command:
<VIO>                  $ updateios -install -accept -dev /mnt/sp01
Initial the install will go through a “Pre-installtion Verification”before you are prompted with the following message to continue install:-
***************************************************************
End of installp PREVIEW.  No apply operation has actually occurred.
***************************************************************

Continue the installation [y|n]?
  1. Verify the compulsory preview install comes back as “success”, then answer “Y” to continue. In total update should take  between 30-45 minutes s to complete.
  2. Review “Installation Summary” and ensure file installations results are “Success”
  3. After the update is complete, verify the correct IOS level by entering the following command:
<VIO>                  $ ioslevel
2.2.0.11-FP24 SP-01

5         Installing Interim Fix IZ91190 for VIOS 2.2.0.10-FP24 SP01

The current level of the VIOS must be 2.2.0.11-FP24-SP01

  1. Verify the correct IOS level by entering the following command:
<VIO>                  $ ioslevel
2.2.0.11-FP24 SP-01

If the command output shows any level other than 2.2.0.11-FP-24 SP01, do not install the fix.
  1. Mount the remote directory onto the Virtual I/O Server.
<VIO-ROOT>    # mount-o soft $CSM: /NIM/vios/vio2.2 /mnt
  1. Commit any applied file set
<VIO>$ updateios -commit
  1. Apply VIOS “IZ91190s31” fix by running the updateios command.
<VIO>                  $ updateios -install -accept -dev /mnt/efix
Initial the install will go through a “Pre-installtion Verification”before you are prompted with the following message to continue install:-
****************************************************************
EFIX MANAGER PREVIEW END
****************************************************************
+-----------------------------------------------------------------------------+
Operation Summary
+-----------------------------------------------------------------------------+
Log file is /var/adm/ras/emgr.log
EPKG NUMBER       LABEL               OPERATION              RESULT           
===========       ==============      =================      ==============   
1                 IZ91190s31          INSTALL PREVIEW        SUCCESS          
Return Status = SUCCESS
Continue the installation [y|n]?
  1. Verify the compulsory preview install comes back as “success”, the answer “Y” to continue. In total fix install should take between 2-5 minutes to complete.
  2. Review “Return Status” and ensure results is “Success”
7.                  Check for installed interim fixes by entering the following command.
<VIO>                  $ lssw

You should see your fix listed in the command output. The update is then complete
ID  STATE LABEL      INSTALL TIME      UPDATED BY ABSTRACT
=== ===== ========== ================= ==========
1   *Q*   IZ91190s31 02/02/11 15:24:19            LACP on etherchannel on SEA  

  1. Reboot the  VIO Server
<VIO>                  $ uname –n
<VIO-ROOT>    # umount /mnt
<VIO>                  $ shutdown –restart

6         VIOS SDDPCM & MPIO Updates


  1. Check SDD file sets are at correct pre-upgrade level

<VIO-ROOT>    # lslpp -l devices.sddpcm.61.rte

Required Level:

“devices.sddpcm.61.rte      2.4.0.5 IBM SDD PCM for AIX V61”

  1. Check MPIO file sets are at correct pre-upgrade level

<VIO-ROOT>    # lslpp -l devices.fcp.disk.ibm.mpio.rte

Required Level before upgrade:

   “devices.fcp.disk.ibm.mpio.rte  1.0.0.17  IBM MPIO FCP Disk Device”


  1. Update MPIO fileset

<VIO-ROOT>    # mount-o soft $CSM:/software /mnt

<VIO-ROOT>    # cd /mnt/AIX-components/SDDPCM/2.6.0.3/mpio

<VIO-ROOT>    # /usr/lib/instl/sm_inst installp_cmd -a -Q -d '.' -f '_all_latest'  '-c' '-N' '-g' '-X'   '-G'   '-Y'

  1. Run lslpp - to verify the installation has completed successfully and MPIO is at following level:-

<VIO-ROOT>    # lslpp -l devices.fcp.disk.ibm.mpio.rte

Require Level after upgrade:-

“devices.fcp.disk.ibm.mpio.rte  1.0.0.21”

  1. Update SDD PCM  fileset

<VIO-ROOT>    # cd /mnt/AIX-components/SDDPCM/2.6.0.3/AIX6.1 ;ls -ltr

<VIO-ROOT>    # /usr/lib/instl/sm_inst installp_cmd -a -Q -d '.' -f '_all_latest'  '-c' '-N' '-g' '-X'   '-G'   '-Y'
  1. Run lslpp - to verify the installation has completed successfully and SDD PCM is at following level:-

<VIO-ROOT>    # lslpp -l |grep -i sddpcm

Require Level:-

“devices.sddpcm.61.rte      2.6.0.3”



7         Installing Interim SAN Fix IZ94290m03 for VIOS 2.2.0.10-FP24 SP01

The current level of the VIOS must be 2.2.10-FP24-SP01

1.       Check VIOS Version Installed
<VIO>                  $ ioslevel

If the command output shows any level other than 2.2.0.11-FP-24 SP01, do not install the fix.
2.      Mount the remote directory onto the Virtual I/O Server.
<VIO-ROOT>    # mount-o soft  $CSM: /NIM/vios/vio2.2 /mnt
3.      Apply VIOS “IZ91190s31” fix by running the updateios command.
<VIO>                  $ updateios -install -accept -dev /mnt/IZ94290m03

Initial the install will go through a “Pre-installtion Verification”before you are prompted with the following message to continue install:-
****************************************************************
EFIX MANAGER PREVIEW END
****************************************************************
+-----------------------------------------------------------------------------+
Operation Summary
+-----------------------------------------------------------------------------+
Log file is /var/adm/ras/emgr.log
EPKG NUMBER       LABEL               OPERATION              RESULT           
===========       ==============      =================      ==============   
1                 IZ94290m03         INSTALL PREVIEW        SUCCESS   
   
ATTENTION: system reboot will be required by the actual (not preview) operation.
Please see the "Reboot Processing" sections in the output above or in the
/var/adm/ras/emgr.log file.

Return Status = SUCCESS  
Continue the installation [y|n]?

4.      Verify the compulsory preview install comes back as “success”, the answer “Y” to continue. In total fix install should take between 2-5 minutes to complete.
5.      Review “Return Status” and ensure results is “Success”
6.      Check for installed interim fixes by entering the following command.
<VIO>                  $ lssw
You should see your fix listed in the command output. The update is then complete
ID  STATE LABEL      INSTALL TIME      UPDATED BY ABSTRACT
=== ===== ========== ================= ==========
1   S   IZ91190s31 02/02/11 15:24:19            LACP on etherchannel on SEA  
2   *Q*   IZ94290m03 02/15/11 17:25:19            VIOS storage BD communications




7.      Change the Fiber and SAN Disk attributes to the Lloyds recommend values
 
<VIO-ROOT>    # mount-o soft  $CSM:/NIM/vios/vio2.2 /mnt
<VIO-ROOT>    # /mnt/script/config_fcs_hdd_fp24.ksh

Check the values

<VIO-ROOT>    # /mnt/script/SAN_attrib_Check.ksh

reserve_policy=no_reserve /algorithm=load_balance /hcheck_interval=60 /hcheck_mode=nonactive/rw_timeout=60

8.      Before  reboot check the  Fiber and disk  attribute status  and change it to below if otherwise.

<VIO-ROOT>    # lsattr -El fscsi<n> -a dyntrk -a fc_err_recov
                               
dyntrk      yes                          Dynamic Tracking of FC Devices        True
                                fc_err_recov fast_fail            FC Fabric Event Error RECOVERY Policy True

<VIO-ROOT>    # lsattr -El fcs<n> -a init_link
   
init_link pt2pt INIT Link flags True

9.      Reboot the  VIO Server

<VIO>                  #umount /mnt
<VIO>                  $ uname -n
<VIO>                  $ shutdown –restart

10.  Once the VIO server is up , change the SEA adapters mode  to “auto”  and carry on the health checks before doing the other VIO in the frame .
<VIO>                  $ lsdev -type sea
<VIO>                  $ chdev –dev entx –attr ha_mode=auto (where entx refers to SEA ent device)
<VIO>                  $ entstat –all entX |grep –i high
11.   Check the Teamquest has started
<VIO-ROOT>    # /usr/teamquest/manager/bin/tqmgr -service

Expected:
                [07-17-2010 17:33:50] [Info] Using data directory group ID: 1
 [07-17-2010 17:33:50] [Info] Agent manager process group ID set to: staff (1)
Info: The service is already running.

12.   Health Check the VIO & Client  servers  which you have updated
<VIO>                  $ lsdev –virtual
<VIO-ROOT>    # errpt, lsvg –l rootvg
# lsdev –Cc adapter;- # lspath; # pcmpath query adapter
# pcmpath query device

8        VIOS Changes

8.1       NTP (if you find this is already in place, please ignore)


1.       Make below additions to the below /home/padmin/config/ntp.conf file :

<VIO-ROOT > # cp -p /home/padmin/config/ntp.conf /home/padmin/config/ntp.conf.orig_fp24

                   <VIO-ROOT>  # vi /home/padmin/config/ntp.conf      

server 172.23.205.102 version 3 # Name:    timegps3.lloydstsb.co.uk Address:  172.23.205.102
server 172.23.165.102 version 3 # Name:    timegps2.lloydstsb.co.uk Address:  172.23.165.102

2.      Run following to start the xntpd service :

<VIO>  $ startnetsvc xntpd

Ensure synchronisation has occured by tailing the below log file looking for the below: it

<VIO-ROOT>  # tail -f /home/padmin/config/ntp.log

29 Jun 11:57:26 xntpd[630790]: synchronized to 172.23.165.102, stratum=1
29 Jun 11:59:59 xntpd[630790]: time reset (step) 152.212928 s
29 Jun 11:59:59 xntpd[630790]: synchronisation lost
29 Jun 12:04:16 xntpd[630790]: synchronized to 172.23.205.102,

9         VIOS ITM Updates

9.1       Check ITM file sets are at correct levels


<VIO-ROOT> # lslpp -l|grep ITM

Required Level:
  itm.base.rte                                      6.2.1.0  COMMITTED  ITM Runtime Install Files
  itm.cec.agent                                  6.2.4.0  COMMITTED  ITM CEC Agent
  itm.premium.rte                                           6.2.4.0  COMMITTED  ITM Runtime Install Files
  itm.vios_premium.agent           6.2.4.0  COMMITTED  ITM VIOS Premium Agent
  itm.cec.agent                                  6.2.4.0  COMMITTED  ITM CEC Agent
  itm.vios_premium.agent           6.2.4.0  COMMITTED  ITM VIOS Premium Agent

Note. We no longer require “itm.vios.agent”, only “itm.vios_premium.agent”.

 If down level update ITM base using the following steps

1.       Stop ITM Agents

<VIO>                  $ stopsvc ITM_premium   (VIOS 2.1 )  

2.      Mount the remote directory onto the Virtual I/O Server.

<VIO-ROOT>    # mount-o soft  $CSM:/NIM/vios/vio2.2 /mnt
<VIO-ROOT>    # cd /mnt/ITM  <smitty install_latest

(performing preview first and accept  license)

3.      Run lslpp - to verify the installation has completed successfully and ITM filesets are at levels detailed above.

4.      Verify LPP file sets consistent

        <VIO-ROOT>    # lppchk  -v                      ( confirm they are NO missing filesets)

5.      Check the ITM  agent is running

<VIO-ROOT>    # ps -eaf |grep aagent
       
  Expected:
root  340152       1   0   10 Feb      -  3:01 /opt/IBM/ITM/aix523/va/bin/kvaagent

if the ITM “VA” agent is not running ,run the below .

<VIO-ROOT>         #/opt/IBM/ITM/bin/CandleAgent start va

6.      Re run the audit script

<CSM>                 # /usr/local/bin/viosaudit3.sh <viohostname-csm>


































9.2    VIOS ITM Premium Agent (DO only if missing base & premium)


7.      Mount the remote directory onto the Virtual I/O Server

<VIO-ROOT>    # mount-o soft $CSM:/NIM/vios/vio2.2 /mnt

8.      Install ITM VIOS Premium and Base Agent

<VIO-ROOT>    # cd /mnt/ITM   << smitty install

9.      Run lslpp - to verify the installation has completed successfully and ITM filesets are at following levels:-

<VIO-ROOT>    # lslpp -l|grep ITM
      
Required Level:

                                 “itm.vios_premium.agent      6.2.1.0” < new install only otherwise V6.2.4.0 >
“itm.base.rte 6.2.1.0”

10.  Update VIOS ITM Premium Agent

<VIO-ROOT>    # mount-o soft $CSM:/NIM/vios/vio2.2 /mnt
<VIO-ROOT>    # cd /mnt/ITM 
<VIO-ROOT>    # smitty install_latest <select  itm.vios_premium.agent  V6.2.4.0 >  

11.   Run lslpp - to verify the installation has completed successfully and ITM VIOS Premium Agaent is at following level:-

<VIO-ROOT>    # lslpp -l itm.vios_premium.agent     

Require Level:-
“itm.vios_premium.agent      6.2.4.0”

Now go back to previous section and carry on






9.4    VIOS   audit FS conversion to jfs2


Check the File system is  not currently jfs2

<VIO-ROOT># lsfs /audit ( if VFS=jfs)

<VIO-ROOT># umount /audit
<VIO-ROOT># rmfs –r /audit
<VIO-ROOT># /usr/sbin/mklv -y'lv00' -t'jfs2' -c'2' rootvg 1
<VIO-ROOT>#crfs -v jfs2 -d  lv00  -m /audit -A yes
<VIO-ROOT>#mount /audit

<VIO-ROOT># lsfs /audit ( VFS=jfs2)



















10    Post Updates Checks

  1. Verify VIO Server level
<VIO>                  $ ioslevel
2.2.0.11-FP-24 SP-01
<VIO-ROOT> # lslpp -l |grep -i sddpcm ;lslpp -l devices.fcp.disk.ibm.mpio.rte
devices.sddpcm.61.rte      2.6.0.3  / devices.fcp.disk.ibm.mpio.rte 1.0.0.21
<VIO-ROOT># emgr -l |head -n 10

                 ID  STATE LABEL           INSTALL TIME      UPDATED BY ABSTRACT
                === ===== ========== ================= ==========
                1    S    IZ91190s31                   02/15/11 17:05:05                LACP on etherchannel on SEA
                2    S    IZ94290m03          02/15/11 17:25:19                      VIOS storage BD communications

STATE codes:
 S = STABLE

No comments:

Post a Comment