Installing Oracle 10g R2 RAC on Windows 2003 with Domain Controller

This is an article on installing Oracle 10g R2 RAC on Windows 2003 with Domain Controller in place. i.e. The Windows Server is an existing Primary or Backup Domain Controller server and we need to install RAC on it. Though Microsoft and Oracle doesn’t recommend installing RAC on a domain controller, it is possible to implement RAC.

Note: Don’t install RAC on normal server and convert into domain controller. It wont work as there will be mismatches with the OCR (Oracle Cluster Registry) entries which wont allow to start the cluster services. For example, the initial user would be say Administrator and when you enable domain controller, the user would be sscblr/Administrator. CRS will fail to start using the initial user, which won’t be available after enabling domain controller.

There are number of combinations you can try while installing Oracle 10g RAC. Like

  • Using Raw devices for cluster files (OCR,Voting Disk) and database files (I would not suggest this as it difficult to deal with files on Raw file systems)
  • Using OCFS (Oracle Cluster File System) for both cluster files and database files.
  • Using Raw file system for cluster files and ASM instance for the database files.
  • Using OCFS for cluster files and ASM instance for the database files.
  • Using Raw file system for cluster files and OCFS for the database files

And so on…..

Initially I thought of using Raw file system for cluster files and ASM for the database files. Once the Clusterware was installed then the database software installation was successful on the both the nodes, I was able to create the ASM instance but while creation of the database I encountered the following errors.

CREATE DATABASE “test1”
ORA-01501: CREATE DATABASE failed
ORA-00200: datafile could not be created
ORA-00202: datafile: ‘+DG1_NR/data/test1_system.dbf’
ORA-17502: ksfdcre:4 Failed to create file
+DG1_NR//data/test1_system.dbf
ORA-15001: diskgroup “DG1_NR” does not exist or is not mounted
ORA-15055: unable to connect to ASM instance

ORA 1031: “insufficient privileges”

After some soul searching I found the listener status was showing

Services Summary…
Service “+ASM1” has 1 instance(s).
Instance “+ASM1”, status BLOCKED, has 1 handler(s) for this service…

 

There are workarounds given in Oracle metalink where you have to edit the network files (listener.ora and sqlnet.ora) to include authentication parameters or modify the listener.ora (Note:431312.1)but it didn’t work for me. When I tried with the workarounds and reloaded the listener, the ASM service would show as status unknown. But the problem was that along with this service automatically another ASM service would come up and display the status as BLOCKED (If anyone knows why this happens please let me know).

I could gather some information that says the way of authentication in Windows 2000 is different from Windows 2003.So ASM was not able to connect with DBCA.I even did a fresh installation using OCFS for the cluster files and ASM instance for the database files. Again I encountered the same set of errors. So I decided to drop the idea of using ASM for the datafiles and to go with raw file system for the cluster files and OCFS for the database files. It worked for me even though the server itself was a domain controller.

There may be other combinations which will work, but this document is with the one which worked for me.

Prerequisites

Here the user account used for RAC installation is the administrator of the domain controller and he is part of ORA_DBA group.

Check the following requirements before installation:

  • Software
  • Hardware
  • Disk Space
  • Network

Software

The software requirements for Oracle Real Application Clusters 10g are:

Requirement

Value

System Architecture

32-bit

Operating System

Windows 2003 with service pack 1

Network Protocol

TCP/IP

Hardware

To ensure that both nodes in the cluster meet the minimum requirements to install Oracle Database 10g Real Application Clusters, perform the following steps:

· Check the physical RAM size is at least 512 MB
· Confirm the size of the configured swap space is at least twice the physical RAM size.

Disk Space

To support your Real Application Clusters database, your database files must be stored on disks that are shared by both nodes in your cluster. Any shared disks supported by your hardware vendor, other than Network Attached Storage (NAS), can be used. The shared disks must be attached to both nodes in your cluster and both nodes must be able to read and write to them.

The requirements for Oracle Real Application Clusters 10g installation and database creation are:

Requirement

Free Space Needed

Location

Ancillary Oracle files

100 MB on both nodes

Any local system drive

Temporary Space

50 MB on both nodes

Any local system drive

Cluster Ready Services software

500 MB on both nodes

Any local system drive but must be the same named drive on both nodes

Database software

1 GB on both nodes

Any local system drive but must be the same named drive on both nodes

Enable Automount

Do the following on every node in the cluster:
1) Run diskpart
2) At the diskpart prompt, run the command, "automount enable"
3) Exit diskpart
4) Reboot the machine  
You need to perform the above steps only once for every node.  
Note:  You can determine the "automount" status in the following manner:
Start diskpart and enter the command "automount" without any additional parameters.  
This will display the current status of automount, such as:
Automatic mounting of new volumes disabled.
 or
 Automatic mounting of new volumes enabled.

Preparing the disks for storage

Here I assume that there is only one shared disk drive Disk1(of say 40GB ).The shared disk must be attached to both nodes in your cluster and both nodes must be able to read and write to them.

Note: You cannot use Network Attached Storage (NAS). The reason being that the if the node hosting the NAS fails then other nodes will go down as the storage will not be available for them .i.e. there is a chance of single point of failure.

Format the disk drive as raw devices initially, no matter what storage option

(ASM, OCFS, or raw devices) you choose.

Steps to create Raw device.

>Right-click My Computer and select Manage >> Disk Management. Format the disk drive as basic disk.
>Right-click a disk drive and select Create disk >> Extended Partition. Use the entire disk drive Disk1 for the extended partition.
>Right-click the partition that you just created and choose New Volume. Choose no drive letter and do not format the partition. After this, if you check the disk drive properties, the File System column should show RAW.

Steps to create drives for Cluster files and database files.

Right-click My Computer and select Manage >> Disk Management. Format the disk drive as basic disk
Right-click a disk drive and select Create Logical drive.
Create Partition Wizard appears.
The partition type you want to create is Logical Drive. Click Next. Enter 300MB for the size for the first logical partition. This is the size that you want for the Oracle Cluster Registry logical drive.
Click Next. Choose the option Do not assign a drive letter or drive path and click Next.
Choose the option Do not format this partition and click Next.
Click Finish.

Repeat the step to create logical drives of 300MB(for OCR mirror) ,600MB(for voting disk),30GB(database files using OCFS) and 9GB(for flash recovery area using OCFS)drives.

Reboot the nodes. Check the two nodes in the cluster to make sure the partitions are visible on both and none of the Oracle partitions have drive letters assigned. If drive letters are assigned, remove it.

Network

Both nodes must meet the following public and private network requirements:

1. Support for two network adapters

  • Public network interface used for client connections
  • Private network interfaces used for communication between database instances

2. Support of interface name restrictions

  • Names are case sensitive
  • Names must not contain any multibyte language characters
  • Public and private network interface names must be different from each other
  • Name for each interface must be the same on both nodes
  • Public and private IP addresses must be on different subnets.

3. An IP address and host name registered in the domain name system (DNS) for the public network interface

4. A private IP address and optionally, a private host name, for each private network interface.

Note: Oracle recommends that you use private network IP addresses for these interfaces, for example: 10.*.*.* or 192.168.*.*. You can use%SystemRoot%\system32\drivers\etc\hosts file on both nodes to associate private host names with private IP addresses. To configure the virtual IP addresses, simply specify the virtual IP addresses in the%SystemRoot%\system32\drivers\etc\hosts file. The remainder of the virtual IP address setup is completed during the software installation steps.

For example ,the hosts files will look this.

192.168.0.1 Node1
192.168.0.2 Node2
192.168.0.3 Node1-vip
192.168.0.4 Node2-vip
10.0.0.1 Node1-priv
10.0.0.2 Node2-priv

Note: don’t bind the Virtual IP physically with the Public interface.

5. Select the same interfaces on all hosts. That is, on each host, eth0 must be public and eth1 must be private. Interface names must be identical on all hosts, with public and private interfaces having separate names. Interface names are case sensitive. In the test setup, all the public interfaces were named “Public” and all the private interfaces were named “Private.”

6. List the public interface first in the bind order—the order in which network services access the host. Use the following steps to verify the bind order.

Select Start >> Control Panel >> Network Connections >> Advanced >>Advanced Settings.

Click the Adapters and Bindings tab.

In the Connections area, observe whether the public interface appears before all others and whether the private interface is second. If the public and the private interface do not appear in the proper order, modify the order.

Installation Pre-requisites

  • Synchronize the clocks. Make sure that the clocks on all of the nodes are synchronized. You can do this either manually or by using windows time servers. If the clocks do not match, the Oracle Clusterware installation gives you errors.
  • Use CVU commands to verify that all requirements up to this point are met.

To check hardware requirements, use this command.

<media>\clusterware\cluvfy\runcluvfy.bat stage -post hwos –n

The result should show that the verification was successful to continue.

  • Disable the Distributed Transaction Coordinator service for the duration of the setup, because it might interfere with Oracle Clusterware installation. Do these steps on each node.

Right-click My Computer and select Manage >> Services and Applications >> Services.

Find the Distributed Transaction Coordinator service. Change the startup type to manual, and stop the service.

Before running the Oracle Universal Installer, from the node where you intend to run the Installer, verify that you have administrative privileges on the other nodes. To do this, enter the following command for each node that is a part of the cluster:

net use \\node_name\C$

where node_name is the node name.

It is recommended to use the same user name and password on each
node in a cluster or use a domain user name. If you use a domain user name, log on under a domain with a user name and password that has administrative privileges on each node.

You also cannot install this product using Remote Desktop unless you invoke Remote
Desktop in ‘ – console ‘ mode such as:

mstsc -v:servername /F –console

Oracle Clusterware Installation

To check whether the nodes are ready for Clusterware installation, use this command.

<media>\clusterware\cluvfy\runcluvfy.bat stage -pre crsinst –n Node1,Node2

The result must display as successful to continue.

Now to install Oracle Clusterware, the following steps should be carried out on any of the nodes.

Go to <media>\clusterware\ and click on setup.exe

Page Action

Initial screen Click Next.

Specify Inventory Directory and Credentials Accept the default inventory location. This location

is shared between the Oracle Clusterware home and the Oracle Database home. The location shown should be $ORACLE_BASE/oraInventory.

Specify Home Details Specify the Oracle Clusterware mount point

Product-Specific Prerequisite Checks All checks should pass if everything is configured correctly..

Specify Cluster Configuration Specify the hosts. i.e Node1 and Node2(Only one node will be displayed.You should add the other node).

Specify Network Interface Usage Specify the Interface Type appropriately as Public or Private.(If there are more interfaces say like LAN 2,make it not in use) using Edit option.

Specify Oracle Cluster Registry (OCR) Location Choose 300MB logical drive and using edit option choose this to be OCR Primary and second 300MB to be OCR Mirror location .

Then choose 600MB logical partition to be the Voting Disk Location .

Do not touch the remaining 30GB and 9GB logical partitions. We shall use it as OCFS for database files later. Click Next.

Click on install. You can notice that once the clusterware installation is complete on this node, remote copy happening which will install clusterware on the other node also.

Then you can see Cluster configuration Assistants window. Though it has been displayed as optional, it is very much required to these assistants to succeed to start the Cluster services

The VIPCA will fail.(Bug 4437727 in 10.2.0.1)Ignore it.

Exit from the installer.

Now run the VIPCA manually.

Running the VIPCA manually

$ORA_CRS_HOME/bin/vipca

Click Next on the initial screen.

Use the VIP Configuration Assistant wizard to configure virtual IP addresses through vipca.

Network Interface Select both interfaces, and then click Next.

Virtual IPs for Cluster Nodes Configure the IP Alias Name and the virtual IP address as you specified in /etc/hosts. (Node1-vip,Node2-vip)

Progress Dialog Watch as Vipca creates the appropriate application resources.

Final screen shows configuration results Click Finish.

Execute Configuration Scripts Click OK.

To check whether the clusterware was installed properly, use this command.

<media>\clusterware\cluvfy\runcluvfy.bat stage -post crsinst –n Node1,Node2

Now using the services window on both the nodes check for OracleCSService,OracleEVM and OracleCRS service. The status should be started to indicate that cluster setup is successful.

You can also check the Oracle Clusterware status by running the $ORA_CRS_HOME/bin/

crs_stat -t command on either host (The processes should be ONLINE)

Oracle RAC Database Software Installation

The following steps should be carried out on any of the nodes.

Go to <media>\database\ and click on setup.exe

Page Action

Initial screen Click Installed Products to make sure that the OracleClusterware is installed successfully.

Select Installation Type Select Enterprise Edition.

Specify Home Details Specify $ORACLE_BASE/<any folder> as the location of the SW home directory for your database.

Specify Hardware Cluster Installation Mode Select Cluster Installation and select all hosts.i.e Node1 and Node2

Product-Specific Prerequisite Checks All checks should pass if everything is configured correctly.

Select Configuration Option Choose Install database software only. We will configure the database later using DBCA.

Execute Configuration Scripts Follow the on-screen directions

Format the Logical partitions using OCFS

Now before creating database we need to format the logical drives to place the database files. Previously we had left out two logical partitions of 30GB and 9GB,remember?I will use those partitions to create OCFS partitions. The formatting should be done from only one of the nodes.

Assign drive letters say F to 9GB partition and G to the 30GB logical partitions. Do not format it using Disk Management!!

Now using ocfsformat we shall format the two partition as shown below:

Ocfsformat tool can be found in CRS_HOME/cfs/ directory

ocfsformat /m F: /c 1024 /v FLASHBACK /f

ocfsformat /m G: /c 1024 /v DATA /f

Note:
Clustersize is essentially the block size.  In general it is recommended to have:  
/c 1024 is 1024 k (1 MB) <=== if the OCFS partition is to be used for Oracle datafiles
/c 4 is 4 k <=== if the OCFS partition is to be used for the ORACLE_HOME

Now reboot the other node.After rebooting the second node should recognize these partitions.

Note: Refer to point 3 in the Troubleshooting RAC issues section which is discussed at the end of the document before the creation of the database.

Invoke DBCA

Welcome Select Real Application Clusters database.

Operations* Select Create a Database

Node Selection Click Select All to select all hosts.

Database Templates Select the database type to create.

Database Identification Name of the database and SID.

Management Options Choose and configure the management options you want.

Database Credentials Specify passwords.

Storage Options Choose the storage option that is appropriate for your environment.

Select Cluster file System

Database File Locations

Select G drive (which was earlier formatted with OCFS)

Recovery Configuration

Choose and configure a recovery option appropriate for the environment. Use F drive for Flash Recovery Area.

Enable Archiving if necessary.

Database Content In the Database Components tab, you can select and configure components.

Database Services Create any database services that you want for TAF(Transparent Application Failover).

Initialization Parameters

Change the parameters according to your requirement.

Database Storage Click an item in the left panel to see its details in the right panel. You can reconfigure in the right panel if necessary.

Summary View the summary information.

Copying database files. During this process it will ask for listener to be created, click on Yes

Database creation complete The new database is created!

 

Troubleshooting RAC issues

I will briefly explain some of the problems I faced during installation of RAC.

1. During installation of the clusterware the CRS(Cluster Ready Services) were failing to start.And when the server was rebooted it would take around 3 to 4 hours to come up.Then I found that the problem in Windows 2003 is that by default the raw devices will not be mounted. You need to enable the mounting part.

There is a procedure to change the automount to enabled as discussed earlier. 
Do the following on every node in the cluster: 
  1) Run diskpart
  2) At the diskpart prompt, run the command, "automount enable" 
  3) Exit diskpart
  4) Reboot the machine 

You only need to do this once for every node.  

So once this was done the servers would get rebooted in less than 10 minutes!!

Refer to Metalink Note: 254611.1

2. If the configuration of the Clusterware goes wrong,it is necessary to remove the CRS installation completely.

Refer Metalink Note: 341214.1

3. When the nodes(either Node1 or Node2) are rebooted, sometimes(on Windows will face this problem for sure!!) the CRS(Cluster Ready Services) doest startup automatically.In services window you can either see OracleCSService in starting status i.e. kind of hung state.Here even the OracleEVM service and OracleCRService will not start because they are dependent on OracleCSService.

I would suggest to resolve this problem before the creation of the database because without the cluster services being not configured properly, database will not make much sense.

There are two reasons for this which I am aware of.

a.The problem related to the order of startup of services when using symbolic link names for the shared disks, and are using the “Oracle Object Service”.

b. It is an Oracle Bug(4537790).The clustering services for Oracle i.e., OracleCSService has two means of communication with other nodes in the cluster – they communicate by disk via means of the Voting Disk and they also communicate across the interconnect.

On a reboot, or at startup, the OracleCSService must be able to initiate communications with the other nodes in the cluster via all means of communication. For that communication to be successful, the operating system subsystems (such as the networking subsystem or disk I/O subsystems) must be fully initialized.

If these clustering services are set to Automatic, the OracleCSService may attempt to startup before the underlying networking components of the OS have been fully initialized. When this happens, the CSS service cannot startup correctly because it cannot fully communicate with the other cluster nodes.

In most cases, this happens on the second node to boot up, because we are able to see via the Voting Disk that there is at least one other node in the cluster – but since the networking components of the OS are not yet fully initialized, we cannot communicate on the interconnect. The result of this situation is a split-brain.

On Oracle10g, if a split-brain is detected by the clusterware, it is expected behavior that the node is rebooted. In this case, you may find that the cluster node trying to rejoin the cluster is caught in a ‘reboot loop’

Because of the split brain when the two nodes are rebooted separately, both the nodes will not be able to communicate each other. So each of the nodes will try to act as master node.In this case you may also notice blue screen appearing on your screen(I faced this) and your nodes go for automatic reboot one by one.

Solution for a:

The fix is to add the “Oracle Object Service” as a dependent for the OracleCSService. Once that is done all services will start as required.

Adding this dependency requires the sc.exe tool which is available on Win2k3, but you may need to locate one for Win2K.

One can use sc.exe to also get a list of the dependent services. The command

“sc qc OracleCSService”

will display all the configuration information for the service, including the names of all the dependent services.

The command to add the dependency is:

sc.exe config OracleCSService depend= OraFenceService/”Oracle Object Service”

This assumes that we already had a dependency setup on the OraFenceService to begin with. You can double check this by looking at the registry key

HKLM\System\CurrentControlSet\Service\OracleCSService\DependOnService

Which should list all the services that OracleCSService is currently depending on.

Once this change is in place, you should be able to setup the OracleCSService to autostart again, and consistently have the service startup properly.And also check in Services window by clicking on the properties and then dependencies of OracleCSService.

Solution for b.

Apply 10.2.0.3 patchset on both Clusterware and database software.Though the problem is not with the database part,better to patch the database software instead of patching it after the database creation

The fix for BUG#4537790 is to provide an executable called ‘opmd.exe’. The opmd.exe allows the creation of a new service called the ‘Oracle Process Manager’. The Oracle Process Manager will control the startup of the clustering services to prevent the OracleCSService) from starting up before the rest of the OS services have started

Note that simply applying the patch by itself is not enough to get the fix for this issue. Applying the patch will not create the ‘Oracle Process Manager’ service. To create the ‘Oracle Process Manager’ service, you will need to run opmd as follows, from the CRS_HOME\bin directory:

opmd -install

On Oracle10g, you will see feedback such as the following:

E:\oracle\product\10.2.0\crs\BIN>opmd -install
OpmdInstall: OPM service successfully installed
OpmdFixupService: Oracle Object Service set to manual startup and added to list
OpmdFixupService: OracleCSService set to manual startup and added to list
OpmdFixupService: OracleEVMService set to manual startup and added to list
OpmdFixupService: OracleCRService set to manual startup and added to list

As noted, this will set the Cluster related services to Manual startup, but the Oracle Process Manager Service itself is set to Automatic. The Oracle Process Manager will start up the Oracle cluster-related services (listed above) at the end of the initialization stage of the boot process, thus ensuring that all required OS components are fully initialized first.

Note that the instructions to run ‘opmd -install’ may not be listed in the README for some patchsets, so bear in mind that you are not taking advantage of this fix until this has been run.

If at some point in the future, it is necessary to remove the Oracle Process Manager service, that can be done via the uninstall option of the command:

E:\oracle\product\10.2.0\crs\BIN>opmd -uninstall
OpmdUninstall: OPM service successfully removed

Note that there will be a logfile called ‘opmd.log’ created in the directory where opmd is run out of.

If you are on an Oracle10g release (such as 10.2.0.1) that does not yet have the patch available, it is possible to use the ‘opmd.exe’ from any 10g Release – it is a standalone utility, that does not require any additional files. Therefore, you can simply copy the opmd.exe from any 10g patchset to the CRS_HOME\bin directory and install it.

You need to implement both the solutions to avoid the failure of CRS services at the startup.

Refer metalink for more information on this.

Hope this article is helpful to you guys. Do feel free to put your comments.

thanks,

Yadu.

About the Author: Yadu works for an MNC and he had performed various Oracle DB Implementations in India . He can be reached at yaducool@gmail.com

 

 

 

 

 

 


 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

16 Responses to “Installing Oracle 10g R2 RAC on Windows 2003 with Domain Controller”

RSS Feed for OraclePitStop – Lets share some knowledge Comments RSS Feed

give me a print screen step by step brow!!!!

Hi Yadu

Great Job, I’m in st. Louis, MO, USA, I got the new project to convert Oracle DB 9i/10g to RAC on windows. I read ur articles, it looks good, I will follow this.

Best regards,
Ather Hussain
St. Louis, Mo
atherhussain9@yahoo.com

Hi Yadu
I;m getting error on windows 2003,
shared storage check failed on nodes
can you help

Thanks
Ather Hussain
atherhussain9@yahoo.com

Yadu wrote:

The potential solution would be,
1.Disable the antivirus software for the duration of RAC install.
2.Or with the help of the anitivirus software vendor customize the software such that it allows all kind of files to be copied from one node to the other node.

As of now,I would suggest solution 1.
In either case,RAC installation has to be done from the scratch which also includes reinstallion of CRS as well.

Regards,
Yadu

On 4/5/08, Ather Hussain wrote:
Yes Yadu, I got that in warning Virusscan Enterprise.

Yadu wrote:

Ok.This error can be caused by AntiVirus software which apparently prevents certain files/file types from being remotely copied to the target node.Can you see any error in the Windows Application Log (Event Viewer) for the remote node showing any message like:
VirusScan Enterprise: Blocked by behaviour blocking rule……..

On 4/4/08, Ather Hussain wrote:
Yes, MCfee

Yadu wrote:
Do you have any Anti virus sofware installed which is currently running?

On 4/4/08, Ather Hussain wrote:
Hi Yadu

Thanks for your reply, here I’m using Netapp SAN using iSCSI protocol, for more info I attached error doc.

waiting for your helpful solution.

Best regards,
Ather Hussain

Yadu wrote:
Hi Ather,

I feel you got this error while running cluvfy. I need to know what kind of device you are using for shared devices.The reason being that Cluvfy currently does not work with devices other than scsi devices.This could be one of the reasons.So devices like emc powerpath, volume groups etc will fail when checked using cluvfy for shared accessibility.So use manual methods of detecting shared devices and check.Thanks

Regards,
-Yadu

Would there be any differnece with a 64bit environment

No.There will no difference in the 64 bit environmeny.

[…] and the virtual IP … Then I found that the problem in windows 2003 is that by default the raw …https://oraclepitstop.wordpress.com/about/installing-oracle-10g-r2-rac-on-windows-2003-with-domain-co…Annoyances.org – Connection error using DNS alias instead of Server …… connect to a windows 2003 […]

by ocfsformat i formated two partition in one node and i restarted the other node. i was able to find the partition but when i tried to install the cluster database through DBCA i was able to get only “ASM” option but the “OCFS” and “raw device” patitions are disabled .
but when i went for single node database creation i got the ocfs option.
I followed all ur steps. What could be the reason of this.
Please help me.

with regards,
Arun. A

very useful article. It must be helpful for the DBAs who do not have worked on Oracle clustering with Windows 2003 server.

Thanks Altaf.

cheers,
OraclePitStop

thanks!!!

Good Article
We can install Oracle RAC 11g,10g without domain.

Actualy, i trying to install RAC on Server 2003 but CRS setup fail.
Any solution for instaling RAC without domain on Server 2003

Awesome!!!!
Must read and follow.

very well yadu , great job this help me a lot

Error oui-25031 is not getting resolved

this is very good document to refer.
I had a ? since i hv not installed RAC on windows
so want to know if ocfs is the only solution to
rac what if we have to conig ASM base RAC on
windows.

Regards
Manoj


Where's The Comment Form?

Liked it here?
Why not try sites on the blogroll...

%d bloggers like this: