Surachart Opun

How to create signed url on google cloud storage?

Mon, 2020-01-27 08:54

A signed URL is a URL that provides limited permission and time to make a request. It's good to be used by someone who does not have a Google Account. I caught up reading on Google Cloud documents and finding how to do it. Assume I would like to share file on google cloud storage to my friend who does have a Google Account. Example: gs://mysurachartbucket/test.txt
[student@centos~]$ gsutil mb gs://mysurachartbucket
Creating gs://mysurachartbucket/...
[student@centos~]$ cat test.txt
TEST
[student@centos~]$ gsutil cp test.txt gs://mysurachartbucket/
Copying file://test.txt [Content-Type=text/plain]...
- [1 files][ 5.0 B/ 5.0 B]
Operation completed over 1 objects/5.0 B.
[student@centos~]$ gsutil ls gs://mysurachartbucket/test.txt
gs://mysurachartbucket/test.txtFirst of all, I need keystore-file from service account. So, To create service account and key file.

[student@centos~]$ gcloud iam service-accounts list
NAME EMAIL DISABLED
Compute Engine default service account ********-compute@developer.gserviceaccount.com False
[student@centos~]$ gcloud iam service-accounts create surachart
Created service account [surachart].
[student@centos~]$ gcloud iam service-accounts list
NAME EMAIL DISABLED
Compute Engine default service account ********-compute@developer.gserviceaccount.com False
surachart@myproject.iam.gserviceaccount.com False
[student@centos~]$ gcloud iam service-accounts keys create ~/surachart.json --iam-account surachart@myproject.iam.gserviceaccount.com
created key [4d6b1bd*********08f966dd31] of type [json] as [/home/student/surachart.json] for [surachart@myproject.iam.gserviceaccount.com]Then, service account should be able to read file in bucket.
[student@centos~]$ gsutil acl ch -u surachart@myproject.iam.gserviceaccount.com:R gs://mysurachartbucket/test.txt
Updated ACL on gs://mysurachartbucket/test.txtFinally, create signed url by using gsutil command.
[student@centos~]$ gsutil signurl -d 20m surachart.json gs://mysurachartbucket/test.txt
CommandException: The signurl command requires the pyopenssl library (try pip install pyopenssl or easy_install pyopenssl)
####As error that need to install pyopenssl.
[student@centos~]$ sudo pip install pyopenssl
[student@centos~]$ gsutil signurl -d 20m surachart.json gs://mysurachartbucket/test.txt
URL HTTP Method Expiration Signed URL
gs://mysurachartbucket/test.txt GET 2020-01-27 21:34:08 https://storage.googleapis.com/mysurachartbucket/test.txt?x-goog-signature=99dbc749d2891eb1d9d22a5ccd03a81d4f0366380ff3bb0c34faf246d20677290778c6033a81fce43363709b244a882308b1c8590eaed409e1c8a0d4aca76cfec8537b1231e6b1f57************c6abaaacd128ac85f798edfb41bfa48d688897882be28cd1838520144ff197a5e84f499da914c2f8b309c32343011974a8f888163cba2a33c491fd858906bce2ad3cb5c5249c1e79127d200dccea553deafe7e1eb43a8b1527cb20e935c66129b0cad1683f01b6474a4c2940b92dd6daaa65da48fba7cbe94ed5881d46f268908735b2ad12ef2b1f7b0e79a2dd4a527cc611ea35718db96db&x-goog-algorithm=GOOG4-RSA-SHA256&x-goog-credential=surachart%40myproject.iam.gserviceaccount.com%2F20200127%2Fus%2Fstorage%2Fgoog4_request&x-goog-date=20200127T140408Z&x-goog-expires=1800&x-goog-signedheaders=hostThis signed url will expire in 20 minutes. Then send it to my friend.

Reference:
https://cloud.google.com/storage/docs/access-control/signed-urls
https://cloud.google.com/storage/docs/gsutil/commands/signurl

Categories: DBA Blogs

Oracle Database EM 18 XE Available to Remote Clients

Mon, 2018-10-22 10:58

I found lot of posts about Oracle Database 18 XE. It's very interesting for me. I didn't blog about how to install, because it's very easy for using rpm package and document very helpful.
I was interested in Enterprise Manager Database Express 18.4.0.0.0. How it looks like?
- Installing. I used CentOS7.
[student@centos-learning ~]$ sudo yum -y localinstall oracle-database*18c*
[student@centos-learning ~]$ sudo rpm -qa |grep oracle
oracle-database-preinstall-18c-1.0-1.el7.x86_64
oracle-database-xe-18c-1.0-1.x86_64

[student@centos-learning ~]$ sudo /etc/init.d/oracle-xe-18c configure
Specify a password to be used for database accounts. Oracle recommends that the password entered should be at least 8 characters in length, contain at least 1 uppercase character, 1 lower case character and 1 digit [0-9]. Note that the same password will be used for SYS, SYSTEM and PDBADMIN accounts:
The password you entered contains invalid characters. Enter password:
Confirm the password:
Configuring Oracle Listener.
Listener configuration succeeded.
Configuring Oracle Database XE.
Enter SYS user password:
*********
Enter SYSTEM user password:
********
Enter PDBADMIN User Password:
*********
Prepare for db operation
7% complete
Copying database files
29% complete
Creating and starting Oracle instance
30% complete
31% complete
34% complete
38% complete
41% complete
43% complete
Completing Database Creation
47% complete
50% complete
Creating Pluggable Databases
54% complete
71% complete
Executing Post Configuration Actions
93% complete
Running Custom Scripts
100% complete
Database creation complete. For details check the logfiles at:
/opt/oracle/cfgtoollogs/dbca/XE.
Database Information:
Global Database Name:XE
System Identifier(SID):XE
Look at the log file "/opt/oracle/cfgtoollogs/dbca/XE/XE.log" for further details.
Connect to Oracle Database using one of the connect strings:
Pluggable database: centos-learning.surachartopun.com/XEPDB1
Multitenant container database: centos-learning.surachartopun.com
Use https://localhost:5500/em to access Oracle Enterprise Manager for Oracle Database XE
[student@centos-learning ~]$ netstat -ltn |grep 5500
tcp 0 0 127.0.0.1:5500 0.0.0.0:* LISTEN

- As I didn't want to connect 127.0.0.1, I changed binding - "Making Oracle Database EM Express Available to Remote Clients"
SQL> !netstat -ltn |grep 5500
tcp 0 0 127.0.0.1:5500 0.0.0.0:* LISTEN
SQL> !lsnrctl status | grep HTTP
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=127.0.0.1)(PORT=5500))(Security=(my_wallet_directory=/opt/oracle/product/18c/dbhomeXE/admin/XE/xdb_wallet))(Presentation=HTTP)(Session=RAW))
SQL>
SQL>
SQL> EXEC DBMS_XDB.SETLISTENERLOCALACCESS(FALSE);
PL/SQL procedure successfully completed.
SQL> !netstat -ltn |grep 5500
tcp 0 0 0.0.0.0:5500 0.0.0.0:* LISTEN
SQL> !lsnrctl status | grep HTTP
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=centos-learning.surachartopun.com)(PORT=5500))(Security=(my_wallet_directory=/opt/oracle/admin/XE/xdb_wallet))(Presentation=HTTP)(Session=RAW))

- Browsed it - https://IP:5500/em

However, I got some error like "Connection with database failed. Database instance might be down."

- Checked alert log file and fixed.
Error: Global ports off in Root, do SetGlobalPortEnabled(TRUE) to enable.
bash-4.2$ tail -f alert_XE.log
2018-10-22T22:06:32.890217+07:00
Global ports off in Root, do SetGlobalPortEnabled(TRUE) to enable.
2018-10-22T22:06:38.489011+07:00
Global ports off in Root, do SetGlobalPortEnabled(TRUE) to enable.
2018-10-22T22:10:32.402822+07:00
Resize operation completed for file# 3, old size 501760K, new size 512000K
2018-10-22T22:15:55.791490+07:00
Global ports off in Root, do SetGlobalPortEnabled(TRUE) to enable.
2018-10-22T22:18:02.248906+07:00
Global ports off in Root, do SetGlobalPortEnabled(TRUE) to enable.Setting the Global Port for EM Express to Manage a CDB and the PDBs. (It might not be the right solution, but I just wanted to see EM).
SQL> select dbms_xdb_config.getHttpsPort() from dual;
DBMS_XDB_CONFIG.GETHTTPSPORT()
------------------------------
5500
SQL> exec dbms_xdb_config.SetGlobalPortEnabled(TRUE)
PL/SQL procedure successfully completed.- Login again.

It worked fine for now.

Reference: Installation Guide for Linux x86-64

Categories: DBA Blogs

How to allow private connectivity across organizations(GCP)?

Wed, 2018-09-19 02:41

It's interesting, when you would like to allow private connectivity across two VPC networks that they belong to the different project/organization on Google Cloud Platform (GCP).

As google document that we can use VPC Network Peering? VPC Network Peering is a decentralized or distributed approach to multi-project networking. Additional, it works with Compute Engine, Kubernetes Engine,and App Engine flexible environments.

I did a lab about Virtual Private Cloud (VPC) Network Peering. There shows to do VPC Network Peering between VPC networks in the same project. So, I would like to see how it works on across organizations.

Example: On My Organization (my project), I would like to connect server (Private IP Address) on another Organization (another project).

As a subnet CIDR prefix in one peered VPC network cannot overlap with a subnet CIDR prefix in another peered network. So, both VPC networks must have the different CIDR prefix.

My Organization [myproject] {ubuntu-test, default/10.146.0.2} <======> No organization [qwiklabs-gcp***]{privatenet-us-vm/172.16.0.2}

On No organization [qwiklabs-gcp***]: VPC network name is "privatenet".

VPC:

Firewall:

My Organization [myproject]: I used "default" VPC and default firewall.

Then, starting to create "VPC Network Peering".

- To create "VPC Network Peering" on myproject: Networking => "VPC network" => "VPC network peering".
Click "Create Peering Connection". name = "peering-to-lab".
Note: you must know Project ID and VPC network name for network destination.

It should show "Waiting for peer network to connect".

- To create "VPC Network Peering" on another Project: On No organization [qwiklabs-gcp***], "Create Peering Connection". name = "peering-to-mygcp".

After clicking "Create". It should show "Connected" on both projects (if configuration corrects) like.

On myproject:

- Finally, test connection: ssh to my vm and test (ssh) connection to {privatenet-us-vm/172.16.0.2}.
Note: (as firewall allow icmp/ssh). No need to do on firewall.

opun@ubuntu-test:~$ ssh 172.16.0.2
opun@172.16.0.2's password:
Linux privatenet-us-vm 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u4 (2018-08-21) x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Sep 19 06:22:05 2018 from 10.146.0.2
Could not chdir to home directory /home/opun: No such file or directory
$ w
06:22:55 up 43 min, 2 users, load average: 0.00, 0.00, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
opun pts/1 10.146.0.2 06:22 1.00s 0.00s 0.00s w ************Reference: https://cloud.google.com/vpc/docs/vpc-peering

Categories: DBA Blogs

GCP - How to manage SSH keys on VM Instance?

Fri, 2018-04-20 23:13

On Google Cloud Platform, adding SSH keys in Metadata (project-wide public SSH keys). It can help to ssh to every VM instances on Compute Engine easily but it's not a good idea. We are able to do for test, but should not use on Production. We should add SSH Key in OS login.

https://cloud.google.com/compute/docs/instances/adding-removing-ssh-keys#risks

Question: How can we block SSH Keys from Metadata(project-wide public SSH keys) on VM instance?
Answer: We can block by checking "Block project-wide SSH keys" on each instance.

In case we have SSH Key on metadata. We are able to ssh by using private key and login like.

So, we block it... On "Compute Engine" - "VM Instances", click [instance name] and "Edit". To check "Block project-wide SSH keys" and "Save".

VM instance should refuse Key what 's not in SSH Keys of VM instance. (You can remove SSH Keys of project owner on instance, but it will be automatic added when you click "SSH" on GUI).

Additional, we should review and remove SSH Keys in metadata(project-wide public SSH keys), if we ensure we have not used. (Don't remove ssh key of project owner).

After removing, We would like to add SSH Key and don't want to add it in OS login. We are able to add it in SSH Keys on Instance like.

Assume: username is "myuser".

First of all, we have to generate Private and Public Keys. Example uses "PuTTY Key Generator". Because I use "Putty.

Then "Save private key" (We have to use when putty to server) and "Save public key".

To use public key on VM instance, click "Add item".

Example: It's [public key] [username]

then "Save".
Note: In picture, it's highlight about [username]

Open "putty", select "Private key file for authentication", fill in ip address and connect.

it's easy, right?
myuser@centos7:~$ id
uid=1003(myuser) gid=1004(myuser) groups=1004(myuser),4(adm),30(dip),44(video),46(plugdev),1000(google-sudoers)If we use command "id [user in project-wide SSH keys], we still see it, but it's unable to ssh on this VM instance.
myuser@centos7:~$ id opun
uid=1001(opun) gid=1002(opun) groups=1002(opun),4(adm),30(dip),44(video),46(plugdev),

Reference:

https://cloud.google.com/compute/docs/instances/adding-removing-ssh-keys

Categories: DBA Blogs

Install OEM 13c Silent mode

Wed, 2016-09-14 09:15

How to install Oracle EM 13c? It's the first time that I have a chance to install Oracle EM 13c. I remember last time I installed Oracle EM 12c. I have to install weblogic first. On OEM 13c, I don't need to do something like that. It installs WLS -12.1.0.3 , JDK – 1.7.0_80.

In this post, I installed Oracle EM 13c Silent mode with "SMALL" deployment. I assume, I installed Oracle Database Software 12.1.0.2 (Software Only), downloaded "12.1.0.2 DB Template for EM 13.1.0.0 on Linux x86-64" template and uncompressed it in $ORACLE_HOME/assistants/dbca/templates folder.

OS:
ip_local_port_range parameter is set between 11000 to 65000
soft nofiles = 30000
file max = 65536
Database:
optimizer_adaptive_features = FALSE

Then, created database by using template.
[oracle@em13c ~]$ diff /u01/app/oracle/product/12.1.0/dbhome_1/assistants/dbca/dbca.rsp dbca-emrep.rsp
78c78
< GDBNAME = "orcl12c.us.oracle.com"
---
> GDBNAME = "emrep"
205c205
< SID = "orcl12c"
---
> SID = "emrep"
265c265
< TEMPLATENAME = "General_Purpose.dbc"
---
> TEMPLATENAME = "12.1.0.2.0_Database_Template_for_EM13_1_0_0_0_Small_deployment.dbc"
286c286
< #SYSPASSWORD = "password"
---
> SYSPASSWORD = "password1"
296c296
< #SYSTEMPASSWORD = "password"
---
> SYSTEMPASSWORD = "password1"
467c467
< #DATAFILEDESTINATION =
---
> DATAFILEDESTINATION = /u02/oradata
487c487
< #STORAGETYPE=FS
---
> STORAGETYPE=FS
525c525
< #CHARACTERSET = "US7ASCII"
---
> CHARACTERSET = "AL32UTF8"
[oracle@em13c ~]$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1
[oracle@em13c ~]$ export ORACLE_BASE=/u01/app/oracle
[oracle@em13c ~]$ /u01/app/oracle/product/12.1.0/dbhome_1/bin/dbca -silent -responseFile dbca-emrep.rsp
Copying database files
1% complete
3% complete
10% complete
16% complete
23% complete
30% complete
33% complete
Creating and starting Oracle instance
35% complete
40% complete
44% complete
49% complete
50% complete
53% complete
55% complete
Completing Database Creation
58% complete
62% complete
65% complete
75% complete
85% complete
88% complete
Running Custom Scripts
100% complete
Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/emrep/emrep.log" for further details.

Checked database after installing.
[oracle@em13c ~]$ . oraenv
ORACLE_SID = [oracle] ? emrep
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@em13c ~]$ sqlplus / as sysdba
SQL*Plus: Release 12.1.0.2.0 Production on Wed Sep 14 14:39:32 2016
Copyright (c) 1982, 2014, Oracle. All rights reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
emrep
SQL> select name from v$datafile;
NAME
------------------------------------------------------------------------------------------------------------------------------------------------------
/u02/oradata/emrep/system01.dbf
/u02/oradata/emrep/sysaux01.dbf
/u02/oradata/emrep/undotbs01.dbf
/u02/oradata/emrep/users01.dbf
/u02/oradata/emrep/mgmt_depot.dbf
/u02/oradata/emrep/mgmt.dbf
/u02/oradata/emrep/mgmt_ad4j.dbf
7 rows selected.
SQL> show parameter spfile
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string /u01/app/oracle/product/12.1.0
/dbhome_1/dbs/spfileemrep.ora
SQL> show parameter optimizer_adaptive_features
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
optimizer_adaptive_features boolean FALSE
SQL> archive log list
Database log mode No Archive Mode
Automatic archival Disabled
Archive destination /u01/app/oracle/product/12.1.0/dbhome_1/dbs/arch
Oldest online log sequence 1
Current log sequence 1
SQL> show parameter control_files
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
control_files string /u02/oradata/emrep/control01.c
tl, /u02/oradata/emrep/control02.ctlConfigured listener.

[oracle@em13c ~]$ diff /u01/app/oracle/product/12.1.0/dbhome_1/assistants/netca/netca.rsp /home/oracle/netca.rsp
46c46
< #SHOW_GUI=false
---
> SHOW_GUI=false
60c60
< #LOG_FILE=""/oracle12cHome/network/tools/log/netca.log""
---
> LOG_FILE=""/u01/app/oracle/product/12.1.0/dbhome_1/network/tools/log/netca.log""
113c113
< NSN_NUMBER=1
---
> #NSN_NUMBER=1
117c117
< NSN_NAMES={"EXTPROC_CONNECTION_DATA"}
---
> #NSN_NAMES={"EXTPROC_CONNECTION_DATA"}
121c121
< NSN_SERVICE={"PLSExtProc"}
---
> #NSN_SERVICE={"PLSExtProc"}
127c127
< NSN_PROTOCOLS={"TCP;HOSTNAME;1521"}
---
> #NSN_PROTOCOLS={"TCP;HOSTNAME;1521"}
[oracle@em13c ~]$ netca -silent -orahome /u01/app/oracle/product/12.1.0/dbhome_1 -responsefile /home/oracle/netca.rsp
Wed Sep 14 15:26:52 ICT 2016 Oracle Net Configuration Assistant
Parsing command line arguments:
Parameter "silent" = true
Parameter "orahome" = /u01/app/oracle/product/12.1.0/dbhome_1
Parameter "responsefile" = /home/oracle/netca.rsp
Parameter "log" = /u01/app/oracle/product/12.1.0/dbhome_1/network/tools/log/netca.log
Done parsing command line arguments.
Oracle Net Services Configuration:
Profile configuration complete.
Oracle Net Listener Startup:
Running Listener Control:
/u01/app/oracle/product/12.1.0/dbhome_1/bin/lsnrctl start LISTENER
Listener Control complete.
Listener started successfully.
Listener configuration complete.
Oracle Net Services configuration successful. The exit code is 0
[oracle@em13c ~]$ lsnrctl status
LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 14-SEP-2016 15:27:35
Copyright (c) 1991, 2014, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=em13c.surachartopun.com)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 12.1.0.2.0 - Production
Start Date 14-SEP-2016 15:26:52
Uptime 0 days 0 hr. 0 min. 42 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /u01/app/oracle/product/12.1.0/dbhome_1/network/admin/listener.ora
Listener Log File /u01/app/oracle/diag/tnslsnr/em13c/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=em13c.surachartopun.com)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Services Summary...
Service "emrep" has 1 instance(s).
Instance "emrep", status READY, has 1 handler(s) for this service...
Service "emrepXDB" has 1 instance(s).
Instance "emrep", status READY, has 1 handler(s) for this service...
The command completed successfully

It was time to install Oracle EM 13c Silent mode.

Uncompressed software and created response file.

[oracle@em13c scratch]$ ls em13100*
em13100_linux64-2.zip em13100_linux64-3.zip em13100_linux64-4.zip em13100_linux64-5.zip em13100_linux64.bin
[oracle@em13c scratch]$ ./em13100_linux64.bin -getResponseFileTemplates -outputLoc /home/oracle/RSP
0%...............................................................100%
Launcher log file is /tmp/OraInstall2016-09-14_03-31-39PM/launcher2016-09-14_03-31-39PM.log.
Copying response file template(s)
to /home/oracle/RSP ...
Copying response file template upgrade.rsp
Copying response file template software_only.rsp
Copying response file template new_install.rsp
Finished copying response file template(s)

Modified response file and installed OEM13c. (I chose to use new install)

[oracle@em13c scratch]$ diff /home/oracle/RSP/new_install.rsp /home/oracle/RSP/my-new_install.rsp
38c38
< UNIX_GROUP_NAME=<string>
---
> UNIX_GROUP_NAME=dba
46c46
< INVENTORY_LOCATION=<string>
---
> INVENTORY_LOCATION=/u01/app/oraInventory
60c60
< DECLINE_SECURITY_UPDATES=false
---
> DECLINE_SECURITY_UPDATES=true
67c67
< MYORACLESUPPORT_USERNAME=<string>
---
> #MYORACLESUPPORT_USERNAME=<string>
74c74
< MYORACLESUPPORT_PASSWORD=<string>
---
> #MYORACLESUPPORT_PASSWORD=<string>
92c92
< STAGE_LOCATION=<string>
---
> #STAGE_LOCATION=<string>
101c101
< MYORACLESUPPORT_USERNAME_FOR_SOFTWAREUPDATES=<string>
---
> #MYORACLESUPPORT_USERNAME_FOR_SOFTWAREUPDATES=<string>
110c110
< MYORACLESUPPORT_PASSWORD_FOR_SOFTWAREUPDATES=<string>
---
> #MYORACLESUPPORT_PASSWORD_FOR_SOFTWAREUPDATES=<string>
117c117
< PROXY_USER=<string>
---
> #PROXY_USER=<string>
124c124
< PROXY_PWD=<string>
---
> #PROXY_PWD=<string>
131c131
< PROXY_HOST=<string>
---
> #PROXY_HOST=<string>
138c138
< PROXY_PORT=<string>
---
> #PROXY_PORT=<string>
145c145
< ORACLE_MIDDLEWARE_HOME_LOCATION=<string>
---
> ORACLE_MIDDLEWARE_HOME_LOCATION=/u01/app/oracle/mw13c
153c153
< ORACLE_HOSTNAME=<string>
---
> ORACLE_HOSTNAME=em13c.surachartopun.com
160c160
< AGENT_BASE_DIR=<string>
---
> AGENT_BASE_DIR=/u01/app/oracle/agent13c
167c167
< WLS_ADMIN_SERVER_USERNAME=<string>
---
> WLS_ADMIN_SERVER_USERNAME=weblogic
174c174
< WLS_ADMIN_SERVER_PASSWORD=<string>
---
> WLS_ADMIN_SERVER_PASSWORD=password1
181c181
< WLS_ADMIN_SERVER_CONFIRM_PASSWORD=<string>
---
> WLS_ADMIN_SERVER_CONFIRM_PASSWORD=password1
188c188
< NODE_MANAGER_PASSWORD=<string>
---
> NODE_MANAGER_PASSWORD=password1
195c195
< NODE_MANAGER_CONFIRM_PASSWORD=<string>
---
> NODE_MANAGER_CONFIRM_PASSWORD=password1
202c202
< ORACLE_INSTANCE_HOME_LOCATION=<string>
---
> ORACLE_INSTANCE_HOME_LOCATION=/u01/app/oracle/gc_inst
221c221
< SOFTWARE_LIBRARY_LOCATION=<string>
---
> SOFTWARE_LIBRARY_LOCATION=/u01/app/oracle/swlib
229c229
< DATABASE_HOSTNAME=<string>
---
> DATABASE_HOSTNAME=em13c.surachartopun.com
237c237
< LISTENER_PORT=<string>
---
> LISTENER_PORT=1521
245c245
< SERVICENAME_OR_SID=<string>
---
> SERVICENAME_OR_SID=emrep
253c253
< SYS_PASSWORD=<string>
---
> SYS_PASSWORD=password1
261c261
< SYSMAN_PASSWORD=<string>
---
> SYSMAN_PASSWORD=password1
268c268
< SYSMAN_CONFIRM_PASSWORD=<string>
---
> SYSMAN_CONFIRM_PASSWORD=password1
275c275
< DEPLOYMENT_SIZE=MEDIUM
---
> DEPLOYMENT_SIZE=SMALL
283c283
< MANAGEMENT_TABLESPACE_LOCATION=<string>
---
> #MANAGEMENT_TABLESPACE_LOCATION=<string>
291c291
< CONFIGURATION_DATA_TABLESPACE_LOCATION=<string>
---
> #CONFIGURATION_DATA_TABLESPACE_LOCATION=<string>
299c299
< JVM_DIAGNOSTICS_TABLESPACE_LOCATION=<string>
---
> #JVM_DIAGNOSTICS_TABLESPACE_LOCATION=<string>
306c306
< AGENT_REGISTRATION_PASSWORD=<string>
---
> AGENT_REGISTRATION_PASSWORD=password1
313c313
< AGENT_REGISTRATION_CONFIRM_PASSWORD=<string>
---
> AGENT_REGISTRATION_CONFIRM_PASSWORD=password1
320c320
< STATIC_PORTS_FILE=<string>
---
> #STATIC_PORTS_FILE=/u01/app/oracle/mw13c/install/portlist.ini
368c368
< CONFIG_LOCATION=<string>
---
> #CONFIG_LOCATION=<string>
376c376
< CLUSTER_LOCATION=<string>
---
> #CLUSTER_LOCATION=<string>Note: As I created database from template. So. I didn't set *TABLESPACE_LOCATION variables. For variables, that didn't define. we should comment it.
[oracle@em13c scratch]$ ./em13100_linux64.bin -silent -ignoreSysPrereqs -responseFile /home/oracle/RSP/my-new_install.rsp
0%...............................................................100%
Launcher log file is /tmp/OraInstall2016-09-14_05-56-59PM/launcher2016-09-14_05-56-59PM.log.
Starting Oracle Universal Installer
Checking if CPU speed is above 300 MHz. Actual 2300.094 MHz Passed
Checking swap space: must be greater than 512 MB. Actual 0 MB Failed <<<<
Checking if this platform requires a 64-bit JVM. Actual 64 Passed (64-bit not required)
Checking temp space: must be greater than 300 MB. Actual 32844 MB Passed
>>> Ignoring failure(s) of required prerequisite checks. Continuing...

Preparing to launch the Oracle Universal Installer from /tmp/OraInstall2016-09-14_05-56-59PM
====Prereq Config Location main===
/tmp/OraInstall2016-09-14_05-56-59PM/stage/prereq
EMGCInstaller args -scratchPath
EMGCInstaller args /tmp/OraInstall2016-09-14_05-56-59PM
EMGCInstaller args -sourceType
EMGCInstaller args network
EMGCInstaller args -timestamp
EMGCInstaller args 2016-09-14_05-56-59PM
EMGCInstaller args -paramFile
EMGCInstaller args /tmp/sfx_g7ViFx/Disk1/install/linux64/oraparam.ini
EMGCInstaller args -silent
EMGCInstaller args -ignoreSysPrereqs
EMGCInstaller args -responseFile
EMGCInstaller args /home/oracle/RSP/my-new_install.rsp
EMGCInstaller args -nocleanUpOnExit
DiskLoc inside SourceLoc/u02/SRC/EM13c/scratch
EMFileLoc:/tmp/OraInstall2016-09-14_05-56-59PM/oui/em/
ScratchPathValue :/tmp/OraInstall2016-09-14_05-56-59PM
EMGCInstallUpdatesInfoOnNext:: calling actionOnClickofNext
Now in EMGCInstallUpdatesInfoOnNext.actionsOnClickofNext
EMGCInstallUpdatesInfoOnNext:: End of actionOnClickofNext
Session log file is /tmp/OraInstall2016-09-14_05-56-59PM/install2016-09-14_05-56-59PM.log
.
.
.
Setup in progress (Wednesday, September 14, 2016 6:03:38 PM ICT)
Setup successful
Saving inventory (Wednesday, September 14, 2016 6:03:38 PM ICT)
Saving inventory complete
End of install phases.(Wednesday, September 14, 2016 6:03:39 PM ICT)
&Decline License Agreement/u01/app/oracle/mw13c/sysman/install/plugins/oracle.sysman.si/13.1.1.0.0/oracle.sysman.si.discovery.plugin-13.1.1.0.0.farb
Session log file is /tmp/OraInstall2016-09-14_05-56-59PM/install2016-09-14_05-56-59PM.log
........................................................................
Installation in progress (Wednesday, September 14, 2016 6:03:40 PM ICT)
72% Done.
Install successful
Linking in progress (Wednesday, September 14, 2016 6:03:40 PM ICT)
Link successful
Setup in progress (Wednesday, September 14, 2016 6:03:40 PM ICT)
Setup successful
Saving inventory (Wednesday, September 14, 2016 6:03:40 PM ICT)
Saving inventory complete
End of install phases.(Wednesday, September 14, 2016 6:03:41 PM ICT)
OMS OracleHome :/u01/app/oracle/mw13c
Applying the required one-off patches.
13NGCHEKAGGREGATE : oracle.sysman.top.agent
13NGCHEKAGGREGATE : oracle.sysman.top.oms
13NGCHEKAGGREGATE : encap_oms
13NGCHEKAGGREGATE : OuiConfigVariables
13NGCHEKAGGREGATE : OuiConfigVariables
2016-09-14_06-07-13PM: Configuration Assistant "Plugins Prerequisites Check" is in progress.
2016-09-14_06-07-32PM: Configuration Assistant "Plugins Prerequisites Check" has Succeeded.
2016-09-14_06-07-32PM: Configuration Assistant "Repository Out Of Box Configuration" is in progress.
2016-09-14_06-42-39PM: Configuration Assistant "Repository Out Of Box Configuration" has Succeeded.
2016-09-14_06-42-39PM: Configuration Assistant "OMS Configuration" is in progress.

Executing the OMSCA command...
Check the log files of the OMS Configuration Assistant at: /u01/app/oracle/mw13c/cfgtoollogs/omsca

OMS Configuration Assistant completed successfully.

2016-09-14_06-51-03PM: Configuration Assistant "OMS Configuration" has Succeeded.
2016-09-14_06-51-03PM: Configuration Assistant "Plugins Deployment and Configuration" is in progress.

2016-09-14_06-59-48PM: Configuration Assistant "Plugins Deployment and Configuration" has Succeeded.
2016-09-14_06-59-48PM: Configuration Assistant "BI Publisher Configuration" is in progress.
2016-09-14_07-05-20PM: Configuration Assistant "BI Publisher Configuration" has Succeeded.
2016-09-14_07-05-20PM: Configuration Assistant "Start Oracle Management Service" is in progress.
Starting OMS ...
Executing the command: /u01/app/oracle/mw13c/bin/emctl start oms
Starting of OMS is successful.
Starting export oms config...
Executing command: /u01/app/oracle/mw13c/bin/emctl exportconfig oms -dir /u01/app/oracle/gc_inst/em/EMGC_OMS1/sysman/backup
Export config of OMS is successful.
2016-09-14_07-09-43PM: Configuration Assistant "Start Oracle Management Service" has Succeeded.
2016-09-14_07-09-43PM: Configuration Assistant "Agent Configuration Assistant" is in progress.
Getting Inet Addresses for host em13c.surachartopun.com
** Agent Port Check completed successfully.**
AgentConfiguration:agent configuration has been started
Validating OMS_HOST and EM_UPLOAD_PORT
Performing free port detection..
AgentConfiguration: Executing emctl deploy agent command...
AgentConfiguration: Executing emctl config agent command...
AgentConfiguration:agent configuration finished with status = true
2016-09-14_07-12-12PM: Configuration Assistant "Agent Configuration Assistant" has Succeeded.
*** The Installation was Successful. ***

This information is also available at:
/u01/app/oracle/mw13c/install/setupinfo.txt
See the following for information pertaining to your Enterprise Manager installation:

Use the following URL to access:
1. Enterprise Manager Cloud Control URL: https://em13c.surachartopun.com:7803/em
2. Admin Server URL: https://em13c.surachartopun.com:7102/console
3. BI Publisher URL: https://em13c.surachartopun.com:9803/xmlpserver
The following details need to be provided while installing an additional OMS:
1. Admin Server Host Name: em13c.surachartopun.com
2. Admin Server Port: 7102
You can find the details on ports used by this deployment at : /u01/app/oracle/mw13c/install/portlist.ini

NOTE:
An encryption key has been generated to encrypt sensitive data in the Management Repository. If this key is lost, all encrypted data in the Repository becomes unusable.
A backup of the OMS configuration is available in /u01/app/oracle/gc_inst/em/EMGC_OMS1/sysman/backup on host em13c.surachartopun.com. See Cloud Control Administrators Guide for details on how to back up and recover an OMS.
NOTE: This backup is valid only for the initial OMS configuration. For example, it will not reflect plug-ins installed later, topology changes like the addition of a load balancer, or changes to other properties made using emctl or emcli. Backups should be created on a regular basis to ensure they capture the current OMS configuration. Use the following command to backup the OMS configuration:
/u01/app/oracle/mw13c/bin/emctl exportconfig oms -dir <backup dir>
Logs successfully copied to /u01/app/oraInventory/logs.

Note: su to "root" and run /u01/app/oracle/mw13c/allroot.sh script.
Starting to execute allroot.sh .........
Starting to execute /u01/app/oracle/mw13c/root.sh ......
/etc exist
Creating /etc/oragchomelist file...
/u01/app/oracle/mw13c
Finished product-specific root actions.
/etc exist
Finished execution of /u01/app/oracle/mw13c/root.sh ......

Starting to execute /u01/app/oracle/agent13c/agent_13.1.0.0.0/root.sh ......
Finished product-specific root actions.
/etc exist
Finished execution of /u01/app/oracle/agent13c/agent_13.1.0.0.0/root.sh ......

[root@em13c ~]# cat /etc/oragchomelist
/u01/app/oracle/mw13c
/u01/app/oracle/agent13c/agent_13.1.0.0.0:/u01/app/oracle/agent13c/agent_inst
[root@em13c ~]# find /etc/ -name ???gcstartup
/etc/rc.d/rc2.d/S98gcstartup
/etc/rc.d/rc2.d/K19gcstartup
/etc/rc.d/rc3.d/S98gcstartup
/etc/rc.d/rc3.d/K19gcstartup
/etc/rc.d/rc5.d/S98gcstartup
/etc/rc.d/rc5.d/K19gcstartup
[root@em13c ~]# cat /etc/oragchomelist
/u01/app/oracle/mw13c
/u01/app/oracle/agent13c/agent_13.1.0.0.0:/u01/app/oracle/agent13c/agent_inst
[root@em13c ~]# find /etc/ -name gcstartup
/etc/rc.d/init.d/gcstartup
[root@em13c ~]# find /etc/ -name ???gcstartup
/etc/rc.d/rc2.d/S98gcstartup
/etc/rc.d/rc2.d/K19gcstartup
/etc/rc.d/rc3.d/S98gcstartup
/etc/rc.d/rc3.d/K19gcstartup
/etc/rc.d/rc5.d/S98gcstartup
/etc/rc.d/rc5.d/K19gcstartup

After installed, I was able to check OMS and agent status.

[oracle@em13c scratch]$ /u01/app/oracle/mw13c/bin/emctl status oms -details
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host : em13c.surachartopun.com
HTTP Console Port : 7788
HTTPS Console Port : 7803
HTTP Upload Port : 4889
HTTPS Upload Port : 4903
EM Instance Home : /u01/app/oracle/gc_inst/em/EMGC_OMS1
OMS Log Directory Location : /u01/app/oracle/gc_inst/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL: https://em13c.surachartopun.com:7803/em
Upload URL: https://em13c.surachartopun.com:4903/empbs/upload
WLS Domain Information
Domain Name : GCDomain
Admin Server Host : em13c.surachartopun.com
Admin Server HTTPS Port: 7102
Admin Server is RUNNING
Oracle Management Server Information
Managed Server Instance Name: EMGC_OMS1
Oracle Management Server Instance Host: em13c.surachartopun.com
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up
BI Publisher Server Information
BI Publisher Managed Server Name: BIP
BI Publisher Server is Up
BI Publisher HTTP Managed Server Port : 9701
BI Publisher HTTPS Managed Server Port : 9803
BI Publisher HTTP OHS Port : 9788
BI Publisher HTTPS OHS Port : 9851
BI Publisher is locked.
BI Publisher Server named 'BIP' running at URL: https://em13c.surachartopun.com:9851/xmlpserver
BI Publisher Server Logs: /u01/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/BIP/logs/
BI Publisher Log : /u01/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/BIP/logs/bipublisher/bipublisher.log
[oracle@em13c scratch]$ /u01/app/oracle/agent13c/agent_13.1.0.0.0/bin/emctl status agent
[Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 13.1.0.0.0
OMS Version : 13.1.0.0.0
Protocol Version : 12.1.0.1.0
Agent Home : /u01/app/oracle/agent13c/agent_inst
Agent Log Directory : /u01/app/oracle/agent13c/agent_inst/sysman/log
Agent Binaries : /u01/app/oracle/agent13c/agent_13.1.0.0.0
Core JAR Location : /u01/app/oracle/agent13c/agent_13.1.0.0.0/jlib
Agent Process ID : 26632
Parent Process ID : 26568
Agent URL : https://em13c.surachartopun.com:3872/emd/main/
Local Agent URL in NAT : https://em13c.surachartopun.com:3872/emd/main/
Repository URL : https://em13c.surachartopun.com:4903/empbs/upload
Started at : 2016-09-14 19:11:49
Started by user : oracle
Operating System : Linux version 3.10.0-327.28.3.el7.x86_64 (amd64)
Number of Targets : 32
Last Reload : (none)
Last successful upload : 2016-09-14 20:03:22
Last attempted upload : 2016-09-14 20:03:22
Total Megabytes of XML files uploaded so far : 1.56
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0
Available disk space on upload filesystem : 40.54%
Collection Status : Collections enabled
Heartbeat Status : Ok
Last attempted heartbeat to OMS : 2016-09-14 20:05:00
Last successful heartbeat to OMS : 2016-09-14 20:05:00
Next scheduled heartbeat to OMS : 2016-09-14 20:06:00
---------------------------------------------------------------
Agent is Running and Ready

Connected OEM 13c by browser.

Reference:
http://www.oracle.com/technetwork/oem/install-upgrade/pr-install-em13-2835293.pdf

https://docs.oracle.com/cd/E63000_01/

http://www.oracle.com/technetwork/oem/enterprise-manager/downloads/index.html

Categories: DBA Blogs

Packt - Time to learn Oracle and Linux

Sat, 2016-01-23 00:01

What is your resolution for learning? Learn Oracle, Learn Linux or both. It' s a good news for people who are interested in improving Oracle and Linux skills. Packt Promotional (discount of 50%) for eBooks & videos from today until 23rd Feb, 2016.

Oracle (Code:XM6lxr0)

Linux (Code:ILYTW)

Categories: DBA Blogs

AWS EC2 API tools: Create snapshot & Check Data in snapshot

Wed, 2015-05-27 02:38

After installed AWS EC2 API tools, It's time for example create/delete snapshot.

- Creating snapshot.
ubuntu@ip-x-x-x-x~$ ec2-describe-volumes
VOLUME vol-41885f55 8 snap-d00ac9e4 ap-southeast-1a in-use 2015-05-26T09:07:04+0000 gp2 24
ATTACHMENT vol-41885f55 i-d6cdb71a /dev/sda1 attached 2015-05-26T09:07:04+0000 true
ubuntu@ip-x-x-x-x:~$ ec2-create-snapshot -d vol-41885f55-$(date +%Y%m%d%H%M) vol-41885f55
SNAPSHOT snap-b20a8c87 vol-41885f55 pending 2015-05-27T05:46:58+0000 843870022970 8 vol-41885f55-201505270546
ubuntu@ip-x-x-x-x:~$ ec2-describe-snapshots
SNAPSHOT snap-b20a8c87 vol-41885f55 pending 2015-05-27T05:46:58+0000 0% 843870022970 8 vol-41885f55-201505270546
ubuntu@ip-x-x-x-x:~$ ec2-create-snapshot -d vol-41885f55-$(date +%Y%m%d%H%M) vol-41885f55
SNAPSHOT snap-bea0d28b vol-41885f55 pending 2015-05-27T05:50:11+0000 843870022970 8 vol-41885f55-201505270550
ubuntu@ip-x-x-x-x:~$ ec2-describe-snapshots
SNAPSHOT snap-b20a8c87 vol-41885f55 completed 2015-05-27T05:46:58+0000 100% 843870022970 8 vol-41885f55-201505270546
SNAPSHOT snap-bea0d28b vol-41885f55 completed 2015-05-27T05:50:11+0000 100% 843870022970 8 vol-41885f55-201505270550- Deleting snapshot (delete snap-b20a8c87).
ubuntu@ip-x-x-x-x:~$ ec2-describe-snapshots |head -1| awk '{print $2}'|xargs ec2-delete-snapshot
SNAPSHOT snap-b20a8c87
ubuntu@ip-x-x-x-x:~$ ec2-describe-snapshots
SNAPSHOT snap-bea0d28b vol-41885f55 completed 2015-05-27T05:50:11+0000 100% 843870022970 8 vol-41885f55-201505270550How to check data in "snap-bea0d28b"? Checking idea on AWS, look like we must create Volume from snapshot and attach it to Instance.
- Creating Volume > Attach to Instance and Mount.
ubuntu@ip-x-x-x-x:~$ ec2-describe-volumes
VOLUME vol-41885f55 8 snap-d00ac9e4 ap-southeast-1a in-use 2015-05-26T09:07:04+0000 gp2 24
ATTACHMENT vol-41885f55 i-d6cdb71a /dev/sda1 attached 2015-05-26T09:07:04+0000 true
ubuntu@ip-x-x-x-x:~$ ec2-describe-availability-zones
AVAILABILITYZONE ap-southeast-1a available ap-southeast-1
AVAILABILITYZONE ap-southeast-1b available ap-southeast-1
ubuntu@ip-x-x-x-x:~$ ec2-create-volume -s 8 --snapshot snap-bea0d28b -z ap-southeast-1a
VOLUME vol-d15087c5 8 snap-bea0d28b ap-southeast-1a creating 2015-05-27T06:24:00+0000 standard
ubuntu@ip-x-x-x-x:~$ ec2-describe-volumes
VOLUME vol-41885f55 8 snap-d00ac9e4 ap-southeast-1a in-use 2015-05-26T09:07:04+0000 gp2 24
ATTACHMENT vol-41885f55 i-d6cdb71a /dev/sda1 attached 2015-05-26T09:07:04+0000 true
VOLUME vol-d15087c5 8 snap-bea0d28b ap-southeast-1a available 2015-05-27T06:24:00+0000 standard
ubuntu@ip-x-x-x-x:~$ sudo fdisk -l
Disk /dev/xvda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders, total 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/xvda1 * 16065 16771859 8377897+ 83 Linux
ubuntu@ip-x-x-x-x:~$ ec2-attach-volume vol-d15087c5 -i i-d6cdb71a -d sdf
ATTACHMENT vol-d15087c5 i-d6cdb71a sdf attaching 2015-05-27T06:31:16+0000
ubuntu@ip-x-x-x-x:~$ sudo fdisk -l
Disk /dev/xvda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders, total 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/xvda1 * 16065 16771859 8377897+ 83 Linux
Disk /dev/xvdf: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders, total 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/xvdf1 * 16065 16771859 8377897+ 83 Linux
ubuntu@ip-x-x-x-x:~$

ubuntu@ip-x-x-x-x:~$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda1 8115168 1212140 6467752 16% /
none 4 0 4 0% /sys/fs/cgroup
udev 503188 12 503176 1% /dev
tmpfs 101632 332 101300 1% /run
none 5120 0 5120 0% /run/lock
none 508144 0 508144 0% /run/shm
none 102400 0 102400 0% /run/user
ubuntu@ip-x-x-x-x:~$ sudo mount /dev/xvdf1 /mnt/
ubuntu@ip-x-x-x-x:~$ ls -l /mnt/
total 92
drwxr-xr-x 2 root root 4096 May 26 09:35 bin
drwxr-xr-x 3 root root 4096 Mar 25 11:52 boot
drwxr-xr-x 5 root root 4096 Mar 25 11:53 dev
drwxr-xr-x 105 root root 4096 May 26 09:35 etc
drwxr-xr-x 3 root root 4096 May 26 09:07 home
lrwxrwxrwx 1 root root 33 Mar 25 11:51 initrd.img -> boot/initrd.img-3.13.0-48-generic
drwxr-xr-x 21 root root 4096 May 26 09:35 lib
drwxr-xr-x 2 root root 4096 Mar 25 11:50 lib64
drwx------ 2 root root 16384 Mar 25 11:53 lost+found
drwxr-xr-x 2 root root 4096 Mar 25 11:50 media
drwxr-xr-x 2 root root 4096 Apr 10 2014 mnt
drwxr-xr-x 2 root root 4096 Mar 25 11:50 opt
drwxr-xr-x 2 root root 4096 Apr 10 2014 proc
drwx------ 3 root root 4096 May 26 09:07 root
drwxr-xr-x 3 root root 4096 Mar 25 11:53 run
drwxr-xr-x 2 root root 4096 May 26 09:35 sbin
drwxr-xr-x 2 root root 4096 Mar 25 11:50 srv
drwxr-xr-x 2 root root 4096 Mar 13 2014 sys
drwxrwxrwt 6 root root 4096 May 27 05:38 tmp
drwxr-xr-x 10 root root 4096 Mar 25 11:50 usr
drwxr-xr-x 12 root root 4096 Mar 25 11:52 var
lrwxrwxrwx 1 root root 30 Mar 25 11:51 vmlinuz -> boot/vmlinuz-3.13.0-48-generic
ubuntu@ip-x-x-x-x:~$ ls /mnt/
bin boot dev etc home initrd.img lib lib64 lost+found media mnt opt proc root run sbin srv sys tmp usr var vmlinuz
ubuntu@ip-x-x-x-x:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.8G 1.2G 6.2G 16% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 492M 12K 492M 1% /dev
tmpfs 100M 332K 99M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 497M 0 497M 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/xvdf1 7.8G 1.2G 6.2G 16% /mnt- After checking data, We can unmount and remove it.
ubuntu@ip-x-x-x-x:~$ sudo umount /mnt
ubuntu@ip-x-x-x-x:~$
ubuntu@ip-x-x-x-x:~$
ubuntu@ip-x-x-x-x:~$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda1 8115168 1212140 6467752 16% /
none 4 0 4 0% /sys/fs/cgroup
udev 503188 12 503176 1% /dev
tmpfs 101632 332 101300 1% /run
none 5120 0 5120 0% /run/lock
none 508144 0 508144 0% /run/shm
none 102400 0 102400 0% /run/user

ubuntu@ip-x-x-x-x:~$ sudo fdisk -l
Disk /dev/xvda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders, total 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/xvda1 * 16065 16771859 8377897+ 83 Linux
Disk /dev/xvdf: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders, total 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/xvdf1 * 16065 16771859 8377897+ 83 Linux
ubuntu@ip-x-x-x-x:~$
ubuntu@ip-x-x-x-x:~$ ec2-describe-volumes
VOLUME vol-41885f55 8 snap-d00ac9e4 ap-southeast-1a in-use 2015-05-26T09:07:04+0000 gp2 24
ATTACHMENT vol-41885f55 i-d6cdb71a /dev/sda1 attached 2015-05-26T09:07:04+0000 true
VOLUME vol-d15087c5 8 snap-bea0d28b ap-southeast-1a in-use 2015-05-27T06:24:00+0000 standard
ATTACHMENT vol-d15087c5 i-d6cdb71a sdf attached 2015-05-27T06:31:16+0000 false
ubuntu@ip-x-x-x-x:~$ ec2-detach-volume vol-d15087c5 -i i-d6cdb71a
ATTACHMENT vol-d15087c5 i-d6cdb71a sdf detaching 2015-05-27T06:31:16+0000
ubuntu@ip-x-x-x-x:~$ ec2-describe-volumes
VOLUME vol-41885f55 8 snap-d00ac9e4 ap-southeast-1a in-use 2015-05-26T09:07:04+0000 gp2 24
ATTACHMENT vol-41885f55 i-d6cdb71a /dev/sda1 attached 2015-05-26T09:07:04+0000 true
VOLUME vol-d15087c5 8 snap-bea0d28b ap-southeast-1a in-use 2015-05-27T06:24:00+0000 standard
ATTACHMENT vol-d15087c5 i-d6cdb71a sdf detaching 2015-05-27T06:31:16+0000 falseubuntu@ip-x-x-x-x:~$ ec2-describe-volumes
VOLUME vol-41885f55 8 snap-d00ac9e4 ap-southeast-1a in-use 2015-05-26T09:07:04+0000 gp2 24
ATTACHMENT vol-41885f55 i-d6cdb71a /dev/sda1 attached 2015-05-26T09:07:04+0000 true
VOLUME vol-d15087c5 8 snap-bea0d28b ap-southeast-1a available 2015-05-27T06:24:00+0000 standard
ubuntu@ip-x-x-x-x:~$ ec2-delete-volume vol-d15087c5
VOLUME vol-d15087c5
ubuntu@ip-x-x-x-x:~$ ec2-describe-volumes
VOLUME vol-41885f55 8 snap-d00ac9e4 ap-southeast-1a in-use 2015-05-26T09:07:04+0000 gp2 24
ATTACHMENT vol-41885f55 i-d6cdb71a /dev/sda1 attached 2015-05-26T09:07:04+0000 trueLook like it's easy to use and adapt with script.

Categories: DBA Blogs

AWS EC2 API tools: Installation

Wed, 2015-05-27 02:08

AWS EC2 API tools help too much for Amazon EC2 to register and launch instances, manipulate security groups, and more. Someone asked me to backup EC2 instance. I thought to use it for backup script. Anyway, No need to explain more how to install Amazon EC2 API tools on Ubuntu? Just say thank for EC2StartersGuide. I fellow this link and installed it easily. Additional, I used this Link for more idea about java.
- Adding Repository and Install EC2 API tools.
ubuntu@ip-x-x-x-x:~$ sudo apt-add-repository ppa:awstools-dev/awstools
Up to date versions of several tools from AWS.
Use this repository by:
sudo apt-add-repository ppa:awstools-dev/awstools
sudo apt-get update
sudo apt-get install ec2-api-tools
.
.
.
ubuntu@ip-x-x-x-x:~$ sudo apt-get update
ubuntu@ip-x-x-x-x:~$ sudo apt-get install ec2-api-tools
ubuntu@ip-x-x-x-x:~$ sudo apt-get install -y openjdk-7-jre
ubuntu@ip-x-x-x-x:~$ file $(which java)
/usr/bin/java: symbolic link to `/etc/alternatives/java'
ubuntu@ip-x-x-x-x:~$ file /etc/alternatives/java
/etc/alternatives/java: symbolic link to `/usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java'- Adding variables in ~/.bashrs file. We should have "Access Key" - Security Credentials.
ubuntu@ip-x-x-x-x:~$ vi ~/.bashrc
.
.
.
export EC2_KEYPAIR=***
export EC2_URL=https://ec2.ap-southeast-1.amazonaws.com
export EC2_PRIVATE_KEY=$HOME/.ec2/pk-***.pem
export EC2_CERT=$HOME/.ec2/cert-***.pem
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/jre
ubuntu@ip-x-x-x-x:~$ source ~/.bashrc- If everything's all right. Time to use it.
ubuntu@ip-x-x-x-x:~$ ec2-describe-regions
REGION eu-central-1 ec2.eu-central-1.amazonaws.com
REGION sa-east-1 ec2.sa-east-1.amazonaws.com
REGION ap-northeast-1 ec2.ap-northeast-1.amazonaws.com
REGION eu-west-1 ec2.eu-west-1.amazonaws.com
REGION us-east-1 ec2.us-east-1.amazonaws.com
REGION us-west-1 ec2.us-west-1.amazonaws.com
REGION us-west-2 ec2.us-west-2.amazonaws.com
REGION ap-southeast-2 ec2.ap-southeast-2.amazonaws.com
REGION ap-southeast-1 ec2.ap-southeast-1.amazonaws.comubuntu@ip-x-x-x-x:~$ ec2-describe-availability-zones
AVAILABILITYZONE ap-southeast-1a available ap-southeast-1
AVAILABILITYZONE ap-southeast-1b available ap-southeast-1

Categories: DBA Blogs

Packt - The $5 eBook Bonanza is here!

Tue, 2014-12-23 00:38

Spread out news for people who are interested in reading IT books. The $5 eBook Bonanza is here! You will be able to get any Packt eBook or Video for just $5 until January 6th 2015.

Categories: DBA Blogs

How Linux Works, 2nd Edition What Every Superuser Should Know by Brian Ward; No Starch Press

Sun, 2014-11-30 08:23

Everyone knows about Linux. It's a popular operating system that is the software on a computer that enables applications and the computer operator to access the devices on the computer to perform desired functions.
You can read more on link what I pointed to it. For me, Linux is a great operating system that I can use it as Desktop and Server. I have used it over ten years. It's very interesting operation system. I have used/worked it with many Open Source Software such as Apache HTTP, Bind, Sendmail, Postfix, Cyrus Imap, Samba and etc. It's operating system that I can play with programming languages as C, PHP, JAVA, Python, Perl and etc. I don't wanna say "too much".
Today, I have a chance to pick up some... a book that was written about Linux - How Linux Works, 2nd Edition What Every Superuser Should Know by Brian Ward. It's a cool book that you can learn about Linux as Starter and Linux Administrator. You could learn some things you have never used, but find in this book. It's fun to learn. However, A book, it's not support every skills in Linux. You will learn

How Linux boots, from boot loaders to init implementations (systemd, Upstart, and System V)
How the kernel manages devices, device drivers, and processes
How networking, interfaces, firewalls, and servers work
How development tools work and relate to shared libraries
How to write effective shell scripts

It might not be something too much for learning as you are expecting. However, It 's a good book that you can enjoy to read a book about Linux. There's easy to read and understanding in a book. It's for some people who are starting with Linux and Linux Administrators who are enjoying to learn and want to get something new that can use in their fields.

Categories: DBA Blogs

Oracle Database 12C Certified Professional SQL Foundations by Steve Ries; Infinite Skills

Wed, 2014-11-19 01:36

How to become Become an Oracle Certified Associate? That's a good question for some people who want to start working on Oracle Database and get first Oracle Certification for their work life. Step 1 - Pass one SQL Exam: Oracle Database 12c: SQL Fundamentals 1Z0-061 or Oracle Database 11g: SQL Fundamentals I 1Z0-051 or Oracle Database SQL Expert 1Z0-047. Step 2 - Pass Exam Oracle Database 12c: Installation and Administration 1Z0-062.
With Step 1 - Oracle Database 12c: SQL Fundamentals 1Z0-061 exam that you should have skills SQL SELECT statement, subqueries, data manipulation, and data definition language. I was not encouraging to learn SQL SELECT statement, subqueries, data manipulation, data definition language and things like that, but in the real-world for working with Oracle Database you have to know them. You can read about them in Oracle Documents, Oracle Learning Library, Oracle University and the Internet. If you are looking for some video training that instruct you for Oracle Database SQL Foundations such as sql-select, dml, ddl and etc. I mention Oracle Database - 12C Certified Professional SQL Foundations by Steve Ries. I watched it on O'reilly,that is very fast for streaming and downloading. The "Oracle Database - 12C Certified Professional SQL Foundations" video training, that helps learning Oracle Foundations looks easier (watch and do by own) and instructor spoke each topic very clear and easy for listening.

FYI, You can watch some free video.

Video Training: Oracle Database - 12C Certified Professional SQL Foundation
Instructor: Steve Ries

Categories: DBA Blogs

Think Stats, 2nd Edition Exploratory Data Analysis By Allen B. Downey; O'Reilly Media

Mon, 2014-11-17 08:15

Lots of Python with data analysis books. This might be a good one that is able to help readers perform statistical analysis with programs written in Python. Think Stats, 2nd Edition Exploratory Data Analysis by Allen B. Downey(@allendowney).
This second edition of Think Stats includes the chapters from the first edition, many of them substantially revised, and new chapters on regression, time series analysis, survival analysis, and analytic methods. Additional, It uses uses pandas, SciPy, or StatsModels in Python. Author developed this book using Anaconda from Continuum Analytics. Readers should use it, that will easy from them. Anyway, I tested on Ubuntu and installed pandas, NumPy, SciPy, StatsModels, and matplotlib packages. This book has 14 chapters relate with processes that author works with a dataset. It's for intermediate reader. So, Readers should know how to program (In a book uses Python), and skill in mathematical + statistical.
Each chapter includes exercises that readers can practice and get more understood. Free Sampler

Develop an understanding of probability and statistics by writing and testing code.
Run experiments to test statistical behavior, such as generating samples from several distributions.
Use simulations to understand concepts that are hard to grasp mathematically.
Import data from most sources with Python, rather than rely on data that’s cleaned and formatted for statistics tools.
Use statistical inference to answer questions about real-world data.

surachart@surachart:~/ThinkStats2/code$ pwd
/home/surachart/ThinkStats2/code
surachart@surachart:~/ThinkStats2/code$ ipython notebook --ip=0.0.0.0 --pylab=inline &
[1] 11324
surachart@surachart:~/ThinkStats2/code$ 2014-11-17 19:39:43.201 [NotebookApp] Using existing profile dir: u'/home/surachart/.config/ipython/profile_default'
2014-11-17 19:39:43.210 [NotebookApp] Using system MathJax
2014-11-17 19:39:43.234 [NotebookApp] Serving notebooks from local directory: /home/surachart/ThinkStats2/code
2014-11-17 19:39:43.235 [NotebookApp] The IPython Notebook is running at: http://0.0.0.0:8888/
2014-11-17 19:39:43.236 [NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
2014-11-17 19:39:43.236 [NotebookApp] WARNING | No web browser found: could not locate runnable browser.
2014-11-17 19:39:56.120 [NotebookApp] Connecting to: tcp://127.0.0.1:38872
2014-11-17 19:39:56.127 [NotebookApp] Kernel started: f24554a8-539f-426e-9010-cb3aa3386613
2014-11-17 19:39:56.506 [NotebookApp] Connecting to: tcp://127.0.0.1:43369
2014-11-17 19:39:56.512 [NotebookApp] Connecting to: tcp://127.0.0.1:33239
2014-11-17 19:39:56.516 [NotebookApp] Connecting to: tcp://127.0.0.1:54395

Book: Think Stats, 2nd Edition Exploratory Data Analysis
Author: Allen B. Downey(@allendowney)

Categories: DBA Blogs

Getting Started with Impala Interactive SQL for Apache Hadoop by John Russell; O'Reilly Media

Sat, 2014-10-25 12:52

Impala is open source and a query engine that runs on Apache Hadoop. With Impala, you can query data, whether stored in HDFS or Apache HBase – including SELECT, JOIN, and aggregate functions – in real time. If you are looking for a book getting start with it - Getting Started with Impala Interactive SQL for Apache Hadoop by John Russell (@max_webster). Assist readers to write, tune, and port SQL queries and other statements for a Big Data environment, using Impala. The SQL examples in this book start from a simple base for easy comprehension, then build toward best practices that demonstrate high performance and scalability. For readers, you can download QuickStart VMs and install. After that, you can use it with examples in a book.
In a book, it doesn't assist readers to install Impala or how to solve the issue from installation or configuration. It has 5 chapters and not much for the number of pages, but enough to guide how to use Impala (Interactive SQL) and has good examples. With chapter 5 - Tutorials and Deep Dives, that it's highlight in a book and the example in a chapter that is very useful.
Free Sampler.

This book assists readers.

Learn how Impala integrates with a wide range of Hadoop components
Attain high performance and scalability for huge data sets on production clusters
Explore common developer tasks, such as porting code to Impala and optimizing performance
Use tutorials for working with billion-row tables, date- and time-based values, and other techniques
Learn how to transition from rigid schemas to a flexible model that evolves as needs change
Take a deep dive into joins and the roles of statistics

[test01:21000] > select "Surachart Opun" Name, NOW() ;
Query: select "Surachart Opun" Name, NOW()
+----------------+-------------------------------+
| name | now() |
+----------------+-------------------------------+
| Surachart Opun | 2014-10-25 23:34:03.217635000 |
+----------------+-------------------------------+
Returned 1 row(s) in 0.14s

Book: Getting Started with Impala Interactive SQL for Apache Hadoop

Author: John Russell (@max_webster)

Categories: DBA Blogs

Learning Spark Lightning-Fast Big Data Analytics by Holden Karau, Andy Konwinski, Patrick Wendell, Matei Zaharia; O'Reilly Media

Sat, 2014-10-18 13:45

Apache Spark started as a research project at UC Berkeley in the AMPLab, which focuses on big data analytics. Spark is an open source cluster computing platform designed to be fast and general-purpose for data analytics - It's both fast to run and write. Spark provides primitives for in-memory cluster computing: your job can load data into memory and query it repeatedly much quicker than with disk-based systems like Hadoop MapReduce. Users can write applications quickly in Java, Scala or Python. In additional, it's easy to run standalone or on EC2 or Mesos. It can read data from HDFS, HBase, Cassandra, and any Hadoop data source.
If you would like a book about Spark - Learning Spark Lightning-Fast Big Data Analytics by Holden Karau, Andy Konwinski, Patrick Wendell, Matei Zaharia. It's a great book for who is interested in Spark development and starting with it. Readers will learn how to express MapReduce jobs with just a few simple lines of Spark code and more...

Quickly dive into Spark capabilities such as collect, count, reduce, and save
Use one programming paradigm instead of mixing and matching tools such as Hive, Hadoop, Mahout, and S4/Storm
Learn how to run interactive, iterative, and incremental analyses
Integrate with Scala to manipulate distributed datasets like local collections
Tackle partitioning issues, data locality, default hash partitioning, user-defined partitioners, and custom serialization
Use other languages by means of pipe() to achieve the equivalent of Hadoop streaming

With Early Release - 7 chapters. Explained Apache Spark overview, downloading and commands that should know, programming with RDDS (+ more advance) as well as working with Key-Value Pairs, etc. Easy to read and Good examples in a book. For people who want to learn Apache Spark or use Spark for Data Analytic. It's a book, that should keep in shelf.

Book: Learning Spark Lightning-Fast Big Data Analytics
Authors: Holden Karau, Andy Konwinski, Patrick Wendell, Matei Zaharia

Categories: DBA Blogs

Using Flume - Flexible, Scalable, and Reliable Data Streaming by Hari Shreedharan; O'Reilly Media

Thu, 2014-10-09 03:37

Hadoop is an open-source software framework for storage and large-scale processing of data-sets on clusters of commodity hardware. How to deliver log to Hadoop HDFS. Apache Flume is open source to integrate with HDFS, HBASE and it's a good choice to implement for log data real-time collection from front end or log data system.
Apache Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data.It uses a simple data model. Source => Channel => Sink
It's a good time to introduce a good book about Flume - Using Flume - Flexible, Scalable, and Reliable Data Streaming by Hari Shreedharan (@harisr1234). It was written with 8 Chapters: giving basic about Apache Hadoop and Apache HBase, idea for Streaming Data Using Apache Flume, about Flume Model (Sources, Channels, Sinks), and some moew for Interceptors, Channel Selectors, Sink Groups, and Sink Processors. Additional, Getting Data into Flume* and Planning, Deploying, and Monitoring Flume.

This book was written about how to use Flume. It's very good to guide about Apache Hadoop and Apache HBase before starting about Flume Data flow model. Readers should know about java code, because they will find java code example in a book and easy to understand. It's a good book for some people who want to deploy Apache Flume and custom components.
Author separated each Chapter for Flume Data flow model. So, Readers can choose each chapter to read for part of Data flow model: reader would like to know about Sink, then read Chapter 5 only until get idea. In addition, Flume has a lot of features, Readers will find example for them in a book. Each chapter has references topic, that readers can use it to find out more and very easy + quick to use in Ebook.
With Illustration in a book that is helpful with readers to see Big Picture using Flume and giving idea to develop it more in each System or Project.
So, Readers will be able to learn about operation and how to configure, deploy, and monitor a Flume cluster, and customize examples to develop Flume plugins and custom components for their specific use-cases.

Learn how Flume provides a steady rate of flow by acting as a buffer between data producers and consumers
Dive into key Flume components, including sources that accept data and sinks that write and deliver it
Write custom plugins to customize the way Flume receives, modifies, formats, and writes data
Explore APIs for sending data to Flume agents from your own applications
Plan and deploy Flume in a scalable and flexible way—and monitor your cluster once it’s running

Book: Using Flume - Flexible, Scalable, and Reliable Data Streaming
Author: Hari Shreedharan

Categories: DBA Blogs

rsyslog: Send logs to Flume

Mon, 2014-10-06 05:12

Good day for learning something new. After read Flume book, that something popped up in my head. Wanted to test "rsyslog" => Flume => HDFS. As we know, forwarding log to other systems. We can set rsyslog:
*.* @YOURSERVERADDRESS:YOURSERVERPORT ## for UDP
*.* @@YOURSERVERADDRESS:YOURSERVERPORT ## for TCPFor rsyslog:
[root@centos01 ~]# grep centos /etc/rsyslog.conf
*.* @centos01:7777Came back to Flume, I used Simple Example for reference and changed a bit. Because I wanted it write to HDFS.
[root@centos01 ~]# grep "^FLUME_AGENT_NAME\=" /etc/default/flume-agent
FLUME_AGENT_NAME=a1
[root@centos01 ~]# cat /etc/flume/conf/flume.conf
# example.conf: A single-node Flume configuration
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
#a1.sources.r1.type = netcat
a1.sources.r1.type = syslogudp
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 7777
# Describe the sink
#a1.sinks.k1.type = logger
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://localhost:8020/user/flume/syslog/%Y/%m/%d/%H/
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.writeFormat = Text
a1.sinks.k1.hdfs.batchSize = 10000
a1.sinks.k1.hdfs.rollSize = 0
a1.sinks.k1.hdfs.rollCount = 10000
a1.sinks.k1.hdfs.filePrefix = syslog
a1.sinks.k1.hdfs.round = true

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
[root@centos01 ~]# /etc/init.d/flume-agent start
Flume NG agent is not running [FAILED]
Starting Flume NG agent daemon (flume-agent): [ OK ]Tested to login by ssh.
[root@centos01 ~]# tail -0f /var/log/flume/flume.log
06 Oct 2014 16:35:40,601 INFO [hdfs-k1-call-runner-0] (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating hdfs://localhost:8020/user/flume/syslog/2014/10/06/16//syslog.1412588139067.tmp
06 Oct 2014 16:36:10,957 INFO [hdfs-k1-roll-timer-0] (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427) - Renaming hdfs://localhost:8020/user/flume/syslog/2014/10/06/16/syslog.1412588139067.tmp to hdfs://localhost:8020/user/flume/syslog/2014/10/06/16/syslog.1412588139067
[root@centos01 ~]# hadoop fs -ls hdfs://localhost:8020/user/flume/syslog/2014/10/06/16/syslog.1412588139067
14/10/06 16:37:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
-rw-r--r-- 1 flume supergroup 299 2014-10-06 16:36 hdfs://localhost:8020/user/flume/syslog/2014/10/06/16/syslog.1412588139067
[root@centos01 ~]#
[root@centos01 ~]#
[root@centos01 ~]# hadoop fs -cat hdfs://localhost:8020/user/flume/syslog/2014/10/06/16/syslog.1412588139067
14/10/06 16:37:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
sshd[20235]: Accepted password for surachart from 192.168.111.16 port 65068 ssh2
sshd[20235]: pam_unix(sshd:session): session opened for user surachart by (uid=0)
su: pam_unix(su-l:session): session opened for user root by surachart(uid=500)
su: pam_unix(su-l:session): session closed for user rootLook good... Anyway, It needs to adapt more...

Categories: DBA Blogs

Packt Publishing - ALL eBooks and Videos are just $10 each or less until the 2nd of October

Tue, 2014-09-30 11:36

Just spread good campaign from Packt Publishing - It's a good news for people who love to learn something new - ALL eBooks and Videos are just $10 or less -- the more you choose to learn, the more you save:

Any 1 or 2 eBooks/Videos -- $10 each
Any 3-5 eBooks/Videos -- $8 each
Any 6 or more eBooks/Videos -- $6 each

Categories: DBA Blogs

I Heart Logs - Event Data, Stream Processing, and Data Integration by Jay Kreps; O'Reilly Media

Sat, 2014-09-27 00:01

As I have worked in server-side a long time as System Administrator. I must spend with logs. To use it for checking and investigation in issue. As some policies in some Companies, they want to keep logs over year or over ten years. So, it is not unusual to find out idea to store, integrate logs and do something.
A book tittle "I Heart Logs - Event Data, Stream Processing, and Data Integration" by Jay Kreps. It's very interesting. I'd like to know what I can learn from it, how logs work in distributed systems and learn from author who works at LinkedIn. A book! Not much for the number of pages. However, it gives much more for data flow idea, how logs work and author still shows readers why logs are worthy of reader's attention. In a book, that has only 4 chapters, but readers will get concept and idea about Data integration (Making all of an organization’s data easily available in all its storage and processing systems), Real-time data processing (Computing derived data streams) and Distributed system design (How practical systems can by simplified with a log-centric design). In addition, I like it. because author wrote from his experience at LinkedIn.

After reviewing: A book refers a lot of information(It's easy on ebook to click links) that's useful. Readers can use them and find out more on the Internet and use. For Data integration, It's focused to Kafka software that is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system. Additional, It gave why the Big Data Lambda Architecture is good for batch system and a stream processing system and point about things a log can do.

So, Readers will be able to learn:

Learn how logs are used for programmatic access in databases and distributed systems
Discover solutions to the huge data integration problem when more data of more varieties meet more systems
Understand why logs are at the heart of real-time stream processing
Learn the role of a log in the internals of online data systems
Explore how Jay Kreps applies these ideas to his own work on data infrastructure systems at LinkedIn

Book - I Heart Logs - Event Data, Stream Processing, and Data Integration
Author: Jay Kreps

Categories: DBA Blogs

Where is my space on Linux filesystem?

Mon, 2014-09-22 06:06

Not Often, I checked about my space after made filesystem on Linux. Today, I have made Ext4 filesystem around 460GB, I found it 437GB only. Some path should be 50GB, but it was available only 47GB.
Thank You @OracleAlchemist and @gokhanatil for good information about it.
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup0-U01LV 50G 52M 47G 1% /u01
Reference - It's for specify the percentage of the filesystem blocks reserved for the super-user. This avoids fragmentation, and allows root-owned daemons, such as syslogd(8), to continue to function correctly after non-privileged processes are prevented from writing to the filesystem. The default percentage is 5%.

After I found out more information. Look like we can set it to zero, but we should not set it to zero for /,/var,/tmp or which path has lots of file creates and deletes.

Reference on RedHat

If you set the reserved block count to zero, it won't affect
performance much except if you run for long periods of time (with lots
of file creates and deletes) while the filesystem is almost full
(i.e., say above 95%), at which point you'll be subject to
fragmentation problems. Ext4's multi-block allocator is much more
fragmentation resistant, because it tries much harder to find
contiguous blocks, so even if you don't enable the other ext4
features, you'll see better results simply mounting an ext3 filesystem
using ext4 before the filesystem gets completely full.
If you are just using the filesystem for long-term archive, where
files aren't changing very often (i.e., a huge mp3 or video store), it
obviously won't matter.
- Ted

Example: Changed reserved-blocks-percentage

[root@mytest01 u01]# df -h /u01
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup0-U01LV 50G 52M 47G 1% /u01
[root@mytest01 u01]# tune2fs -m 1 /dev/mapper/VolGroup0-U01LV
tune2fs 1.43-WIP (20-Jun-2013)
Setting reserved blocks percentage to 1% (131072 blocks)
[root@mytest01 u01]# df -h /u01
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup0-U01LV 50G 52M 49G 1% /u01
[root@mytest01 u01]# tune2fs -m 5 /dev/mapper/VolGroup0-U01LV
tune2fs 1.43-WIP (20-Jun-2013)
Setting reserved blocks percentage to 5% (655360 blocks)
[root@mytest01 u01]# df -h /u01
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup0-U01LV 50G 52M 47G 1% /u01Finally, I knew it was reserved for super-user. Checked more for calculation.
[root@ottuatdb01 ~]# df -m /u01
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup0-U01LV 50269 52 47657 1% /u01
[root@ottuatdb01 ~]# tune2fs -l /dev/mapper/VolGroup0-U01LV |egrep 'Block size|Reserved block count'
Reserved block count: 655360
Block size: 4096
Available = 47657MB
Used = 52M
Reserved Space = (655360 x 4096) / 1024 /1024 = 2560MB
Total = 47657 + 2560 + 52 = 50269

OK.. I felt good after it cleared for me. Somehow, I believe On Hug space, 5% of the filesystem blocks reserved that's too much. We can reduce it.

Other Links:
https://www.redhat.com/archives/ext3-users/2009-January/msg00026.html
http://unix.stackexchange.com/questions/7950/reserved-space-for-root-on-a-filesystem-why
http://linux.die.net/man/8/tune2fs
https://wiki.archlinux.org/index.php/ext4#Remove_reserved_blocks

Categories: DBA Blogs

Extend linux partition on vmware

Mon, 2014-09-22 03:24

It was a quiet day, I worked as System Administrator and installed Oracle Linux on Virtual Machine guest. After installed Operating System, I wanted to extend disk on guest. So, I extended disk on guest. Anyway, I came back in my head what I was supposed to do on Linux then ?

- Create new disk (and Physical Volume) and then add in Volume Group.

http://surachartopun.com/2012/01/just-add-disk-to-volume-group-linux.html

Checked my partition:

[root@mytest01 ~]# fdisk -l /dev/sda
Disk /dev/sda: 697.9 GB, 697932185600 bytes
255 heads, 63 sectors/track, 84852 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00061d87
Device Boot Start End Blocks Id System
/dev/sda1 * 1 131 1048576 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 131 78326 628096000 8e Linux LVM
[root@mytest01 ~]# pvdisplay
--- Physical volume ---
PV Name /dev/sda2
VG Name VolGroup0
PV Size 599.00 GiB / not usable 3.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 153343
Free PE 0
Allocated PE 153343
PV UUID AcujnG-5XVc-TWWl-O4Oe-Nv03-rJtc-b5jUlW

I thought I should be able to extend (resize) /dev/sda2 - Found out on the Internet, get some example.

http://unix.stackexchange.com/questions/42857/how-to-extend-centos-5-partition-on-vmware

- Extend Physical Volume (Chose this idea)
Started to do it: Idea is Deleting/Recreating/run "pvresize".

[root@mytest01 ~]# fdisk /dev/sda
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): p
Disk /dev/sda: 697.9 GB, 697932185600 bytes
255 heads, 63 sectors/track, 84852 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00061d87
Device Boot Start End Blocks Id System
/dev/sda1 * 1 131 1048576 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 131 78326 628096000 8e Linux LVM
Command (m for help): d
Partition number (1-4): 2
Command (m for help): p
Disk /dev/sda: 697.9 GB, 697932185600 bytes
255 heads, 63 sectors/track, 84852 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00061d87
Device Boot Start End Blocks Id System
/dev/sda1 * 1 131 1048576 83 Linux
Partition 1 does not end on cylinder boundary.
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (131-84852, default 131):
Using default value 131
Last cylinder, +cylinders or +size{K,M,G} (131-84852, default 84852):
Using default value 84852
Command (m for help): p
Disk /dev/sda: 697.9 GB, 697932185600 bytes
255 heads, 63 sectors/track, 84852 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00061d87
Device Boot Start End Blocks Id System
/dev/sda1 * 1 131 1048576 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 131 84852 680524090 83 Linux
Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): L
0 Empty 24 NEC DOS 81 Minix / old Lin bf Solaris
1 FAT12 39 Plan 9 82 Linux swap / So c1 DRDOS/sec (FAT-
2 XENIX root 3c PartitionMagic 83 Linux c4 DRDOS/sec (FAT-
3 XENIX usr 40 Venix 80286 84 OS/2 hidden C: c6 DRDOS/sec (FAT-
4 FAT16 <32m 85="" boot="" br="" c7="" extended="" inux="" nbsp="" prep="" yrinx=""> 5 Extended 42 SFS 86 NTFS volume set da Non-FS data
6 FAT16 4d QNX4.x 87 NTFS volume set db CP/M / CTOS / .
7 HPFS/NTFS 4e QNX4.x 2nd part 88 Linux plaintext de Dell Utility
8 AIX 4f QNX4.x 3rd part 8e Linux LVM df BootIt
9 AIX bootable 50 OnTrack DM 93 Amoeba e1 DOS access
a OS/2 Boot Manag 51 OnTrack DM6 Aux 94 Amoeba BBT e3 DOS R/O
b W95 FAT32 52 CP/M 9f BSD/OS e4 SpeedStor
c W95 FAT32 (LBA) 53 OnTrack DM6 Aux a0 IBM Thinkpad hi eb BeOS fs
e W95 FAT16 (LBA) 54 OnTrackDM6 a5 FreeBSD ee GPT
f W95 Ext'd (LBA) 55 EZ-Drive a6 OpenBSD ef EFI (FAT-12/16/
10 OPUS 56 Golden Bow a7 NeXTSTEP f0 Linux/PA-RISC b
11 Hidden FAT12 5c Priam Edisk a8 Darwin UFS f1 SpeedStor
12 Compaq diagnost 61 SpeedStor a9 NetBSD f4 SpeedStor
14 Hidden FAT16 <3 63="" ab="" arwin="" boot="" br="" f2="" hurd="" nbsp="" or="" secondary="" sys="">16 Hidden FAT16 64 Novell Netware af HFS / HFS+ fb VMware VMFS
17 Hidden HPFS/NTF 65 Novell Netware b7 BSDI fs fc VMware VMKCORE
18 AST SmartSleep 70 DiskSecure Mult b8 BSDI swap fd Linux raid auto
1b Hidden W95 FAT3 75 PC/IX bb Boot Wizard hid fe LANstep
1c Hidden W95 FAT3 80 Old Minix be Solaris boot ff BBT
1e Hidden W95 FAT1
Hex code (type L to list codes): 8e
Changed system type of partition 2 to 8e (Linux LVM)
Command (m for help): p
Disk /dev/sda: 697.9 GB, 697932185600 bytes
255 heads, 63 sectors/track, 84852 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00061d87
Device Boot Start End Blocks Id System
/dev/sda1 * 1 131 1048576 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 131 84852 680524090 8e Linux LVM
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks. -- I chose to "Reboot" :-) --[root@mytest01 ~]# pvdisplay
--- Physical volume ---
PV Name /dev/sda2
VG Name VolGroup0
PV Size 599.00 GiB / not usable 3.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 153343
Free PE 0
Allocated PE 153343
PV UUID AcujnG-5XVc-TWWl-O4Oe-Nv03-rJtc-b5jUlW
[root@mytest01 ~]# pvresize /dev/sda2
Physical volume "/dev/sda2" changed
1 physical volume(s) resized / 0 physical volume(s) not resized
[root@mytest01 ~]# pvdisplay
--- Physical volume ---
PV Name /dev/sda2
VG Name VolGroup0
PV Size 599.00 GiB / not usable 2.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 153343
Free PE 0
Allocated PE 153343
PV UUID AcujnG-5XVc-TWWl-O4Oe-Nv03-rJtc-b5jUlW
[root@mytest01 ~]#
[root@mytest01 ~]# reboot
.
.
.
[root@mytest01 ~]# pvdisplay
--- Physical volume ---
PV Name /dev/sda2
VG Name VolGroup0
PV Size 599.00 GiB / not usable 2.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 153343
Free PE 0
Allocated PE 153343
PV UUID AcujnG-5XVc-TWWl-O4Oe-Nv03-rJtc-b5jUlW
[root@mytest01 ~]# pvresize /dev/sda2
Physical volume "/dev/sda2" changed
1 physical volume(s) resized / 0 physical volume(s) not resized
[root@mytest01 ~]# pvdisplay
--- Physical volume ---
PV Name /dev/sda2
VG Name VolGroup0
PV Size 649.00 GiB / not usable 1.31 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 166143
Free PE 12800
Allocated PE 153343
PV UUID AcujnG-5XVc-TWWl-O4Oe-Nv03-rJtc-b5jUlW

Note: This case I had 2 partitions (/dev/sda1, /dev/sda2). So, it was a good idea extending Physical Disk. However, I thought creating physical volume and adding in Volume Group, that might be safer.

Finally, I had VolGroup0 with new size, then extended Logical Volume.

[root@mytest01 ~]# df -h /u02
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup0-U02LV 460G 70M 437G 1% /u02
[root@mytest01 ~]# lvdisplay /dev/mapper/VolGroup0-U02LV
--- Logical volume ---
LV Path /dev/VolGroup0/U02LV
LV Name U02LV
VG Name VolGroup0
LV UUID 8Gdt6C-ZXQe-dPYi-21yj-Fs0i-6uvE-vzrCbc
LV Write Access read/write
LV Creation host, time mytest01.pythian.com, 2014-09-21 16:43:50 -0400
LV Status available
# open 1
LV Size 467.00 GiB
Current LE 119551
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:2

[root@mytest01 ~]#
[root@mytest01 ~]# vgdisplay
--- Volume group ---
VG Name VolGroup0
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 7
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 4
Open LV 3
Max PV 0
Cur PV 1
Act PV 1
VG Size 649.00 GiB
PE Size 4.00 MiB
Total PE 166143
Alloc PE / Size 153343 / 599.00 GiB
Free PE / Size 12800 / 50.00 GiB
VG UUID thGxdJ-pCi2-18S0-mrZc-cCJM-2SH2-JRpfQ5
[root@mytest01 ~]#
[root@mytest01 ~]# -- Should use "e2fsck" in case resize (shrink). This case no need.
[root@mytest01 ~]# e2fsck -f /dev/mapper/VolGroup0-U02LV
e2fsck 1.43-WIP (20-Jun-2013)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/VolGroup0-U02LV: 11/30605312 files (0.0% non-contiguous), 1971528/122420224 blocks
[root@mytest01 ~]#
[root@mytest01 ~]# pvscan
PV /dev/sda2 VG VolGroup0 lvm2 [649.00 GiB / 50.00 GiB free]
Total: 1 [649.00 GiB] / in use: 1 [649.00 GiB] / in no VG: 0 [0 ]
[root@mytest01 ~]#
[root@mytest01 ~]#
[root@mytest01 ~]# lvextend -L +50G /dev/mapper/VolGroup0-U02LV
Extending logical volume U02LV to 517.00 GiB
Logical volume U02LV successfully resized
[root@mytest01 ~]#
[root@mytest01 ~]# resize2fs /dev/mapper/VolGroup0-U02LV
resize2fs 1.43-WIP (20-Jun-2013)
Resizing the filesystem on /dev/mapper/VolGroup0-U02LV to 135527424 (4k) blocks.
The filesystem on /dev/mapper/VolGroup0-U02LV is now 135527424 blocks long.
[root@mytest01 ~]#
[root@mytest01 ~]#
[root@mytest01 ~]# lvdisplay /dev/mapper/VolGroup0-U02LV
--- Logical volume ---
LV Path /dev/VolGroup0/U02LV
LV Name U02LV
VG Name VolGroup0
LV UUID 8Gdt6C-ZXQe-dPYi-21yj-Fs0i-6uvE-vzrCbc
LV Write Access read/write
LV Creation host, time mytest01.pythian.com, 2014-09-21 16:43:50 -0400
LV Status available
# open 0
LV Size 517.00 GiB
Current LE 132351
Segments 2
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:2
[root@mytest01 ~]#

[root@mytest01 ~]# df -h /u02
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup0-U02LV 509G 70M 483G 1% /u02

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/ext4grow.html

Note: resize2fs can use online, If the filesystem is mounted, it can be used to expand the size of the mounted filesystem, assuming the kernel supports on-line resizing. (As of this writing, the Linux 2.6 kernel supports on-line resize for filesystems mounted using ext3 and ext4.).

Look like today, I learned too much about linux partitioning.

Categories: DBA Blogs

Site search

Site navigation

Aggregator

How to create signed url on google cloud storage?

Oracle Database EM 18 XE Available to Remote Clients

How to allow private connectivity across organizations(GCP)?

GCP - How to manage SSH keys on VM Instance?

Install OEM 13c Silent mode

Packt - Time to learn Oracle and Linux

AWS EC2 API tools: Create snapshot & Check Data in snapshot

AWS EC2 API tools: Installation

Packt - The $5 eBook Bonanza is here!

How Linux Works, 2nd Edition What Every Superuser Should Know by Brian Ward; No Starch Press

Oracle Database 12C Certified Professional SQL Foundations by Steve Ries; Infinite Skills

Think Stats, 2nd Edition Exploratory Data Analysis By Allen B. Downey; O'Reilly Media

Getting Started with Impala Interactive SQL for Apache Hadoop by John Russell; O'Reilly Media

Learning Spark Lightning-Fast Big Data Analytics by Holden Karau, Andy Konwinski, Patrick Wendell, Matei Zaharia; O'Reilly Media

Using Flume - Flexible, Scalable, and Reliable Data Streaming by Hari Shreedharan; O'Reilly Media

rsyslog: Send logs to Flume

Packt Publishing - ALL eBooks and Videos are just $10 each or less until the 2nd of October

I Heart Logs - Event Data, Stream Processing, and Data Integration by Jay Kreps; O'Reilly Media

Where is my space on Linux filesystem?

Extend linux partition on vmware

Pages

Site search

User login

Site navigation

Aggregator

You are here

Surachart Opun

Pages