Connect to remote postgreSQL database using Ruby

Ever wrote a script, meaning you needed to connect to a remote PG database using PORO (plain old ruby object)?

This can be quite tricky as I found out. Here is a how to.

Let’s say you are parsing some page on the interwebs.

require 'nokogiri'
require 'open-uri'
require 'pg'

@doc = Nokogiri::HTML(open("http://www.some-url.com/products"))

attributes = @doc.css("td[class=some class] a @href")

attrArray = attributes.map { |attr| attr.value}


@newDocArray = Array.new
@nameArray = Array.new

attrArray.each_with_index do |link, index|
 newDoc = Nokogiri::HTML(open("http://www.some-url.com/products/#{link}"))
 
 nameDoc = newDoc.css("div h1")
 newDoc = newDoc.xpath("//div[@class='cardpage']")
 
 newDoc.search("form").remove
 newDoc.search("a").remove

 @nameArray << nameDoc.map { |attr| attr.inner_text}[0]
 @newDocArray << newDoc.map { |attr| attr.inner_text}[0]

end

Now you need to prepare you postgres config on the remote machine to allow connections from your current local machine (per default postgres allows only localhost connections, meaning from remote machine to itself).

There is an amazing tutorial on this page – http://www.cyberciti.biz/faq/postgresql-remote-access-or-connection/

I will sum it up:

Edit $ nano /var/lib/pgsql/data/pg_hba.conf It might be in some other different folder. If that is the case, just search for “pg_hba.conf” – find / -name pg_hba.conf

Now search for your own ip and paste it in pg_nba.conf

host all all 192.168.1.0 255.255.255.0 trust

(Replace 192.168.1.0 with your obtained ip. I left DNS mask unchanged.

Now find and edit $ nano /var/lib/pgsql/data/postgresql.conf

Changing this value – tcpip_socket = true . Now comes the important part – if you don’t find any tcpip_socket value, then DON’T add it! Instead search for

# – Connection Settings –

listen_addresses = ‘*’
port = 5432

And change listen_addresses = ‘localhost’  to listen_addresses = ‘*’ .

That’s it! Restart # /etc/init.d/postgresql restart postgreSQL and try connecting $ psql -h 82.102.42.56 -U username -d database where:

-h 82.102.42.56 is the remote server ip where PG db is running
-U username is the username of the PG user
-d database is the database name

Now we’re all ready to rock and roll!

@conn = PGconn.connect("82.102.42.56", 5432, '', '', "my_database", "my_user", "my_topsecret_password")

def prepareInsertProduct
    	@conn.prepare("insert_product", "insert into products (name, description,) values ($1, $2)")
	end

def addProduct(name, description)
    @conn.exec_prepared("insert_product", [name, description])
end

def connect(arrayContent, arrayNames)

	prepareInsertProduct

	length = arrayContent.length

	arrayContent.each_with_index do |contentValue, index|		
		name = arrayNames[index]
		description = contentValue

		addProduct(name, description)

		print "Processing entry - #{index}"
		print "\r"
	end

end

connect(@newDocArray, @nameArray)

 

 

Properly create and restore your development postgreSQL on production

So maybe you are like me, who is parsing a lot of static data into a development database, make an app around this data and then you don’t want to have an empty database on production, but instead you want to have a full copy of your development database.

After spending a few hours trying to make it work, there is one solution, that yielded exactly 0 errors (yay!)

On your local machine (in the shell, wherever you currently are) :

pg_dump --no-owner your_db_develoment > backup.bak

Then put it in some safe place on your remote machine:

scp backup.bak deploy@192.168.111.222:/home/restore/backup.bak

Make a backup of your production database for safety sake (on your production server, of course)

pg_dump --no-owner your_db_production > /home/backups/04-03-2016/backup.bak

If there is no production database yet on the server, proceed with creating it, elsewise you might need (if the app has users) to cut the current sessions, so check these out:

http://stackoverflow.com/questions/12924466/capistrano-with-postgresql-error-database-is-being-accessed-by-other-users

or

http://stackoverflow.com/questions/1237725/copying-postgresql-database-to-another-server?rq=1

Ok, now we have no database on production machine at all. Let’s create new production database:

postgres=# CREATE DATABASE database_production;
CREATE DATABASE

postgres=# CREATE USER production_db_user WITH password 'qwerty';
CREATE ROLE

postgres=# GRANT ALL privileges ON DATABASE database_production TO production_db_user;
GRANT

Make sure to populate the password with the same password, as set up in database.yml for production user.

That’s it! No need to change the owner. Now let’s restore the backup file:

sudo psql -d mydb_production -U production_user < /home/deploy/restore/backup.bak;

Password for user production_user: 
SET
SET
SET
SET
SET
SET
CREATE EXTENSION
ERROR:  must be owner of extension plpgsql
SET
SET
SET
CREATE TABLE
CREATE SEQUENCE
ALTER SEQUENCE
CREATE TABLE
CREATE SEQUENCE
ALTER SEQUENCE
CREATE TABLE
CREATE SEQUENCE
ALTER SEQUENCE
CREATE TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
COPY 14964
 setval 
--------
 114041
(1 row)

COPY 3151
 setval 
--------
   3236
(1 row)

COPY 28011
 setval 
--------
   7081
(1 row)

COPY 8
ALTER TABLE
ALTER TABLE
ALTER TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
ALTER TABLE
ALTER TABLE

As of now I have no issues regarding the one error above.