Archive

Archive for February, 2010

How-To Synchronize Files Using “rsync” Command

February 22, 2010 11 comments

Placing several backups in a safe place and keeping files synchronized either locally or remotely is an urgent necessity for all users whose everyday tasks solely rely on the use of computers. Hard drives get damaged, files get deleted, memory sticks get lost …etc. You have to make synchronized backups of your stuff!

Many backup applications are there. Some are GUI-based, while others run through the command line. Personally, I prefer to use the command line due to its high flexibility. One of the most widely used command for synchronization purposes has been the rsync command.

By the way, one of the reasons behind my migration from Windows to Linux was the lack of the FREE synchronization tools that run under Windows. Anyway, if I start talking about this again I won’t finish, and I already wrote about this in an earlier post. So, let’s start talking about “rsync” command.

rsync

rsync is an open source utility that provides fast incremental file transfer. “rsync” is freely available under the GNU General Public License

Features

Features of rsync are already mentioned in different sites, but this is how I see it:

  • Incremental File Transfer
  • Compression while Transferring
  • Synchronizing Remote Machines
  • Encrypted and Secure
  • Better Alternative to FTP
  • Very Flexible

Uses

If you have a look at the rsync manual you will notice a huge number of arguments, and that might discourage you from using the command. Well, don’t worry! All those arguments are for advanced usages, and as the synchronization process gets more complex you will need to understand more arguments and when to use them; otherwise, you only need to used to little arguments to do you the job. Therefore , you will find yourself running the same command over and over with same arguments all the time.

Here is the Generic Syntax:

rsync [options] Src Dst

Where Src and Dst respectively are:

Src: Source location from where files will be synchronized.
Dst: Destination location to where files will be synchronized.

Src and Dst can hold these location combinations except for Remote to Remote directories:

Src Dst
Local Local
Local Remote
Remote Local
Local: "/directory"
Remote: "username@hostname:"

Note: Remote Directory will be the HOME directory of the username used at the Remote machine by default!

Full Sync

A full synchronization process implies what is in this table will take place:

Command Process Current Directory Sub-Directories
rsync Create Yes Yes
Update
Delete

A very frequently used arguments with “rsync” command:

rsync -avz -u --progress --delete Src Dst -n
-a Archive mode
-v Verbose mode
-z Compress while transferring files. This will make the transferring faster and will save the network bandwidth
-u Update files which are newer in the Src
--progress Show the transfer progress of each file while transferring
--delete Delete files from Dst if they don’t exist in Src
-n Preview mode. Changes won’t take effect till this argument is omitted
Sync, without Sub-Directories

One of the syncing tasks I perform on a daily basis is that I sometimes want to sync a certain directory without syncing the subdirectories. This is how I do it:

rsync -dvz -u --progress --delete Src Dst -n

If you want the same effect that the argument ‘-a’ have, such as preserving links, owner, group …etc. Then, this is another option:

rsync -dlptgoDvz -u --progress --delete Src Dst -n
Arguments Worthy Looking at
--exclude=PATTERN exclude files matching PATTERN
--include=PATTERN don’t exclude files matching PATTERN
--existing skip creating new files on receiver
--ignore-existing skip updating files that exist on receiver

Important Notes

  1. Synchronization process always works from Src to Dst. So, Any change in the Src will be reflected in the Dst.
  2. A probable confusing matter here is the deleted files. Just remember, a deleted file in the Src will be deleted from the Dst, but any file deleted from the Dst will be recreated from the Src while syncing!
  3. As a rule of thumb, always make the necessary modification in the Src then sync it with the Dst.
  4. Be careful with the Trailing Slash. If the Trailing Slash appears, then content of that directory will be synced, otherwise, that directory AND its content will be synced. Try it out and see what happens!

Too Many Arguments!!

If you found any difficulty in using the proper arguments of rsync command due to their numbers, then I suggest you start with GUI rsync Application called Grsync. The good thing about it is that it shows you the arguments that are going to be used to perform the synchronization process!

Categories: Linux/Unix Tags: , ,

How-To Set Up SVN and Trac

February 2, 2010 16 comments

What are SVN and Trac?

SVN: is a free/open-source Version Control System. Subversion (SVN) manages files and directories over time. Files are placed into a central repository and every change every made to the files or directories are remembered.

Trac: is an enhanced wiki and issue tracking system for software development projects. It provides a Graphical front end to SVN where diffs in files can be obtained. It cannot update the SVN repository. It simply provides a Project Management interface, wiki, ticketing system, and SVN front end.

Installation

sudo apt-get install apache2
sudo apt-get install subversion
sudo apt-get install libapache2-svn
sudo apt-get install trac
sudo apt-get install libapache2-mod-fastcgi libapache2-mod-fcgid

Setting Up SVN and Trac

To make things simple and flexible in case you want to make some changes in the future, configurations will be set in individual files, i.e. as virtual sites, then they will be enabled/disabled when needed.

1-Creating SVN Repository:

cd /var
sudo mkdir MyProjects
cd MyProjects
sudo mkdir svn
cd svn
sudo svnadmin create SVN_Project_01
sudo chown -R www-data.www-data /var/MyProjects

2-Setting Up Trac

cd /var/MyProjects/
sudo mkdir trac
cd trac
sudo trac-admin Trac_Project_01 initenv
-->Project Name [My Project]>
-->Database connection string [sqlite:db/trac.db] (choose defaults)
-->Repository type [svn]>
-->Path to repository [/path/to/repos]> /var/MyProjects/svn/SVN_Project_01
sudo chown -R www-data.www-data /var/MyProjects
sudo trac-admin /var/MyProjects/trac/Trac_Project_01 permission add username MILESTONE_ADMIN REPORT_ADMIN ROADMAP_ADMIN TICKET_ADMIN TRAC_ADMIN

Setting Up Apache:

At this stage, we need to secure the web access to the repository so only authorized persons who can modify it, enable the SSL mode so connection is encrypted, and modify apache configurations accordingly:

Securing Web Access

  1. Enable the Authenticated Access by adding a username and password:
    sudo htpasswd -cb /etc/apache2/passwords username password
    
  2. Enabling the SSL Module:
    Refer to this blog: How-To Enable SSL on Apache2 Server

Trac Configurations

Note: You will need to download the Trac package and link to the cgi-bin/trac.cgi and cgi-bin/trac.fcgi in the configuration.
In my case, this the path to these files after extracting:

ls /usr/local/Trac-0.11.6/cgi-bin

Create the following files or you may create one of them:

Track through CGI
cd /etc/apache2/sites-available/
cat > svn.cgi
##SVN

<Location /MyProjects/svn/SVN_Project_01>
	DAV svn
	SVNPath /var/MyProjects/svn/SVN_Project_01

	AuthType Basic
	AuthName "Subversion Repository - SVN_Project_01"
	AuthUserFile /etc/apache2/.htpasswd

#	<LimitExcept GET PROPFIND OPTIONS REPORT>
		Require valid-user
		SSLRequireSSL
#	</LimitExcept>
</Location>


##Trac

ScriptAlias /MyProjects/trac /usr/local/Trac-0.11.6/cgi-bin/trac.cgi
<Location /MyProjects/trac>
	SetEnv TRAC_ENV_PARENT_DIR /var/MyProjects/trac
</Location>

<Location "/MyProjects/trac">
        SSLRequireSSL
        AuthType Basic
        AuthName "Trac Login for Projectname Website"
        AuthUserFile /etc/apache2/.htpasswd
        Require valid-user
</Location>


<Location "/MyProjects/trac/Trac_Project_01/login">
	SSLRequireSSL
	AuthType Basic
	AuthName "Trac Login for Projectname Website"
	AuthUserFile /etc/apache2/.htpasswd
	Require valid-user
</Location>
Track through FCGI
cd /etc/apache2/sites-available/
cat > svn.fcgi
##SVN

<Location /MyProjects/svn/SVN_Project_01>
	DAV svn
	SVNPath /var/MyProjects/svn/SVN_Project_01

	AuthType Basic
	AuthName "Subversion Repository - SVN_Project_01"
	AuthUserFile /etc/apache2/.htpasswd

#	<LimitExcept GET PROPFIND OPTIONS REPORT>
		Require valid-user
		SSLRequireSSL
#	</LimitExcept>
</Location>



##Trac

ScriptAlias /MyProjects/trac /usr/local/Trac-0.11.6/cgi-bin/trac.fcgi
##fastcgi
#FastCgiConfig -initial-env TRAC_ENV=/var/MyProjects/trac/Trac_Project_01
#FastCgiConfig -initial-env TRAC_ENV_PARENT_DIR=/var/MyProjects/trac

##fcgi
DefaultInitEnv TRAC_ENV_PARENT_DIR /var/MyProjects/trac

#<Location "/MyProjects/trac">
	##fastcgi
	#SetEnv TRAC_ENV_PARENT_DIR "/var/MyProjects/trac"
	#SetEnv TRAC_ENV "/var/MyProjects/trac/Trac_Project_01"
	#AddHandler fastcgi-script .fcgi

	#AuthType Basic
        #AuthName "Trac Repository - Trac_Project_01"
        #AuthUserFile /etc/apache2/.htpasswd
	#Require valid-user
	#SSLRequireSSL
#</Location>

<Location "/MyProjects/trac/Trac_Project_01/login">
	SSLRequireSSL
	AuthType Basic
	AuthName "Trac Login for Projectname Website"
	AuthUserFile /etc/apache2/.htpasswd
	Require valid-user
</Location>

Track through FastCGI
cd /etc/apache2/sites-available/
cat > svn.fastcgi
##SVN

<Location /MyProjects/svn/SVN_Project_01>
	DAV svn
	SVNPath /var/MyProjects/svn/SVN_Project_01

	AuthType Basic
	AuthName "Subversion Repository - SVN_Project_01"
	AuthUserFile /etc/apache2/.htpasswd

#	<LimitExcept GET PROPFIND OPTIONS REPORT>
		Require valid-user
		SSLRequireSSL
#	</LimitExcept>
</Location>



##Trac

ScriptAlias /MyProjects/trac /usr/local/Trac-0.11.6/cgi-bin/trac.fcgi
#FastCgiConfig -initial-env TRAC_ENV=/var/MyProjects/trac/Trac_Project_01
FastCgiConfig -initial-env TRAC_ENV_PARENT_DIR=/var/MyProjects/trac

#DefaultInitEnv TRAC_ENV /var/MyProjects/trac/Trac_Project_01

<Location "/MyProjects/trac">
	#SetEnv TRAC_ENV_PARENT_DIR "/var/MyProjects/trac"
	#SetEnv TRAC_ENV "/var/MyProjects/trac/Trac_Project_01"
	AddHandler fastcgi-script .fcgi

	#AuthType Basic
        #AuthName "Subversion Repository - SVN_Project_01"
        #AuthUserFile /etc/apache2/.htpasswd
	#Require valid-user
	#SSLRequireSSL
</Location>

<Location "/MyProjects/trac/Trac_Project_01/login">
	SSLRequireSSL
	AuthType Basic
	AuthName "Trac Login for Projectname Website"
	AuthUserFile /etc/apache2/.htpasswd
	Require valid-user
</Location>
Track through Python
cd /etc/apache2/sites-available/
cat > svn.python
##SVN

<Location /MyProjects/svn/SVN_Project_01>
	DAV svn
	SVNPath /var/MyProjects/svn/SVN_Project_01
	
	AuthType Basic
	AuthName "Subversion Repository - SVN_Project_01"
	AuthUserFile /etc/apache2/.htpasswd
	
#	<LimitExcept GET PROPFIND OPTIONS REPORT>
		Require valid-user
		SSLRequireSSL
#	</LimitExcept>
</Location>


##Trac

<Location /MyProjects/trac>
	SetHandler mod_python
	PythonInterpreter main_interpreter
	PythonHandler trac.web.modpython_frontend
	PythonOption TracEnvParentDir /var/MyProjects/trac
	PythonOption TracUriRoot /MyProjects/trac
	
	#SSLRequireSSL
	#AuthType Basic
	#AuthName "Trac Login for Projectname Website"
	#AuthUserFile /etc/apache2/.htpasswd
	#Require valid-user
</Location>

<Location "/MyProjects/trac/Trac_Project_01/login">
	SSLRequireSSL
	AuthType Basic
	AuthName "Trac Login for Projectname Website"
	AuthUserFile /etc/apache2/.htpasswd
	Require valid-user
</Location>

After that, you will need to enable ONLY one of them:

sudo a2ensite svn.cgi

Or

sudo a2ensite svn.fcgi

Or

sudo a2ensite svn.fastcgi

Or

sudo a2ensite svn.python

Finally, reload apache server:

sudo /etc/init.d/apache2 reload

Try to open these links:
https://localhost/MyProjects/svn/SVN_Project_01
https://localhost/MyProjects/trac

Next

You will need to configure your prefered IDE with the created SVN repository if it supports the synchronization with SVN repositories. In my case as a Java/Java EE Developer, I’ ll refer to the most known open IDEs which are: NetBeans and Eclipse. SVN is just a little part of them to control versioning!

Hence, this would help you get a well managed and organized development environment!

___________________

References

https://help.ubuntu.com/community/UbuntuTracHowto
http://wiki.kartbuilding.net/index.php/Trac_and_SVN
http://trac.edgewall.org/wiki/TracGuide