Adam M. Erickson » data

Category Archives: Linux

rsnapshot: Auto Version Controlled Backup for Unix/Linux/Mac/BSD…

  By | , , , , , , ,
Leave a comment

Red Had Enterprise Linux (RHEL) 6. I missed the ease of configuration and all the free tools that people smarter than me have created.

Systems that can take advantage of Rsnapshot

Systems that can take advantage of Rsnapshot

I would like to do a fast post on rsnapshot. I have seen ssh and rsnapshot scheduled in cron to automate backups of OSX to a Linux server. Since we didn’t want the wireless to slow down we only used the physical MAC address of the MAC. What makes rsnapshot so great is that it will wok on so many systems that are out there (Ubuntu, Debian GNU/Linux, Red Hat Linux, Fedora Linux, SuSE Linux, Gentoo Linux, Slackware Linux, FreeBSD, OpenBSD, NetBSD, Solaris, Mac OS X, and even IRIX) .

For now I’m using it for personal automated backups to my external hard drive. There are plenty of other advanced options and examples on the Internet. I just wanted to get out a fast an easy example.

  1. First – find and install rsnapshot. for Red Hat this was
    $ sudo yum install rsnapshot
    (rsynch is a dependancy that should already be installed).
  2. After install if you do not have this file /etc/rsnapshot.conf. Use the command:
    $ sudo cp /etc/rsnapshot.conf.default /etc/rsnapshot.conf
  3. Edit rsnapshot.conf – The defaults I changed from the default configuration file are below. These options allow me to back up everything in /etc/ and /home/. Backups kept will be twice a day, 7 days a week, 4 weeks, 12 months and 5 years (change this as you see fit).  Most important is that switch to make sure that the mount point will not be created and wrote to locally if the disk is not attached.
    1. WHERE TO PLACE BACKUPS
      # All snapshots will be stored under this root directory.
      #
      snapshot_root   /media/myexternal/rsnapshot/
    2. DO NOT CREATE IF DISK IS NOT CREATED
      # If no_create_root is enabled, rsnapshot will not automatically create the
      # snapshot_root directory. This is particularly useful if you are backing
      # up to removable media, such as a FireWire or USB drive.
      #
      no_create_root 1
    3. INTERVALS (make sure this is tabbed – do NOT use spaces)
      #########################################
      #           BACKUP INTERVALS            #
      # Must be unique and in ascending order #
      # i.e. hourly, daily, weekly, etc.      #
      #########################################
      interval        hourly  12
      interval        daily   7
      interval        weekly  4
      interval        monthly 12
      interval        yearly  5
  4. Time to configure cron. Most people will tell you to create your jobs using $ crontab e
    I prefer to use the root crontab using $ sudo vim /etc/crontab shown below:

    0 */12 * * * root /usr/bin/rsnapshot hourly # Every 12 hours
    30 23 * * * toot /usr/bin/rsnapshot daily   # Daily at 11:30PM
    20 2 * * 0 root /usr/bin/rsnapshot weekly   # Sunday at 2:20AM
    10 5 1 * * root /usr/bin/rsnapshot monthly  # First day of the month at 5:10AM
    01 8 1 1 * root /usr/bin/rsnapshot yearly   # January 1st at 8:01AM
  5. Test It – Following these steps you should have the basic setup needed to run rsnapshot on your personal computer to an external hard drive or usb. Just one last thing to do. Make sure that your hard drive is plugged in and  run:
    $ sudo rsnapshot -V hourly
    rsnapshot should give you plenty of verbose information as it creates your first hourly backup inside the location you specified. If there is a issue with the lock file, remove the lock file and try again.

Still stuck?

There are many other helpful documents out there  start with the rsnapshot how to:
http://www.rsnapshot.org/howto/1.2/rsnapshot-HOWTO.en.html#installation

If you want to learn how to do remote backup and use OSX? try this article:
http://blog.philippmetzler.com/?p=138

As I said in the beginning of this article, this was a fastpost and not meant to cove everything about rsnapshot. It took longer to write this article than it did to set up rsnapshot.
Good Luck – Adam M. Erickson

 

[/fusion_builder_column][/fusion_builder_row][/fusion_builder_container]

Linux: Compiz Grid Plugin

  By | , ,
Leave a comment

After working on a few projects in windows, I finally moved my primary workstation back to Linux (RHEL 6.6).

Out of the box everything worked fine, only one thing was missing, the ability to place a window were I needed it. In Windows 7  you can drag a full screen window to the left or right side of the screen and Win-7 will auto-re-size to that side but only half of the screen. Moving the window to the top of the screen makes the window full screen again.

Sadly, this option is not an available default in Linux. You can drag a window to a different desktop, you can double click the header of a window to have it maximize to full screen or back to default size. I wanted some sort of ability that Win-7 had.

SOLUTION – Compiz

Compiz Grid gives you 9 options to place your window using your numeric key pad and unlike Win-7, this will work on multiple monitor setups for each monitor. That means you can have 18 small windows quickly placed in dual monitor mode! Windows only allows the far left side of the left monitor and the far right side of the right monitor.

You cannot move the window with your mouse because that would kill the ability to drag a window to another workspace (don’t worry, Linux has plenty or work-a-rounds but I’m happy with this solution). If you already have compiz installed to do the “cool” cube thing that Windows users ask to see. Then you are already ahead of the game and depending on the version of Gnome you have, this should work flawlessly. I am only going to go into details for how to do this for RHEL 6.6 with Gnome 2.28.2 but with a few pointers from your favorite LUG (Linux User Group), you should be able to do this from the command line interface (CLI) with no problem using zypper, apt-get, yum, yast or even using a GUI software installer.

There is 2 additional packages for  that need to be installed in order for this to work.
You may have to enable 3rd part packages in order to get these.

$sudo yum install :
  1. compiz-fusion.x86_64 0.8.2-8.el6
  2. compiz-fusion-extras.x86_64 0.8.2-6.el6

These are the 64 bit versions “x86” with out the “_64” is for 32 bit systems.

Once packages are installed:Grid for CCSM

  1.  open the Compiz Configuration Settings Manager (CCSM)
  2. Scroll to the bottom
  3. Enable Grid as seen in this picture

Now give any window focus and hit these combination of keys.

The numbers must be from the numeric keypad!

  •  Ctrl+Alt+7 (window goes to top left corner)
  •  Ctrl+Alt+8 (window goes to fills top or 1/3rd center)
  •  Ctrl+Alt+9 (window goes to files top right corner)
  •  Ctrl+Alt+4 (window goes to fills left side)
  • Ctrl+Alt+5 (maximize window)
  • Ctrl+Alt+6 (window goes fills right)
  •  Ctrl+Alt+1 (window goes to bottom left corner)
  • Ctrl+Alt+2 (window goes to fills bottom or 1/3rd center)
  •  Ctrl+Alt+3 (window goes to bottom right corner)

Press the same keys more than once in a row on the same window to get different sizing options. Basically the same directions in a game that you might be playing using the numeric keypad and it does not matter if number locks is on or not.

Thank you and good luck – Adam M. Erickson

Linux: Grep Script to Search Multiple Directories

  By | , ,
Leave a comment

Audience that may find this useful will mostly be new Linux administrators.ascii art for Grep

This is a Command Line Interface (CLI) Shell script that once set up using the directories you want to search.  All you will have to do is enter the word or phrase that you are trying to search and it will search across the directories (if you use *) and files for the key word/phrase.

In the past, I have used it to search dozens of logs in production at the same time for errors.

#!/bin/bash### First add the directories you want to search
locationAlpha=”/LOGS/Websphere-01/General.log”
locationBravo=”/LOGS/Websphere-02/General.log”
locationCharlie=”/LOGS/Websphere-03/General.log”
locationDelta=”/LOGS/Websphere-04/General.log”
locationEcho=”/LOGS/Websphere-05/General.log”
locationFoxtrot=”/LOGS/Websphere-06General.log”
locationGulf=”/LOGS/Websphere-07/General.log”### Next line will prompt for the word or phrase you want to search
read -p “Enter the word you want to find: ” word
 
## returns word you entered
echo “Searching for ‘$word'”
 
##Starts searching and returning hits
grep -iH “$word” $locationAlpha $locationBravo $locationCharlie $locationDelta $locationEcho $locationFoxtrot $locationGulf

–>                                               View file here without comments–> countProd.txt

Initially called countProd.sh because I was using it to count the number of certain errors. If you would like to only count the number of error. Change “grep -iH” to “grep -ic” (c for count, H to return filename for each match, and i to ignore case) . 

That’s it, short and simple.

Expect – For more remote work, automation, & CI

  By | , , ,
Leave a comment

If you ever have a situation were you want to run scripts automatically this will help.
You can even time the events through cron or CI (continual integration) software like Hudson/Jenkins.
Let me introduce you to a new command I was not aware of until this year.

It is called Expect.expect

Expect can be installed like any other package in a Linux environment.

  • $ sudo yum install expect
  • $ sudo apt-get install expect

Here is an example of what created inside Jenkins to run after a build as an shell script.

Instead of #!/usr/bin/bash

Use:hudson

#!/usr/bin/expect
set prompt "$ "
spawn ssh -t user@remote.server
set timeout 30
expect {
timeout {
puts "Connection timed out"
exit 1
}
"yes/no" {
send "yes\r"
exp_continue
}
"assword:" {
send -- "job00ibm\r"
exp_continue
}
"$prompt" {
send "cd ~/test/\r"
}
}
Now, as long as you have Internet, the correct username, remote server, and password. You should of just created a script the will SSH into a remote server and accept the RSA fingerprint. Or you just created a script that tells you the connection timed out.
Don’t alter this part of the script much because you want to be able to re-use and if there is not a yes/no at the prompt it will continue to expect assword:.
You can get change the last command to anything you want. I added that part because you remotes into a server for no reason?

Add this to the script and it will send the command to find anything with a certain name in it to be removed from the current directory.
expect {
"$prompt" {
send "find . -name '*filename*' -type f -print0 |xargs -0 rm -f\r"
}
}
Add this to the script to find files of a certain name and copy to a new location. You should be able to use modified times in the find switch if that helps.
expect {
"$prompt" {
send "find . -name 'coke*' -type f | xargs -n1 -i cp {} /data02/home/ibmcorp/12180-us-mcrtest1/upload/\r"
}
}
Add this to the script to run a script in the current directory.
expect {
"$prompt" {
send ". name.sh\r"
}
Does your script prompt for anything? Expect can enter the reply for you.
}
expect {
"the expected promt" {
send -- "the reply\r"
}
}
Some things need to e escaped like these ampersands. It is not way to find out what all needs to be escaped but you can figure that through trial and error, intuition, or because of how smart you are.
expect {
":" {
send "me\@a-erickson.com\r"
}
}

That’s my real address if you want to reach me.

All done running your script? Don’t forget to exit your connection.
expect {
"$ " {
send "exit\r"
}
}remote

  Have fun work remotely 🙂

Force Jenkins to connect to remote sites.

  By | , , ,
1 Comment

Problem:

  1. You created an awesome script on Linux that builds your application that will automatically upload to another server via sftp or ssh.
  2. You want to go a step further and have Jenkins run the same script.
  3. Every time Jenkins runs the script it fails.
  4. You Curse Jenkins/Hudson and start pricing Atlassian tools instead.

One possible reasons could be because you have already accepted the RSA fingerprint.

The authenticity of host ‘216.64.211.4 (216.64.211.4)’ can’t be established.CLI Root
RSA key fingerprint is 2c:78:0d:08:b7:06:ac:1a:be:5f:9a:4e:37:06:8f:53.
Are you sure you want to continue connecting (yes/no)?

You cannot get through to a remote sight for upload using Jenkins because even with Jenkins using your credentials, Jenkins still needs to accept the RSA Fingerprint to proceed.

Answer:

You need to have sudo rights
There is a user switch in sudo that will allow you to ssh as jenkins.

$ sudo  -u jenkins username@server

Manually accept the RSA fingerprint key.
Now test the script in Jenkins again. It should work unless there are other rights and directory paths that you need to work out.

JenkinsRemember:

Jenkins is a user with his own user directory. My Jenkin’s user directory was /var/lib/jenkins/
If you want to have Jenkins run a script, make sure it is in Jenkins directory.

Don’t name any of your Jenkin Items with spaces.
The Items you create are made into a directory inside Jenkin’s ~/workspaces/item-name

I am not a Linux expert, I am not certified in Linux. I have been using Linux systems for over 10 years now and love coming up with solutions using Open Source Software.

Develope a Web Based CMS Using PHP

  By | , , , , , , , ,
2 Comments

Download this File Here

Abstract
The Content Management System (CMS) is a web based application using a Linux Server,
Apache Web-server, MySQL Database, and PHP Programming Language (LAMP). The
objective of managing users, and information in any given network environment can only be
hindered by the creativity of an information technology professional and not by technology. The
main objective of this thesis is to develop the early development steps of a LAMP software bundleCMS. By creating the
building blocks for developing, and taking into consideration basic methods for creating the core
platform of a CMS for further development. All information gathered, and experience gained will
assist with developing and offering my own personal e-commerce business solutions in the future
and to obtain additional business and practical knowledge in an open source software and ecommerce.
Continue reading →

Enhance VIM

  By | , , , , ,
Leave a comment

While VI has been used for years, in current Linus & iOS systems a lot of people have resorted to using Vim.

There are a lot of tricks to Vim that can make it easier to use. Some may not be active by default and I wanted a chance to introduce some of them to you. One in general that I have found very useful.

set background=dark

Lets start by making sure that you have the packaged “vim-enhanced” installed on your system. Use one of these steps from a terminal prompt to install it.

Mint/Ubuntu (Debian) sudo apt-get install vim-enhanced

Fedora (redhat, centOS) sudo yum -y install vim-enhanced

OpenSuse (Suse) sudo zypper install  vim-enhanced 

If one of these does not work or your operating systems is different, there are several other options for installing packages. Try the GUI option as well.

The best way to modify settings in Vim is to have the .vimrc file (Vim Run Command file) in your home directory.

cp etc/vimrc  ~/.vimrc

Open the file,

vim ~/.vimrc

From here you can fix all sorts of text editor effects in Vim.

Just remove the ” from the comment in order to use it.

The one that made a difference for me was.

set background=dark

This is a GREAT setting if you are working in Vim and have a dark background.

It will change what you see from this:

Vim background not set.

“set background=dark

To this:

Vim

set background=dark

set background=dark

Making it very easy to see commenting, now there are several ways to change the config file and in the last image you can read some of the ones in the file we copied (cp) over.

Besides being able to read things better know I also like it when the cursor is in t he same place from the last time I was in the file. Another on on the top of the list is the ability to scroll the cursor with me mouse.

Play around with the file, you can always delete it and start from the beginning again if you mess it up to bad.[/fusion_builder_column][/fusion_builder_row][/fusion_builder_container]

Open Source Software in Digital Forensics

  By | , , , ,
4 Comments

The purpose of this research paper is to research information on open source digital forensic tools that are assess-able for free, usually online. To review types of digital forensic tools available and what they do. The basic definition of what open source and digital forensics is will be defined, and how Open Source Software (OSS) digital forensic tools can help accomplished data retrieval. The pros and cons of why OSS should be considering as a viable digital forensic tool-set is also covered.

Digital Forensics and Incident Response and Tools

Digital forensics and Incident Response (DFIR) is the method of investigating and analyzing data information for the purpose of presenting, an ordered report that shows a chain of evidence to find out what happened on a computer and who was responsible, to a court of law. SearchSecurity. (September 2004). DFIR is being more commonly used as more and more people use computers in their daily life, from smart phones, game stations, and laptops. DFIR can help convict anyone of any crime that involved a computer, wither it is prostitution, child pornography or a white collar crime like embezzlement.

DFIR Tools are the free and proprietary applications used by DFIR experts to retrieve the results to hand over to the legal system. They allow investigators the ability to examine the contents of the hard drive without making changes to the data held within. Information that is retrieved can come from deleted files, encrypted, or damaged files SearchSecurity. (September 2004).

Open Source Software

Open Source

Some things are priceless.

Open Source Software (OSS) is a set of practices used to collaborate with software source code that has been made freely available through copywriting laws. It is also commonly known as FOSS (Free Open Source Software), although most OSS is free, not all is but for this research paper I will be covering mostly the free version of OSS. Individuals separated can come from diverse cultural, corporate boundaries, language and other characteristics in order to work together to create complex, non-proprietary software. Software is open sourced when it is free to redistribute, the source code is redistributed with it as well as in compiled mode. The open source licensing was created to make the source code of a program readily available to anyone that requests it. By making the source code available for anyone, it helps in developing stable software because the whole community is able to create changes and redistribute their own version of the software. Open source software protects the original author of the software, does not discriminate in anyway on how it can be used, cannot be specific to a product or software, cannot restrict other software and has to be technology neutral (open source, n.d.). There are several variants of the open source licensing contract that can be reviewed at opensource.org (http://www.opensource.org/licenses/category).

Some of the more widely known open source licenses cover the GNU(Graphic environment of sever Linux desktops), Mozilla (Firefox, Thunderbird), MIT, BSD (like Unix), and Eclipse (Eclipse IDE). Because of the lack of dependency on software vendors, open source software allows the software to transform and morph into potentially anything the users and developers need the software to do. It gives users the freedom to use it when they want, how they want and on their own terms.

Why OSS DFIR Tools

Open Source digital forensic tools addresses specific gap in forensic capabilities of proprietary DFIR tools.  The tools range for analyzing memory dumps, disks, network traces, cell phones, and memory images from game consoles. Besides the fact that some of the tools focus on one specific area of digital forensics of the incident response building, make them invaluable to some investigators who find the complete packaging of some DFIR proprietary tools cumbersome and lacking in some areas.

Financially, companies and governments are always looking for ways to cut budget costs that is the same for DFIR investigators trying to find work. An investigator using DFIR tools can offer a lower price to customers than a person that has to forward the expensive cost of proprietary tools to the customer. Even law enforcement that has an easier time justifying expenses to budget reports in other departments like traffic enforcement, and drug trafficking. Because of the high cost involved in proprietary applications, the follow up cost for updates may be neglected, leaving the software antiquated and not viable in future investigations.

Legally procedures for finding digital evidence need to be defended in court as being testable, published under peer review, show the possible error rate, and are marginally accepted in the relevant scientific community. Because proprietary tools are closed source and the companies offering the tools do not what to acknowledge mistakes in their software, it makes a case all by itself for choosing OSS digital forensic tools during investigation. OSS allows the source code to be evaluated, tested and error rates to be traced. OSS tools are also greatly accepted by the DFIR community (Carrier, 2002). As Brian Carrier, (Carrier, 2002) reported, “The digital forensic market should not be approached in the same way that other software markets are. The goal of a digital forensic tool should not be market domination by keeping procedural techniques secret.” While Carrier may be a little bias since he developed most of the code in Sleuth Kit, Autopsy, and mac-robber, his experience in digital forensic just proves the fact that it is important to keep OSS DGIR tools in mind.

Counterproductive to this paper, it is important to note that there usually is a larger learning curve when dealing with OSS DFIR tools since some run from command prompts and from Nix (Linux, Unix, BSD) operating systems. Also since they usually focus on one component of DFIR, it requires several different applications that need to be tied together to build a report. Because some of the tools take a lot of time to college and arrange data for a report it is sometimes better to use in a lab then in the field.

Conclusion

While open source digital forensic  tools are abound, and you can take advantage of all of them while  avoiding paying fees for commercial products, there are several good commercial tools that are available also. Because digital forensics is such a vast field of study, it is important to not rely on just one set of tools and to research and test other methods to discover and fight anti-forensics.

Good luck with any future digital forensic test cases you attempt, please make sure that it is done ethically and legally.

More information on OSS DFIR tools can be found at sites like IEEE, open source references at the National Institute of Technology (NIST), the National Software Reference Library (NSRL) form NIST, and government studies and college studies on OSS alternatives in DFIR. Additionally there is a growing amount of information from personal websites and OSS developers on DFIR that may be useful. I totally agree with Schneier (2010), that we, “would encorage everybody to download and learn the tools not just because they can do forensics but because most of them can also be used for other things such as finding things in memory and hard drives that should not be there which many AV tools cannot do and to help put systems back together again.”

Future Note: I plan to compare and contrast some of the more common tools that I will continue to study.References

References

Carrier B, (2012). Slueth Kit. Retrieved 4/15/2012. From http://www.sleuthkit.org/sleuthkit/index.php

Cmihai. (October 2007). UNIX System Administration: Solaris, AIX, HP-UX, Tru64, BSD. Retrieved 4/27/2012, from http://blog.boreas.ro/2007/10/digital-forensic-tools-imaging.html

DFF (n.d.) Open Source Digital Investigation Framework. Retrieved 4/17/2012, from http://www.digital-forensic.org/

Forristal, J., Shipley G.. January 8, 2001. Vulnerability Assessment Scanners. Network Computing. http://www.nwc.com

LinuxLinks (n.d.) 6 of the Best Free Linux Digital Forensics Tools, retrieved 4/5/201. From http://www.linuxlinks.com/article/20110115103656314/DigitalForensics.html

Nikkel, B. (June 2012) Practical Computer Forensics using Open Source tools, retrieved 4/23/2012, from www.digitalforensics.ch/nikkel08.pdf

Open Source. (n.d.) The Open Source Definition. Open Source Initiative. Retrieved from http://www.opensource.org/docs/osd

Open Source Digital Forensics (n.d.) Tools, retrieved 4/5/2012 

Schneier B.. (December 2010), Open Source Digital Forensics. Retrieved 4/6/2012. From

SearchSecurity. (September 2004). computer forensics (cyberforensics). Retrieved 5/4/2012. From http://searchsecurity.techtarget.com/definition/computer-forensics

[/fusion_builder_column][/fusion_builder_row][/fusion_builder_container]