Metasploit Changes to Git

Metasploit is changing from using their own SVN server to host their repository to GitHub and by this move to Git as their tool for managing the main repository available to the public for getting access to the Framework source code. This also changes the way commits are done, if any none Rapid7 employee or contractor member of the development team wants to contribute code it will have to be thru GitHub pull request feature. This will allow Rapid7 better control over who commits and the quality of the commits making sure that their commercial products Metasploit Community, Pro and Express do not get affected by a contribution that did not go thru a proper test procedure and quality assurance. In addition the shift from SVN to Git will allow greater flexibility to the Rapid7 team to make modification to the the framework on forks and branches on their own systems allowing them to keep the main repository as stable as possible and changes to be pushed in a less risky manner. This is great business move since it will reduce risk and accelerate development of the base foundation of their products, allowing the team to focus more on the technical an engineering aspects of the projects and less on the over heads of managing code on their machines. In terms of management of community commits the pull requests will centralize the process from Redmine and the emails to msfdev mailing list making it easier for them to get contributions for the Framework. I do have to say I will miss the ability to be able to push my own changes and fixes and will have to rely like everybody else to the fork process and GitHub pull request method like everybody else but in the long run this a better solution for the stability of the code, faster innovation and risk reduction allowing Rapid7 to further advanced the Framework that is base of some of their commercial products. 

 Now this does changes my workflow for the code I write for use in Metasploit. I do have a GitHub account that I used as my temporary account for plugins and modules, I will be consolidating this one in to one single project in GitHub and making sure it follows the folder structure as in the framework so I can just have it in my machine under ~/.msf4 that way I can test modify and test modules and plugins without the need of putting them in the framework folder it self and move them in to the forked version if I wish to contribute them to Rapid7 if not they will still be accessible for sharing under my GitHub page. So there are now 2 new ways to use the framework repository depending on your need, If you are only to consume the code in it and do not wish to contribute your code to Rapid7 you just need to have Git on your system and clone the repository. You first start by installing Git

Installing Git

On OS X you only need to install the latest Xcode Tools from the AppStore On CentOS 6 and latest Fedora Systems you would run as root

yum update
yum install git

On Ubuntu and Debian systems you would run as root

apt-get update
apt-get install git-core

 Cloning the Repository

I’m a person who likes having several copies of the Framework to work in, I tend to keep in my home folder on my boxes a folder call dev where I keep all the project repositories I use. So I recommend you start by creating the folder to host the project and its copies if you later decide to fork and work on coding inside the Framework.

mkdir -p ~/dev
cd ~/dev

 Once the folder is create you only need to clone the Git repository that is on GitHub

git clone git://github.com/rapid7/metasploit-framework.git

Now you should be able to use and work from inside the metasploit-framework folder created there. To keep you copy updated you only need to run from the folder 

git pull

This will fetch the latest changes and merge them together. 

Forking the Project

If you are planning on contributing to the project then it is recommended that you fork the project so you can issue pull request from GitHub. To be able to do this you must first have a GitHub account at http://github.com once you have an account there you can navigate to https://github.com/rapid7/metasploit-framework and from there click on the fork button on the page to create a fork under your account

Once it has been forked you can now clone it your development system by running the following command in your dev folder, In the example bellow you will see I amed it msf-fork so I can quickly see this is my forked copy 

git clone git@github.com:<your username>/metasploit-framework.git msf-fork

Next I want to be able to update from the main Framework repository so I will set the one managed by Rapid7 as upstream and test updating

git remote add upstream git://github.com/rapid7/metasploit-framework.git
git fetch upstream

 Once you have done several commits to the clone in your machine you can push those to the fork on your GitHub account by doing running the following command

git push origin master

Any time the framework is updated and you want to merge those changes in to your forked copy you would run 

git fetch upstream
git merge upstream/master

 To send a pull request to Rapid7 so as to contribute your code or changes you would follow the GitHub guide that would do a better job than me explaining it in this blog post

http://help.github.com/send-pull-requests/

I also recommend the use of branches so you can have several branches of the fork you have so you can later merge those in to your local clone of the fork and push for latter submitting for pull requests a great guide for this is on Online Git Book http://book.git-scm.com/3_basic_branching_and_merging.html

I hope you find this post useful and look forward to people contributing to the Framework.

Finding Domain Names from Discovery

Many time when performing a penetration test against a Windows environment one of the most critical pieces of information to get is the domain name. As part of my discovery process one of the first things I do inside Metasploit after running a Nmap scan or a TCP Scan thru a pivot is to run the auxiliary module smb_version to get a more accurate finger print of this hosts. As part of the information gathered it gets:

• OS Type and Name
• Host Name
• Domain/Workgroup

Now one of the things I noticed was that Domain and Workgroups where all tagged as domain. The information is saved with the service in the info field of the service, so what I did was parse that string for the machine name and workgroup name and placed those in variables so I can work with those for each service, check that it is not the same as the computer name or the 2 most common workgroup names out there that are MSHOME and WORKGROUP and show the output.

  1: <ruby>
  2: framework.db.workspace.services.find_all_by_state("open").each do |s|
  3:   if s.port == 445 and s.info =~ /windows/i
  4:     name = s.info.scan(/name:(\S*)\)/)[0].join
  5:     domain = s.info.scan(/domain:(\S*)\)/)[0].join
  6:     if ( domain !~ /WORKGROUP|MSHOME/) and (domain != name)
  7:       print_good("Name: #{name} Address: #{s.host.address} Domain: #{domain}")
  8:       print_good("Info: #{s.info}\n")
  9:     end
 10:   end
 11: end
 12: </ruby>

To do all of this I used the the resource file you see above. I first looked at the services saved in my current workspace and looked for all of those with a state of “open” and iterated thru each as seen in line 2 of the code. For each found service with the state of open I checked for the open port of 445 the SMB port and where the smb_version module saves it’s information and checked with a regular expression that the work windows was part of the information. For each one of the services that matched that criteria I extracted the name and domain using regular expressions as seen in lines 4 and 5 and saved those so I could compare then. On line 6 of the code I check that the workgroup does not matches the 2 common ones I mentioned and that the Domain Name is not the same as the computer as it happens on some versions of Windows XP, specially the home edition and print the information.

This is a quick and dirty way to enumerate possible domain names and the hosts in it to perform more specific windows attacks. Hope you found this little excerpt of code useful and servers as an example of how one is able to play with the information inside the database of Metasploit. 

Extending Metasploit Resource Files

Today I saw an email on the Metasploit mailing list asking how one could scan hosts detected by other auxiliary modules and not scanned by Nmap so as to enumerate all services that might have been missed on this hosts. This gave me the excuse to play a bit with ruby inside resources files, something I have not done much of and came up with this little dab of ruby code that could be placed inside a resource file and used to scan al host. The script will actually check the notes for hosts that have any note with a type that starts with host.nmap and add the Host ID to an array, I use the uniq! method to remove any duplicates then go thru the entire list of host in the database and check if there ID is on the list of hosts already scanned by Nmap, if they are not then I run an Nmap scan against them. Do not know if you guys might find this useful but I will definitely keep it inside a resource file for those cases when I need to make sure I’m not missing anything in an internal assessment. Here is the code for it

My Basic Setup on OSX Lion

On this blog post I will covered what I learned to setup my basic environment that use from terminal to do my development and research on my Macbook, in addition to the steps you will see here I also. In addition to this I download and install VMware Fusion, Nessus, Netbeans, Colloquy, Chrome, Firefox, Adium, Skype, WebSecurify just to mention a few. But the major pain point has always been not the pre-package apps but the terminal environment so here I will share my basic setup of that environment.

The first thing is to do a software update and make sure you have any new patches that there may be from Apple for the OS, then go in to the App Store and download and install the latest version of Xcode for Lion once this has been done we can install the latest version of Homebrew, an alternative package manager for OSX similar to MacPorts.

---

Install Homebrew

ruby -e "$(curl -fsSL https://raw.github.com/gist/323731)"

Before installing any packages I recommend you modify that shell profile to add environment variables for the new compiler in version 4 of Xcode so we do not get any errors compiling Ruby Gems, Python Eggs and software found in the Homebrew Formulas.  To be able to modify the system wide profile we must first make it writable and then open it for editing:

  sudo chmod +w /etc/profile
  sudo vim /etc/profile

Once open append the lines bellow to set the proper variables. These variables will enable color for terminal and the files shown just like most default settings on Linux, set compiler flags and the ever so annoying SVN Keywords command:

---

Install Base Packages

On my system I started by adding some base tools for formulas that did not needed any modifications on my part, I installed Nmap, THC Hydra, MacVim, Tmux, Hping and PostgreSQL:

brew install nmap hydra macvim tmux wget hping postgresql readline

The PostgreSQL package after install will need a bit of more work since a system database must be created, set up the engine to start at user logon and create a base user and database for Metasploit that is why I use PostgreSQL for. Lets start by initializing the database:

  initdb /usr/local/var/postgres

Configure Database for Startup at Logon

  mkdir -p ~/Library/LaunchAgents
  cp /usr/local/Cellar/postgresql/9.0.4/org.postgresql.postgres.plist ~/Library/LaunchAgents/

Start PostgreSQL Server and Create User and Database  for Metasploit

  1: # Start databse sever
  2: pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log start
  3: # Create user named msf, provide a good password for it and answer no to all other questions
  4: createuser msf -P -h localhost
  5: # Create database for use with metasploit called msf and make the user msf the owner
  6: createdb -O msf msf -h localhost

Once PostgreSQL is up and running I can work on installing other packages that need a bit more work.

The first thing would be to install Apples Package of Java, to do this just issue the command Java in the console and follow the instructions that will appear on the screen.

I start by editing John the Ripper formula to download and Install the latest community patched version

  brew edit john

Make sure it looks like this:

Make sure you modified the url variable and the md5 variable, and remove the patch section like shown above. Once modified save and install using the command below:

  1: brew install john

Next we install Libdnet since we will need this to be able to install scapy later on, we need to edit the formula and add the additional call to install the python libraries for us:

The line that needs to be added is line afte the make install command where you move in to the python directory and install the python libraries. Installation is the same as the other formulas above using the brew install command.

---

Configuring RVM and Installing Ruby Versions

As many of you may already know I contribute a lot of time, code and resources for free to the Metasploit project and consider my self a Metasploit Junkie when it comes to coding for it, so for me having Ruby install just right is important as well as to have several versions of ruby to test against, for this I use RVM the Ruby Version Manager.

To install RVM system wide you issue the following command:

bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)

This will install RVM on your system, use sudo and specify ruby befole launchin program like msfconsole to ensure you are using the right ruby when root privilages are needed. Sadly Ruby on OSX is compiled to use LibEdit and not GNU Readline, Metasploit takes advantage of many of the features of GNU Readline for that reason we must do some configuration ahead of time to make sure the versions of ruby we configure are compatible and will not cause problems down the road.

Once it is installed we must override the compilation parameters of RVM to make sure it always compiles against the GNU version of Readline we just installed we open the RVM db file with the following command:

vim ~/.rvm/user/db

and we append to the end the following line:

ruby_configure_flags=--with-readline-dir=/usr/local/Cellar/readline/6.2.1/

Now that this is done, we can start installing the most used versions of Ruby, you will notice that I installed a specific patch set for Ruby 1.9.1 since versions above that one have a know problem with Gem and you will see I set the system default to 1.9.2.

rvm install 1.8.7

rvm install 1.9.1-p378

rvm install 1.9.2

rvm --default 1.9.2

Once this is done we can start installing the necessary gems, but first I like to set up a gem resource file so documentation and ri are not generated so as to speed installation and update of the gems, if you use from terminal the docs and ri you can skip this step:

echo "gem: --no-ri --no-rdoc" >> ~/.gemrc

Now we can install the gems we need on the RVM versions of Ruby and on the local version that comes with Lion:

rvm gem install wirble sqlite3 pg activerecord wirb hirb awesome_print interactive_editor
gem install wirble sqlite3 pg activerecord wirb hirb awesome_print interactive_editor

Now that we have Ruby we can download and install Metasploit

---

Installing Metasploit

In my case since I do mostly development with Metasploit and do not use it on a daily basis now that I changed jobs, my set up will not be a system wide one, since many times I will branch and make copies of the framework to experiment and modify. For this I create in my home directory a folder called Development and place Metasploit there:

mkdir ~/Development

I change to that directory and download my latest copy of the framework.

svn co https://www.metasploit.com/svn/framework3/trunk/ msf

Once this finishes you can move inside the folder a launch msfconsole by running the command

cd ~/Development/msf

./msfconsole

An you should be greeted by the banner and the prompt. Once this is done we must install the pcaprub gem that comes with the framework on all the version of Ruby that we have installed with RVM.

cd external/pcapryb

rvm 1.8.7

ruby extconf.rb && make && sudo make install

rvm 1.9.1-p378

ruby extconf.rb && make && sudo make install

rvm 1.9.2

ruby extconf.rb && make && sudo make install

rvm system

ruby extconf.rb && make && sudo make install

Once this is done we have to configure MSF to always logon to the PostgreSQL Database we configured. For this we create a yaml configuration file in our msf pofile.

vim ~/.msf4/database.yml

We then enter the following yaml parameters:

Next time we launch msfconsole we should see the creation of the tables and when a db_status is issued we should be connected to the database we specified.

---

Installing Scapy and DNSRecon

Another of the tools I tend to use and code against is Scapy, I also wrote a small enumeration tool in python call DNSRecon that I maintain and plan to expand shortly . To install lit first we must prep our Python environment and make sure we have all the necessary libraries, lets start with those we can automate, I use pip for python since it will not install incomplete downloads and will allow me to uninstall and install a later version of a library.

sudo easy_install pip 

sudo pip install Mercurial

sudo pip install pycrypto

sudo pip install pybonjour

sudo pip install dnspython

sudo pip install netaddr

One library that we can not download from the command like and needs to be download by hand is pylibpcap, once we have downloaded the tar.gz file we can decompress it and install it:

tar xvzf pylibpcap-0.6.2.tar.gz

cd pylibpcap-0.6.2

sudo python setup.py install

After we have the libraries install I can download the latest repos of the projects:

cd ~/Development

# Scapy Community Repo

hg clone http://hg.secdev.org/scapy-com

# Scapy Main Branch

hg clone http://hg.secdev.org/scapy

# DNSRecon

git clone https://github.com/darkoperator/dnsrecon.git

To install the latest version of Scapy we just go in to the Scapy folder and install it

cd scapy

sudo python setup.py install

The reason I do not install the community edition since it has the latest contributions is that anybody can contribute to it and I tend to check each update before even thinking of running scapy out of it.

---

Configuring System Resource Files

Now that I have this packages set, I setup my tmux.conf and vimrc files since I use tmux to manage my terminal screens and positions and vim for editing files quickly in the termenial. I start by copying my tmux conf file to my home folder and to the root user home folder, since some times I do stuff as root. the filename is .tmux.conf and the contents is as follows:

I invite you to read it and modify as to your own preferences.

Next I modify and set my .vimrc file to my liking for highlighting text instances, syntax highlighting, line numbers and some specifics for NASL and Ruby.  Firs I download and install my backup of my .vim folder. You will see that it is pretty simple sine I do not do any major coding on vim and tend to use an IDE for it. Vim Resource File:

I do tend to have a custom IRB Resource file so as to make life easier and text easier to read when working on irb, here is the simple version of it:

It does looks like much but in fact this is my basic setup for Lion. Now to make a backup  

Took me 2 days to get all of this sorted and tested. Hope some of you find it useful.

Update on 2011-07-25 17:48 by Carlos Perez

After testing with otool found that readline was not linking properly, also had several problems with system wide RVM so I changed the guide to single user.

Update on 2011-07-30 16:39 by Carlos Perez

Made some minor fixes and removed the Wireshark Brew install, I do recomend the package as a better option, one less application to link to the /Applications folder and less dependancies to worry abaout, since a lot of custom modding of formulas can be counter productive in the long run, at least for my setup. 

Automating Post Modules and Meterpreter Across Sessions

I wrote a couple of weeks ago a Metasploit plugin for automating running Metasploit post modules across several sessions while writing and testing the post exploitation mixin for Linux since there are so many distros I had a large number of sessions including some to Solaris and Windows host and testing one by one of the sessions was a bit of a pain. I tried using sessions –s command like I used to for Meterpreter scripts but I would had to modify and fix the the sessions command for it and since I had not found any bug reports or even people asking for the feature I decided that I would do it as a plugin to practice. Once the plugin was done I shared it with several people who found it useful and gave great feedback. Yesterday on twitter a person asked for the same thing but as modules and I do have to say it made sense so I turned the plugin and improved on the code and came out with 4 modules of auxiliary type and placed those modules in my Git Hub Repo under the BSD License at https://github.com/darkoperator/Meterpreter-Scripts/tree/master/post the modules are:

• multi_meter_command – Module for running against all sessions or a list of specific sessions a given Meterpreter console command.
• multi_meter_resource – Module for running against all session or specified sessions in a given resource file the specific Meterpreter console command. Each entry in the resource file would be <sessions><space><command>
• multi_post - Module for running against all sessions or a list of specified sessions a given post exploitation module.
• multi_post_resource - Module for running against all session or specified sessions in a given resource file the specific post module and options. Each entry in the resource file would be <sessions><space><module><option list>

I recommend that you put the modules in your home directory .msf3 folder so as to not mess with your current Metasploit install and you are able to use it with multiple copies of Metasploit. On a Linux or OSX system you just need to create the folder where the module will reside and that they conform to the structure found in Metasploit. In my case even do the modules are auxiliary one I placed mine under post since they serve for the purpose of post exploitation.

You can download the modules I have written using Git or by hand. I recommend you use Git since it will allow you to keep them updated and add any additional modules I work on in a very easy manner. First navigate to the folder when you want to have the Git Repository to be and run

git clone git://github.com/darkoperator/Meterpreter-Scripts.git

Once you have the repo you link the post folder with your own .msf3 folder and use git pull command to keep it updated.

mkdir -p ~/.msf3/modules

Once we made sure we have the proper folder to keep our own copy of the modules we can link this folder to the Git Repository

ln -s ~/<git repo>/post ~/.msf3/modules/post

The modules are now ready to use.

For the examples I have several sessions on my box so as to show you how the modules behave:

msf exploit(handler) > sessions 
Active sessions
===============
  Id  Type                   Information                                      Connection
  --  ----                   -----------                                      ----------
  1   meterpreter x86/win32  WIN-YR4V852V71Y\Administrator @ WIN-YR4V852V71Y  192.168.1.100:4444 -> 192.168.1.114:49160
  2   meterpreter x86/win32  CARLOS-192FCD91\Administrator @ CARLOS-192FCD91  192.168.1.100:4444 -> 192.168.1.119:1122
  3   shell linux                                                             192.168.1.100:4448 -> 192.168.1.123:46113
  4   shell linux                                                             192.168.1.100:4448 -> 192.168.1.115:52949
  5   shell linux                                                             192.168.1.100:4448 -> 192.168.1.118:34272

Lets start by loading the multi_post module and looking at the options we have:

msf exploit(handler) > use post/multi/manage/multi_post
msf auxiliary(multi_post) > show options 
Module options (auxiliary/multi/manage/multi_post):
   Name      Current Setting  Required  Description
   ----      ---------------  --------  -----------
   MODULE                     yes       Post Module to run
   OPTIONS                    no        Commans Separated list of Options for post module
   SESSIONS                   yes       Specify either ALL for all sessions or a comman separated list of sessions.

Lets use the checkvm module for Meterpreter and make it run against all current sessions we could have also given it a list of session in a comma separated list

msf auxiliary(multi_post) > set SESSIONS all
SESSIONS => all
msf auxiliary(multi_post) > set MODULE windows/gather/checkvm
MODULE => windows/gather/checkvm

To execute we use the run command:

msf auxiliary(multi_post) > run
[*] Loading windows/gather/checkvm
[*] Running Against 1
[*] Checking if WIN-YR4V852V71Y is a Virtual Machine .....
[*] This is a VMware Virtual Machine
[*] Running Against 2
[*] Checking if CARLOS-192FCD91 is a Virtual Machine .....
[*] This is a VMware Virtual Machine
[*] Auxiliary module execution completed

Now lets say we want to execute only against sessions 1 and 5, look at the output you will see the module identified the session 5 as not compatible and skipped execution against it:

msf auxiliary(multi_post) > set SESSIONS 1,5
SESSIONS => 1,5
msf auxiliary(multi_post) > run
[*] Loading windows/gather/checkvm
[*] Running Against 1
[*] Checking if WIN-YR4V852V71Y is a Virtual Machine .....
[*] This is a VMware Virtual Machine
[-] Session 5 is not compatible with windows/gather/checkvm
[*] Auxiliary module execution completed

Now lets take a look at the multi_post_resource module, it performs the same tasks as multi_post but using a resource file, an example one is provided in the Git Repository

Sample Resource file:

all linux/gather/checkvm
1,2 windows/gather/enum_logged_on_users CURRENT=true,RECENT=false

Lets load the module and look at the options:

msf auxiliary(multi_post) > use post/multi/manage/multi_post_resource 
msf auxiliary(multi_post_resource) > show options 
Module options (auxiliary/multi/manage/multi_post_resource):
   Name      Current Setting  Required  Description
   ----      ---------------  --------  -----------
   RESOURCE                   yes       Resource file with space separate values <session> <module> <options>, per line.

Lets set the resource file and execute:

msf auxiliary(multi_post_resource) > set RESOURCE /tmp/postrc.rc
RESOURCE => /tmp/postrc.rc
msf auxiliary(multi_post_resource) > run
[*] Loading linux/gather/checkvm
[*] Running Against 3
[*] Gathering System info ....
[+] This appears to be a VMware Virtual Machine
[*] Running Against 4
[*] Gathering System info ....
[+] This appears to be a VMware Virtual Machine
[*] Running Against 5
[*] Gathering System info ....
[+] This appears to be a VMware Virtual Machine
[*] Loading windows/gather/enum_logged_on_users
[*] Running Against 1
[*] 	Setting Option CURRENT to true
[*] 	Setting Option RECENT to false
[*] Running against session 1
Current Logged Users
====================
 SID                                            User
 ---                                            ----
 S-1-5-21-2757829322-3393694802-1237719419-500  WIN-YR4V852V71Y\Administrator
[*] Running Against 2
[*] 	Setting Option CURRENT to true
[*] 	Setting Option RECENT to false
[*] Running against session 2
Current Logged Users
====================
 SID                                          User
 ---                                          ----
 S-1-5-21-1292428093-706699826-725345543-500  CARLOS-192FCD91\Administrator
[*] Auxiliary module execution completed

Lets take a look at the multi_meter_command module and its options:

msf auxiliary(multi_post_resource) > use post/multi/manage/multi_meter_command 
msf auxiliary(multi_meter_command) > show options 
Module options (auxiliary/multi/manage/multi_meter_command):
   Name      Current Setting  Required  Description
   ----      ---------------  --------  -----------
   COMMAND                    yes       Meterpreter Console command.
   SESSIONS                   yes       Specify either ALL for all sessions or a comman separated list of sessions.

Now lets set the sysinfo command to run against all sessions and execute it you will see that it will identify the none Meterpreter sessions and skip those:

msf auxiliary(multi_meter_command) > set SESSIONS all 
SESSIONS => all
msf auxiliary(multi_meter_command) > set COMMAND sysinfo
COMMAND => sysinfo
msf auxiliary(multi_meter_command) > run
[+] Running command sysinfo against sessions 1
System Language : en_US
OS              : Windows 2008 (Build 6001, Service Pack 1).
Computer        : WIN-YR4V852V71Y
Architecture    : x86
Meterpreter     : x86/win32
[+] Running command sysinfo against sessions 2
System Language : en_US
OS              : Windows XP (Build 2600, Service Pack 2).
Computer        : CARLOS-192FCD91
Architecture    : x86
Meterpreter     : x86/win32
[-] Sessions 3 is not a Meterpreter Sessions!
[-] Sessions 4 is not a Meterpreter Sessions!
[-] Sessions 5 is not a Meterpreter Sessions!
[*] Auxiliary module execution completed

The other module is just the same but using a resource file. I think this modules will be useful to some of the users of the framework specially those not using Express or Pro and need to automate running modules or command against several sessions from the console.