Aug 24, 2012

Copy and Separate Files by Size

Playing around with Powershell again, and wrote a nifty little script to copy and separate files in a folder, limiting each new folder based on a user set size.
It requests input and output folders, size of output folders, and size type (ie. KB, MB, GB). Then copies from the input folder into the output folder, creating output folder if needed, and separating by numeric sub-folders. Before copying the objects are sorted by length(size) as it creates less total output folders, or should depending on your input. This sorting is entirely optional and could be removed if you wish.
We ran into a simple enough, issue initially when calculating total data to copy. We were checking if the division of moved size to total wanted  equaled the initial number given for total size. This caused the  total amount copied to be, wanted copied size times initial size number, resulting in quite an issue. Some debugging and head smacking fixed the issue by comparing to a total of 1. Enjoy the script and feel free to modify as needed!

Usage: CopyAndSeparate .\InputFolder .\OutputFolder 5 GB - Copies from .\InputFolder, moves to .\OutputFolder created sub-folders  to place files in every 5gb or so.

CopyAndSeparate.ps1

# Spenser Reinhardt
# 7-27-2012
# Modified 8-7-2012
# CopyandSeparate.ps1
# Finds files in a directory and separates them into new folders, creating a new folder each time a given folder size limit is reached
# Released under GPLv3 Licensing

Param (
    [parameter(Mandatory=$true)][string]$InputFolder,
    [parameter(Mandatory=$true)][string]$OutputFolder,
    [parameter(Mandatory=$true)][int]$FolderSize,
    [parameter(Mandatory=$true)][string]$SizeType
    )

#### Variables #####
$LoopCount = 0 #initializes loopcount for use later
[double]$LoopSize = 0 #initializes loopsize for use with loop lower on created as double to hold gb sized values
$LoopOutput = "" #initializes loopoutput for use with loop later on
$SizeDiv = "$FolderSize$SizeType"

#### Functions #####
Function NewFLCopy {

    Param ($FullName, [string]$Output)
   
    Write-host "Output: $Output`nFullName: $FullName"
    Try {
        New-Item -Path $Output -ItemType Directory # Creates new folder
    }
    Catch {
        Write-Error -message "Could not create $Output for destination files.`nStoping here." -RecommendedAction "stop"
    }
       
    Try {
        Copy-Item "$FullName" -Destination $Output
        }
    Catch {
        Write-Error -message "Could not copy file $FullName.`nStoping here." -RecommendedAction "stop"
    }
   
} # ends function

Function JustCopy {

    Param ($FullName, [string]$Output)
   
    Write-host "Output: $Output`nFullName: $FullName"
   
    Try {
        Copy-Item "$FullName" -Destination $Output
    }
    Catch {
        Write-Error -message "Could not copy file $FullName.`nStoping here." -RecommendedAction "stop"
    }

}

#### Logic ####

# tests if $input folder both exists and is only a folder not file
If ($(Test-Path $InputFolder -pathtype container) -eq $false) {
  
    Write-Error -message "Input folder is not a valid folder, please specify an existing folder and not individual files." -RecommendedAction "Stop"
    exit 1
}

# tests if $output folder exists
If ($(Test-Path $OutputFolder -pathtype container) -eq $false) {
  
    Write-Host Creating output folder: $Outputfolder
    Try {
        New-Item -Path $OutputFolder -ItemType Directory
    }
    Catch {
        Write-Error -message "Could not create $OutputFolder for destination files" -RecommendedAction "stop"
    }
      
}
Else {Write-Host Appears that the output folder already exists at: $OutputFolder`n Using this directory}

#tests sizetype for being valid
If (($SizeType -ne "B") -and ($SizeType -ne "KB") -and ($SizeType -ne "MB") -and ($SizeType -ne "GB") -and ($SizeType -ne "TB")) {

    Write-Error -message "Size type is incorrect, needs to be KB, MB, or GB.`nPlease try again"  -RecommendedAction "Stop"
    exit 2
    }

# If bytes, we don't need to add a modifier to sizediv to get the correct size, otherwise above it has already been concatenated correctly for other modifiers
If (($SizeType -eq "b") -or ($SizeType -eq "B")) {
    $SizeDiv = $FolderSize
    }

   
Get-ChildItem $InputFolder | sort Length | ForEach-Object {

    #checks if loopsize equals zero, indicating a new folder should be created for output and change of location to move files
    If ($LoopSize -eq 0) {
  
        $LoopSize = $_.Length # sets loopsize equal to first file of this folder
        $LoopOutput = "$OutputFolder\$LoopCount\" # sets name for new output folder
      
        Write-host "LoopOutput: $LoopOutput`nFullName: $_.FullName"
        NewFLCopy $_.FullName $LoopOutput
      
    } # ends if
  
    #checks if adding file will be less than or equal to alloted size and if so proceeds
    ElseIf ((($LoopSize + $_.Length)/$SizeDiv) -le 1) {
      
        $LoopSize = $_.Length + $LoopSize
      
        Write-host "LoopOutput: $LoopOutput`nFullName: "$_.FullName
        JustCopy $_.FullName $LoopOutput
     
    } # ends elseif
  
    #checking for adding current file would be greater than alloted space, if so resets for next itteration while copying current file into that new dir
    ElseIf ((($LoopSize + $_.Length)/$SizeDiv) -gt 1) {
   
        $LoopSize = $_.Length #resets loopsize
        $LoopCount++ # incriments loopcount
        $LoopOutput = "$OutputFolder\$LoopCount\" # sets name for new output folder
             
        Write-host "LoopOutput: $LoopOutput`nFullName: "$_.FullName
        NewFLCopy $_.FullName $LoopOutput
      
    } # ends elseif

} # ends foreach loop


CopyAndSeparate.psm1

# Spenser Reinhardt
# 7-27-2012
# Modified 8-7-2012
# CopyandSeparate.ps1
# Finds files in a directory and separates them into new folders, creating a new folder each time a given folder size limit is reached
# Released under GPLv3 Licensing

Param (
    [parameter(Mandatory=$true)][string]$InputFolder,
    [parameter(Mandatory=$true)][string]$OutputFolder,
    [parameter(Mandatory=$true)][int]$FolderSize,
    [parameter(Mandatory=$true)][string]$SizeType
    )

#### Variables #####
$LoopCount = 0 #initializes loopcount for use later
[double]$LoopSize = 0 #initializes loopsize for use with loop lower on created as double to hold gb sized values
$LoopOutput = "" #initializes loopoutput for use with loop later on
$SizeDiv = "$FolderSize$SizeType"

#### Functions #####
Function NewFLCopy {

    Param ($FullName, [string]$Output)
   
    Write-host "Output: $Output`nFullName: $FullName"
    Try {
        New-Item -Path $Output -ItemType Directory # Creates new folder
    }
    Catch {
        Write-Error -message "Could not create $Output for destination files.`nStoping here." -RecommendedAction "stop"
    }
       
    Try {
        Copy-Item "$FullName" -Destination $Output
        }
    Catch {
        Write-Error -message "Could not copy file $FullName.`nStoping here." -RecommendedAction "stop"
    }
   
} # ends function

Function JustCopy {

    Param ($FullName, [string]$Output)
   
    Write-host "Output: $Output`nFullName: $FullName"
   
    Try {
        Copy-Item "$FullName" -Destination $Output
    }
    Catch {
        Write-Error -message "Could not copy file $FullName.`nStoping here." -RecommendedAction "stop"
    }

}

Function CopyAndSeparate {
    # tests if $input folder both exists and is only a folder not file
    If ($(Test-Path $InputFolder -pathtype container) -eq $false) {
      
        Write-Error -message "Input folder is not a valid folder, please specify an existing folder and not individual files." -RecommendedAction "Stop"
        exit 1
    }

    # tests if $output folder exists
    If ($(Test-Path $OutputFolder -pathtype container) -eq $false) {
      
        Write-Host Creating output folder: $Outputfolder
        Try {
            New-Item -Path $OutputFolder -ItemType Directory
        }
        Catch {
            Write-Error -message "Could not create $OutputFolder for destination files" -RecommendedAction "stop"
        }
          
    }
    Else {Write-Host Appears that the output folder already exists at: $OutputFolder`n Using this directory}

    #tests sizetype for being valid
    If (($SizeType -ne "B") -and ($SizeType -ne "KB") -and ($SizeType -ne "MB") -and ($SizeType -ne "GB") -and ($SizeType -ne "TB")) {

        Write-Error -message "Size type is incorrect, needs to be KB, MB, or GB.`nPlease try again"  -RecommendedAction "Stop"
        exit 2
        }

    # If bytes, we don't need to add a modifier to sizediv to get the correct size, otherwise above it has already been concatenated correctly for other modifiers
    If (($SizeType -eq "b") -or ($SizeType -eq "B")) {
        $SizeDiv = $FolderSize
        }

       
    Get-ChildItem $InputFolder | sort Length | ForEach-Object {

        #checks if loopsize equals zero, indicating a new folder should be created for output and change of location to move files
        If ($LoopSize -eq 0) {
      
            $LoopSize = $_.Length # sets loopsize equal to first file of this folder
            $LoopOutput = "$OutputFolder\$LoopCount\" # sets name for new output folder
           
            Write-host "LoopOutput: $LoopOutput`nFullName: $_.FullName"
            NewFLCopy $_.FullName $LoopOutput
           
        } # ends if
      
        #checks if adding file will be less than or equal to alloted size and if so proceeds
        ElseIf ((($LoopSize + $_.Length)/$SizeDiv) -le 1) {
          
            $LoopSize = $_.Length + $LoopSize
          
            Write-host "LoopOutput: $LoopOutput`nFullName: "$_.FullName
            JustCopy $_.FullName $LoopOutput
          
        } # ends elseif
      
        #checking for adding current file would be greater than alloted space, if so resets for next itteration while copying current file into that new dir
        ElseIf ((($LoopSize + $_.Length)/$SizeDiv) -gt 1) {
       
            $LoopSize = $_.Length #resets loopsize
            $LoopCount++ # incriments loopcount
            $LoopOutput = "$OutputFolder\$LoopCount\" # sets name for new output folder
                  
            Write-host "LoopOutput: $LoopOutput`nFullName: "$_.FullName
            NewFLCopy $_.FullName $LoopOutput
           
        } # ends elseif

    } # ends foreach loop

} # ends function
 

Jul 11, 2012

Finding and Resolving IP's From Log Files with Powershell

Today I was given a fun little challenge from a friend. He needed to search through some server log files for IP addresses and recursively resolve hostnames from the found addresses. Considering this was on a Windows machine we made the obvious choice to play with Powershell. What came out is a useful script\function!

The script takes in an input file or folder, and an output file that ends up being a csv. With the input variable, it determines which you have provided, and moves on. If it is a folder, it recursively finds all files within the folder and uses the full path name to search files and proceeds from there. If it is a single file, it simply gets the contents and begins searching for IP addresses using regular expressions. Once all IPs are found and only unique objects are left, it uses [System.Net.Dns] to do a reverse look-up and determine hostnames if possible, if not the name is left as "Unresolved."

Please feel free to use and modify the scripts as you need. If you choose to put them in your own postings or packages, please reference where you found them.
* Note sorry for bloggers formatting... not too much I can do *
*Note 2 - Found a minor issue with it not sorting duplicates from multiple files, this has been corrected and now no matter how many files an IP is found in it only shows once in the resulting output *

FindIPs.ps1 - Used as a script

# Spenser Reinhardt
# Created 7-10-2012
# Last Modified 7-11-2012
# FindIPs.ps1
# Finds unique IPs from any sort of text file and outputs them to a csv

param([parameter(Mandatory=$true)][string]$InputType, [parameter(Mandatory=$true)][string]$OutputFile)

    new-variable -Name RegexIP -force -value ([regex]'\b(([01]?\d?\d|2[0-4]\d|25[0-5])\.){3}([01]?\d?\d|2[0-4]\d|25[0-5])\b')
    $BlankTtable = @{}

    # checks if directory if so, gets full path for each file and runs search\resolve on contents
    If ($(Test-Path $InputType -PathType container)) { # checks for dir
       
        Write-Output "Hostname, IP Address" > $OutputFile # Starts csv file from scratch
       
        Get-ChildItem -Path $InputType -Recurse | ForEach-Object{ # gets file names from dir and starts foreach
            $InputFile = $_.Fullname # sets input file to full name incase we are not in the same dir
            $IPs = $IPs + $RegexIP.matches($(Get-Content($InputFile)))
           
            } # ends regex foreach

        $IPs | select -unique value | ForEach-Object { # matches for correct regex, selects unique objects only, starts foreach
               
                $Resolved = [System.Net.Dns]::GetHostByAddress($_.value) # resolves IP to hostname
               
                # checks if resolved is empty, if so writes blank hostname and correct IP
                if ($Resolved -eq $BlankTable) { $IP = $_.value; Write-Output "Unresolved,$IP" >> $OutputFile }
                # else if not empty writes correct hostname and IP
                else { $Name = $Resolved.HostName; $IP = $Resolved.AddressList; Write-Output "$Name,$IP" >> $OutputFile }
               
                $Resolved=$BlankTable # Sets resolved back to blank for restart of loop
               
            } # ends multiple object foreach  
    } # ends if for dir check

    ElseIf ((Test-Path $InputType -PathType leaf)) { # Checks for single file
       
        $InputFile = $InputType # sets inputfile to inputtype as is only one file
        Write-Output "Hostname, IP Address" > $OutputFile # starts csv from scratch
       
        $RegexIP.matches($(Get-Content($InputFile))) | select -unique value | ForEach-Object { # matches for correct regex, selects unique objects only, starts foreach
            $Resolved = [System.Net.Dns]::GetHostByAddress($_.value) # resolves IP to hostname
           
            #checks if resolved is empty, if so writes blank hostname and correct IP
            if ($Resolved -eq $BlankTable) { $IP = $_.value; Write-Output "Unresolved,$IP" >> $OutputFile }
            #else if not empty writes correct hostname and IP
            else { $Name = $Resolved.HostName; $IP = $Resolved.AddressList; Write-Output "$Name,$IP" >> $OutputFile }
           
            $Resolved=$BlankTable # Sets resolved back to blank for restart of loop
           
        } # ends regex foreach
    } # ends elseif for file check

    Else { Write-Output "Input File is either not file or folder, or not valid!`n`nUsage: .\FindIP.ps1 [Input file\folder] [Output file]" }

FindIPs.psm1 - Can be imported as a function

 # Spenser Reinhardt
# Created 7-10-2012
# Last Modified 7-11-2012
# FindIPs.ps1
# Finds unique IPs from any sort of text file and outputs them to a csv
Function FindIPs {

param([parameter(Mandatory=$true)][string]$InputType, [parameter(Mandatory=$true)][string]$OutputFile)

    new-variable -Name RegexIP -force -value ([regex]'\b(([01]?\d?\d|2[0-4]\d|25[0-5])\.){3}([01]?\d?\d|2[0-4]\d|25[0-5])\b')
    $BlankTtable = @{}

    # checks if directory if so, gets full path for each file and runs search\resolve on contents
    If ($(Test-Path $InputType -PathType container)) { # checks for dir
       
        Write-Output "Hostname, IP Address" > $OutputFile # Starts csv file from scratch
       
        Get-ChildItem -Path $InputType -Recurse | ForEach-Object{ # gets file names from dir and starts foreach
            $InputFile = $_.Fullname # sets input file to full name incase we are not in the same dir
            $IPs = $IPs + $RegexIP.matches($(Get-Content($InputFile)))
           
            } # ends regex foreach

        $IPs | select -unique value | ForEach-Object { # matches for correct regex, selects unique objects only, starts foreach
               
                $Resolved = [System.Net.Dns]::GetHostByAddress($_.value) # resolves IP to hostname
               
                # checks if resolved is empty, if so writes blank hostname and correct IP
                if ($Resolved -eq $BlankTable) { $IP = $_.value; Write-Output "Unresolved,$IP" >> $OutputFile }
                # else if not empty writes correct hostname and IP
                else { $Name = $Resolved.HostName; $IP = $Resolved.AddressList; Write-Output "$Name,$IP" >> $OutputFile }
               
                $Resolved=$BlankTable # Sets resolved back to blank for restart of loop
               
            } # ends multiple object foreach  
    } # ends if for dir check

    ElseIf ((Test-Path $InputType -PathType leaf)) { # Checks for single file
       
        $InputFile = $InputType # sets inputfile to inputtype as is only one file
        Write-Output "Hostname, IP Address" > $OutputFile # starts csv from scratch
       
        $RegexIP.matches($(Get-Content($InputFile))) | select -unique value | ForEach-Object { # matches for correct regex, selects unique objects only, starts foreach
            $Resolved = [System.Net.Dns]::GetHostByAddress($_.value) # resolves IP to hostname
           
            #checks if resolved is empty, if so writes blank hostname and correct IP
            if ($Resolved -eq $BlankTable) { $IP = $_.value; Write-Output "Unresolved,$IP" >> $OutputFile }
            #else if not empty writes correct hostname and IP
            else { $Name = $Resolved.HostName; $IP = $Resolved.AddressList; Write-Output "$Name,$IP" >> $OutputFile }
           
            $Resolved=$BlankTable # Sets resolved back to blank for restart of loop
           
        } # ends regex foreach
    } # ends elseif for file check

    Else { Write-Output "Input File is either not file or folder, or not valid!`n`nUsage: .\FindIP.ps1 [Input file\folder] [Output file]" }
 } # ends function

Jul 6, 2012

Exploit-Exercises.com - Stack Overflows

This was a presentation that I gave at our local DC612 group. The premise was to walk through each of the stack overflow vulnerabilities in the Protostar virtual machine. Overall everything seemed to go very smooth and everyone seemed to appreciate the change from just being talked to, to more of a teaching environment and being walked through the examples on an individual basis when needed. Hopefully we can continue to do more of these in the future, and after speaking with @dc612 leaders it certainly sounds like this will be a welcome change!

Enjoy the presentation and feel free to contact me or comment if you need any help!



Feb 13, 2011

Scanning with NMAP

Scanning with NMAP is a talk I did this last week for our local Defcon group. Aside from a few comical mishaps, it was quite a pleasant and hopefully an informative talk. We over viewed basic scanning, output, and ids evasion techniques. Unfortunately the demo didn't work so well without our hosts dhcp working, apparently someone was having a bit of fun before the presentation... :) Below is a link to the slides, and hopefully next month we can look forward to an intro to assembly and debugging in Linux!

mplsCTFgames.org - Scanning With NMAP



Oct 7, 2010

B-Sides January

Very successful team building and planning event tonight. Not much to be said other than if everything plans out as we hope, this is gonna be one KICK ASS event! Presently from a CTF perspective many thoughts are floating about. Sidelines within the attack area for "Demonstrate\Teach an exploit" or "basic techniques of web penetration on live targets". Obviously much of this engages and looks to pull in the technically savvy, While I certainly wouldn't expect anything to the level of greatness that are Defcon, Schmoocon, or so many others, it would certainly be an amazing opportunity for both the trainer and students or budding engineers. Another immediate thought is subdivisions within the area it's self. Within each subdivision could be a live defense area, that is being attacked by a separate group, as well as network monitoring or forensics in another area. I really like this idea although do not believe it could be a center piece yet, likely more so an addition to a center layered lab environment. The lab it's self will likely remain secret to all but a few until the day of fun. However I would expect; wifi and hopefully rfid challenges, passive network IDS, use of pivoting in higher levels, and definitely vulnerable web applications and databases. Right now... this is certainly just a thought and a dream, but with any luck and whole lot of hard work, well let's just see what we can do!

Sep 9, 2010

DC612 September Meeting

I and a large portion of our local defcon group had a wonderful discussion on mplsctfgames.org. A new capture the flag\computer security event site that I've begun. With any luck and a lot of work, we hope to accomplish all of the following:

  • Monthly DC612 Challenges with prizes to be determined
  • Bi\weekly tutorials on, or help with selected challenges 
  • Conspire with Security B-Sides to create a challenge or event for the January meet.
  • Corporate sponsorship, shared resources, or of course beer and pizza money!
In addition I'd like to mention a forum created by a friend and past co-worker, that's a bit more disclosure and vulnerability discovery based. http://forum.Cipherflux.net/ Still in it's infancy, there is already some good talent posting recent news, tool reviews, and plenty of interesting stuff. While it could cause some friendly competition, I would encourage people to check it out and possibly use it as a side line to my own capture the flag events.

On a final note, I believe I owe some python scripts related to Windows memory analysis... Their done and still coming, maybe a week or two this time :)

Apr 5, 2010

Automated Volatility Memory Scanner

The latest Honeynet Project Forensic Challenge, is right up my alley! Live system and memory analysis is a large part of my interest in secure computing, especially related to malware and exploitation. While playing around with the sample file, it occurred to me that something like Volatility, could be much more useful, in a day to day basis, scripted out to run a general and\or more detailed scan, if needed.

At the moment, a quick scan shows the results for pslist, DLLlist, modules, connections, and sockets. The advanced scan is not yet complete, but should include something like entire registry export(may move to quick), driverscan, idt, ssdt, apihooks,and possibly mutantscan. I do not intend to overlap scans, but instead to offer a third complete scan, calling them both.

The script will be made available 4/19/2010, after the challenge is completed and submitted, and after I've had proper time to complete a V 1.0.

Feb 20, 2010

Nepenthes with Pharm, Dionaea, Glastopf, MWCollectd, Ubuntu Karmic, and The Fun That Is Honeypots!

I have been planning on deploying my own honeypot\net for several months. Its been a battle of having the resources for equipment, time for installation and analysis,  and simply determining which software to go with.  As you may know there are 2 main types of honeypots, low interaction, and high interaction, . A low interaction honeypot is generally, a piece of software running on a machine to emulate an exploitable service or software, to attract unwanted guests in your network, and log them accordingly. A high interaction honeypot is much the same, with the exception that it is generally an actual OS open to attack, with live vulnerabilities that are exploitable. The main differences are that the low interaction solutions are generally only able to accept and catalog known exploitations, and apply a best effort to capture unknown samples. A high interaction honeypot on the other hand, is theoretically able to capture any and all exploits as it is a live OS and would react exactly as the vulnerability would expect. There are many more subtleties and differences that are out of the scope of this post.

Back to the topic at hand, so in addition to the different types of honeypots, there also are different implementations of each, and as expected, some are far more efficient for an analyst, are more recently updated, have larger databases of vulnerabilities, or may be the only one that offers the exploitable service that your looking for. One of the main driving factors in honeypot development and advancement is the Honeynetproject.org. They seem to be a great and very helpful bunch of people from all over the world, intent on making the internet a safer place, by attempting to capture and analyze the nastiness in the world of computers. Two other major sites that seem to be very influential and helpful were carnivore.it and mwcollect.org The main projects that interest us for the sake of the article and following postings are:

Nepenthes
Nepenthes PHARM Client\Server Backend
Dionaea
Project Glastopf 
MWCollectd

For the sake of simplicity and completeness, I should mention that UnrealIRCd and Anope, mysql, Apache 2, and plenty of hard drive space were pre-installed on a separate Ubuntu Karmic 9.10 JEOS machine for use as a secured and separate backend for monitoring and storing results from our sensors activities. At this point, Nepenthes is up and running, along with the sql entries for the Pharm backend. I had some issues with the cgi scripts for PHARM as I installed and put everything in place as root, you must "chmod -R 0755 pharm/" and "chmod -R a+x pharm/cgi-bin/", now the only issue is the error on line 18 of the pharm_dologin.cgi, that I can't manage to pinpoint. In the meantime Glastopf is up next and already beginning setup!

As some may know or have thought to themselves, this is all well and good but certainly some of these applications have spillover or monitor the same ports, and you would most certainly be correct. However with all of the following applications. module based implementation and almost absolute configuration is key. They realize and have already thought that you may not want to use a certain portion of their specific application. The idea that I will be going for, is to give each of the softwares their fair shot with overlaying ports for a short period of time, otherwise  for the most part will follow the order below with decending priority when deciding overlapping ports;


Glastoph - Handles HTTP and port 80, 443, and first priority for needed ports
Dionaea - Focuses mainly on SMB and port 445, it may have others that I am not aware of, or develop in the future.
MWCollectd - A newer and better implementation of Nepenthes and Honeytrap.
Nepenthes and PHARM backend - Implemetation of further use of PHARM and Nepenthes without mwcollectd are doubtful as mwcollectd is signifigantly updated and revised code.

If you have created, develop, or otherwise work with any of the aforementioned projects or software, and would like to comment on my theory of merging projects, feel that I am misusing your code, or have any thoughts and suggestions, I certainly welcome it! Also if you know of other software, forums, projects, or otherwise that I should be informed of, please by all means share.

Note: Complete port scans with nmap and iptables turned off on the honeypot, will be uploaded per software as it is installed and working, along with different configurations with all or some programs running.

Thanks for reading,

Commie