Playing around with Powershell again, and wrote a nifty little script to copy and separate files in a folder, limiting each new folder based on a user set size.
It requests input and output folders, size of output folders, and size type (ie. KB, MB, GB). Then copies from the input folder into the output folder, creating output folder if needed, and separating by numeric sub-folders. Before copying the objects are sorted by length(size) as it creates less total output folders, or should depending on your input. This sorting is entirely optional and could be removed if you wish.
We ran into a simple enough, issue initially when calculating total data to copy. We were checking if the division of moved size to total wanted equaled the initial number given for total size. This caused the total amount copied to be, wanted copied size times initial size number, resulting in quite an issue. Some debugging and head smacking fixed the issue by comparing to a total of 1. Enjoy the script and feel free to modify as needed!
Usage: CopyAndSeparate .\InputFolder .\OutputFolder 5 GB - Copies from .\InputFolder, moves to .\OutputFolder created sub-folders to place files in every 5gb or so.
CopyAndSeparate.ps1
# Spenser Reinhardt
# 7-27-2012
# Modified 8-7-2012
# CopyandSeparate.ps1
# Finds files in a directory and separates them into new folders, creating a new folder each time a given folder size limit is reached
# Released under GPLv3 Licensing
Param (
[parameter(Mandatory=$true)][string]$InputFolder,
[parameter(Mandatory=$true)][string]$OutputFolder,
[parameter(Mandatory=$true)][int]$FolderSize,
[parameter(Mandatory=$true)][string]$SizeType
)
#### Variables #####
$LoopCount = 0 #initializes loopcount for use later
[double]$LoopSize = 0 #initializes loopsize for use with loop lower on created as double to hold gb sized values
$LoopOutput = "" #initializes loopoutput for use with loop later on
$SizeDiv = "$FolderSize$SizeType"
#### Functions #####
Function NewFLCopy {
Param ($FullName, [string]$Output)
Write-host "Output: $Output`nFullName: $FullName"
Try {
New-Item -Path $Output -ItemType Directory # Creates new folder
}
Catch {
Write-Error -message "Could not create $Output for destination files.`nStoping here." -RecommendedAction "stop"
}
Try {
Copy-Item "$FullName" -Destination $Output
}
Catch {
Write-Error -message "Could not copy file $FullName.`nStoping here." -RecommendedAction "stop"
}
} # ends function
Function JustCopy {
Param ($FullName, [string]$Output)
Write-host "Output: $Output`nFullName: $FullName"
Try {
Copy-Item "$FullName" -Destination $Output
}
Catch {
Write-Error -message "Could not copy file $FullName.`nStoping here." -RecommendedAction "stop"
}
}
#### Logic ####
# tests if $input folder both exists and is only a folder not file
If ($(Test-Path $InputFolder -pathtype container) -eq $false) {
Write-Error -message "Input folder is not a valid folder, please specify an existing folder and not individual files." -RecommendedAction "Stop"
exit 1
}
# tests if $output folder exists
If ($(Test-Path $OutputFolder -pathtype container) -eq $false) {
Write-Host Creating output folder: $Outputfolder
Try {
New-Item -Path $OutputFolder -ItemType Directory
}
Catch {
Write-Error -message "Could not create $OutputFolder for destination files" -RecommendedAction "stop"
}
}
Else {Write-Host Appears that the output folder already exists at: $OutputFolder`n Using this directory}
#tests sizetype for being valid
If (($SizeType -ne "B") -and ($SizeType -ne "KB") -and ($SizeType -ne "MB") -and ($SizeType -ne "GB") -and ($SizeType -ne "TB")) {
Write-Error -message "Size type is incorrect, needs to be KB, MB, or GB.`nPlease try again" -RecommendedAction "Stop"
exit 2
}
# If bytes, we don't need to add a modifier to sizediv to get the correct size, otherwise above it has already been concatenated correctly for other modifiers
If (($SizeType -eq "b") -or ($SizeType -eq "B")) {
$SizeDiv = $FolderSize
}
Get-ChildItem $InputFolder | sort Length | ForEach-Object {
#checks if loopsize equals zero, indicating a new folder should be created for output and change of location to move files
If ($LoopSize -eq 0) {
$LoopSize = $_.Length # sets loopsize equal to first file of this folder
$LoopOutput = "$OutputFolder\$LoopCount\" # sets name for new output folder
Write-host "LoopOutput: $LoopOutput`nFullName: $_.FullName"
NewFLCopy $_.FullName $LoopOutput
} # ends if
#checks if adding file will be less than or equal to alloted size and if so proceeds
ElseIf ((($LoopSize + $_.Length)/$SizeDiv) -le 1) {
$LoopSize = $_.Length + $LoopSize
Write-host "LoopOutput: $LoopOutput`nFullName: "$_.FullName
JustCopy $_.FullName $LoopOutput
} # ends elseif
#checking for adding current file would be greater than alloted space, if so resets for next itteration while copying current file into that new dir
ElseIf ((($LoopSize + $_.Length)/$SizeDiv) -gt 1) {
$LoopSize = $_.Length #resets loopsize
$LoopCount++ # incriments loopcount
$LoopOutput = "$OutputFolder\$LoopCount\" # sets name for new output folder
Write-host "LoopOutput: $LoopOutput`nFullName: "$_.FullName
NewFLCopy $_.FullName $LoopOutput
} # ends elseif
} # ends foreach loop
CopyAndSeparate.psm1
# Spenser Reinhardt
# 7-27-2012
# Modified 8-7-2012
# CopyandSeparate.ps1
# Finds files in a directory and separates them into new folders, creating a new folder each time a given folder size limit is reached
# Released under GPLv3 Licensing
Param (
[parameter(Mandatory=$true)][string]$InputFolder,
[parameter(Mandatory=$true)][string]$OutputFolder,
[parameter(Mandatory=$true)][int]$FolderSize,
[parameter(Mandatory=$true)][string]$SizeType
)
#### Variables #####
$LoopCount = 0 #initializes loopcount for use later
[double]$LoopSize = 0 #initializes loopsize for use with loop lower on created as double to hold gb sized values
$LoopOutput = "" #initializes loopoutput for use with loop later on
$SizeDiv = "$FolderSize$SizeType"
#### Functions #####
Function NewFLCopy {
Param ($FullName, [string]$Output)
Write-host "Output: $Output`nFullName: $FullName"
Try {
New-Item -Path $Output -ItemType Directory # Creates new folder
}
Catch {
Write-Error -message "Could not create $Output for destination files.`nStoping here." -RecommendedAction "stop"
}
Try {
Copy-Item "$FullName" -Destination $Output
}
Catch {
Write-Error -message "Could not copy file $FullName.`nStoping here." -RecommendedAction "stop"
}
} # ends function
Function JustCopy {
Param ($FullName, [string]$Output)
Write-host "Output: $Output`nFullName: $FullName"
Try {
Copy-Item "$FullName" -Destination $Output
}
Catch {
Write-Error -message "Could not copy file $FullName.`nStoping here." -RecommendedAction "stop"
}
}
Function CopyAndSeparate {
# tests if $input folder both exists and is only a folder not file
If ($(Test-Path $InputFolder -pathtype container) -eq $false) {
Write-Error -message "Input folder is not a valid folder, please specify an existing folder and not individual files." -RecommendedAction "Stop"
exit 1
}
# tests if $output folder exists
If ($(Test-Path $OutputFolder -pathtype container) -eq $false) {
Write-Host Creating output folder: $Outputfolder
Try {
New-Item -Path $OutputFolder -ItemType Directory
}
Catch {
Write-Error -message "Could not create $OutputFolder for destination files" -RecommendedAction "stop"
}
}
Else {Write-Host Appears that the output folder already exists at: $OutputFolder`n Using this directory}
#tests sizetype for being valid
If (($SizeType -ne "B") -and ($SizeType -ne "KB") -and ($SizeType -ne "MB") -and ($SizeType -ne "GB") -and ($SizeType -ne "TB")) {
Write-Error -message "Size type is incorrect, needs to be KB, MB, or GB.`nPlease try again" -RecommendedAction "Stop"
exit 2
}
# If bytes, we don't need to add a modifier to sizediv to get the correct size, otherwise above it has already been concatenated correctly for other modifiers
If (($SizeType -eq "b") -or ($SizeType -eq "B")) {
$SizeDiv = $FolderSize
}
Get-ChildItem $InputFolder | sort Length | ForEach-Object {
#checks if loopsize equals zero, indicating a new folder should be created for output and change of location to move files
If ($LoopSize -eq 0) {
$LoopSize = $_.Length # sets loopsize equal to first file of this folder
$LoopOutput = "$OutputFolder\$LoopCount\" # sets name for new output folder
Write-host "LoopOutput: $LoopOutput`nFullName: $_.FullName"
NewFLCopy $_.FullName $LoopOutput
} # ends if
#checks if adding file will be less than or equal to alloted size and if so proceeds
ElseIf ((($LoopSize + $_.Length)/$SizeDiv) -le 1) {
$LoopSize = $_.Length + $LoopSize
Write-host "LoopOutput: $LoopOutput`nFullName: "$_.FullName
JustCopy $_.FullName $LoopOutput
} # ends elseif
#checking for adding current file would be greater than alloted space, if so resets for next itteration while copying current file into that new dir
ElseIf ((($LoopSize + $_.Length)/$SizeDiv) -gt 1) {
$LoopSize = $_.Length #resets loopsize
$LoopCount++ # incriments loopcount
$LoopOutput = "$OutputFolder\$LoopCount\" # sets name for new output folder
Write-host "LoopOutput: $LoopOutput`nFullName: "$_.FullName
NewFLCopy $_.FullName $LoopOutput
} # ends elseif
} # ends foreach loop
} # ends function
Aug 24, 2012
Jul 11, 2012
Finding and Resolving IP's From Log Files with Powershell
Today I was given a fun little challenge from a friend. He needed to search through some server log files for IP addresses and recursively resolve hostnames from the found addresses. Considering this was on a Windows machine we made the obvious choice to play with Powershell. What came out is a useful script\function!
The script takes in an input file or folder, and an output file that ends up being a csv. With the input variable, it determines which you have provided, and moves on. If it is a folder, it recursively finds all files within the folder and uses the full path name to search files and proceeds from there. If it is a single file, it simply gets the contents and begins searching for IP addresses using regular expressions. Once all IPs are found and only unique objects are left, it uses [System.Net.Dns] to do a reverse look-up and determine hostnames if possible, if not the name is left as "Unresolved."
Please feel free to use and modify the scripts as you need. If you choose to put them in your own postings or packages, please reference where you found them.
* Note sorry for bloggers formatting... not too much I can do *
*Note 2 - Found a minor issue with it not sorting duplicates from multiple files, this has been corrected and now no matter how many files an IP is found in it only shows once in the resulting output *
The script takes in an input file or folder, and an output file that ends up being a csv. With the input variable, it determines which you have provided, and moves on. If it is a folder, it recursively finds all files within the folder and uses the full path name to search files and proceeds from there. If it is a single file, it simply gets the contents and begins searching for IP addresses using regular expressions. Once all IPs are found and only unique objects are left, it uses [System.Net.Dns] to do a reverse look-up and determine hostnames if possible, if not the name is left as "Unresolved."
Please feel free to use and modify the scripts as you need. If you choose to put them in your own postings or packages, please reference where you found them.
* Note sorry for bloggers formatting... not too much I can do *
*Note 2 - Found a minor issue with it not sorting duplicates from multiple files, this has been corrected and now no matter how many files an IP is found in it only shows once in the resulting output *
FindIPs.ps1 - Used as a script
# Spenser Reinhardt
# Created 7-10-2012
# Last Modified 7-11-2012
# FindIPs.ps1
# Finds unique IPs from any sort of text file and outputs them to a csv
param([parameter(Mandatory=$true)][string]$InputType, [parameter(Mandatory=$true)][string]$OutputFile)
new-variable -Name RegexIP -force -value ([regex]'\b(([01]?\d?\d|2[0-4]\d|25[0-5])\.){3}([01]?\d?\d|2[0-4]\d|25[0-5])\b')
$BlankTtable = @{}
# checks if directory if so, gets full path for each file and runs search\resolve on contents
If ($(Test-Path $InputType -PathType container)) { # checks for dir
Write-Output "Hostname, IP Address" > $OutputFile # Starts csv file from scratch
Get-ChildItem -Path $InputType -Recurse | ForEach-Object{ # gets file names from dir and starts foreach
$InputFile = $_.Fullname # sets input file to full name incase we are not in the same dir
$IPs = $IPs + $RegexIP.matches($(Get-Content($InputFile)))
} # ends regex foreach
$IPs | select -unique value | ForEach-Object { # matches for correct regex, selects unique objects only, starts foreach
$Resolved = [System.Net.Dns]::GetHostByAddress($_.value) # resolves IP to hostname
# checks if resolved is empty, if so writes blank hostname and correct IP
if ($Resolved -eq $BlankTable) { $IP = $_.value; Write-Output "Unresolved,$IP" >> $OutputFile }
# else if not empty writes correct hostname and IP
else { $Name = $Resolved.HostName; $IP = $Resolved.AddressList; Write-Output "$Name,$IP" >> $OutputFile }
$Resolved=$BlankTable # Sets resolved back to blank for restart of loop
} # ends multiple object foreach
} # ends if for dir check
ElseIf ((Test-Path $InputType -PathType leaf)) { # Checks for single file
$InputFile = $InputType # sets inputfile to inputtype as is only one file
Write-Output "Hostname, IP Address" > $OutputFile # starts csv from scratch
$RegexIP.matches($(Get-Content($InputFile))) | select -unique value | ForEach-Object { # matches for correct regex, selects unique objects only, starts foreach
$Resolved = [System.Net.Dns]::GetHostByAddress($_.value) # resolves IP to hostname
#checks if resolved is empty, if so writes blank hostname and correct IP
if ($Resolved -eq $BlankTable) { $IP = $_.value; Write-Output "Unresolved,$IP" >> $OutputFile }
#else if not empty writes correct hostname and IP
else { $Name = $Resolved.HostName; $IP = $Resolved.AddressList; Write-Output "$Name,$IP" >> $OutputFile }
$Resolved=$BlankTable # Sets resolved back to blank for restart of loop
} # ends regex foreach
} # ends elseif for file check
Else { Write-Output "Input File is either not file or folder, or not valid!`n`nUsage: .\FindIP.ps1 [Input file\folder] [Output file]" }
FindIPs.psm1 - Can be imported as a function
# Spenser Reinhardt
# Created 7-10-2012
# Last Modified 7-11-2012
# FindIPs.ps1
# Finds unique IPs from any sort of text file and outputs them to a csv
Function FindIPs {
param([parameter(Mandatory=$true)][string]$InputType, [parameter(Mandatory=$true)][string]$OutputFile)
new-variable -Name RegexIP -force -value ([regex]'\b(([01]?\d?\d|2[0-4]\d|25[0-5])\.){3}([01]?\d?\d|2[0-4]\d|25[0-5])\b')
$BlankTtable = @{}
# checks if directory if so, gets full path for each file and runs search\resolve on contents
If ($(Test-Path $InputType -PathType container)) { # checks for dir
Write-Output "Hostname, IP Address" > $OutputFile # Starts csv file from scratch
Get-ChildItem -Path $InputType -Recurse | ForEach-Object{ # gets file names from dir and starts foreach
$InputFile = $_.Fullname # sets input file to full name incase we are not in the same dir
$IPs = $IPs + $RegexIP.matches($(Get-Content($InputFile)))
} # ends regex foreach
$IPs | select -unique value | ForEach-Object { # matches for correct regex, selects unique objects only, starts foreach
$Resolved = [System.Net.Dns]::GetHostByAddress($_.value) # resolves IP to hostname
# checks if resolved is empty, if so writes blank hostname and correct IP
if ($Resolved -eq $BlankTable) { $IP = $_.value; Write-Output "Unresolved,$IP" >> $OutputFile }
# else if not empty writes correct hostname and IP
else { $Name = $Resolved.HostName; $IP = $Resolved.AddressList; Write-Output "$Name,$IP" >> $OutputFile }
$Resolved=$BlankTable # Sets resolved back to blank for restart of loop
} # ends multiple object foreach
} # ends if for dir check
ElseIf ((Test-Path $InputType -PathType leaf)) { # Checks for single file
$InputFile = $InputType # sets inputfile to inputtype as is only one file
Write-Output "Hostname, IP Address" > $OutputFile # starts csv from scratch
$RegexIP.matches($(Get-Content($InputFile))) | select -unique value | ForEach-Object { # matches for correct regex, selects unique objects only, starts foreach
$Resolved = [System.Net.Dns]::GetHostByAddress($_.value) # resolves IP to hostname
#checks if resolved is empty, if so writes blank hostname and correct IP
if ($Resolved -eq $BlankTable) { $IP = $_.value; Write-Output "Unresolved,$IP" >> $OutputFile }
#else if not empty writes correct hostname and IP
else { $Name = $Resolved.HostName; $IP = $Resolved.AddressList; Write-Output "$Name,$IP" >> $OutputFile }
$Resolved=$BlankTable # Sets resolved back to blank for restart of loop
} # ends regex foreach
} # ends elseif for file check
Else { Write-Output "Input File is either not file or folder, or not valid!`n`nUsage: .\FindIP.ps1 [Input file\folder] [Output file]" }
} # ends function
Jul 6, 2012
Exploit-Exercises.com - Stack Overflows
This was a presentation that I gave at our local DC612 group. The premise was to walk through each of the stack overflow vulnerabilities in the Protostar virtual machine. Overall everything seemed to go very smooth and everyone seemed to appreciate the change from just being talked to, to more of a teaching environment and being walked through the examples on an individual basis when needed. Hopefully we can continue to do more of these in the future, and after speaking with @dc612 leaders it certainly sounds like this will be a welcome change!
Enjoy the presentation and feel free to contact me or comment if you need any help!
Enjoy the presentation and feel free to contact me or comment if you need any help!
Subscribe to:
Posts (Atom)