Practical PowerShell Scripting for DevOps - Part 5
File Operations, Zip Archiving, Error Handling, and Logging
Table of Contents
Hi, thank you for joining me at Part 5. On this part, I’ll ask you to do a simple task, that’ll add valuable tools to your toolchain. I’ll ask you to move and rename files around while changing the folder structure of some configuration files and create zip archives. Within the same script, I’ll also ask you to use error handling and logging practices to make our script more usable, and easy to debug for our users.
One thing, I really value in DevOps engineers, is that their ability to produce work that is suitable for collaboration. When we write a piece of code, especially that’ll be used in an automation scenario, that code should be verbose and robust to ensure, it performs as required all of the time and doesn’t generate unwanted results. Even if it doesn’t work for whatever reason, it should generate meaningful, easy to understand logs that even a first time user can easily understand, that user would know what happened during and after the execution of the code.
Problem
Hey fellow DevOps Engineer! We have a system where we need to upload the configuration files through the portal, they don’t have an officially supported API other than the web application user interface, so this needs to be performed by our engineers. The files are source controlled in a git repository however, they are not in a format that the UI upload process excepts. The UI needs each file in .zip extension, and we also follow a unique naming convention for the files. We prefix all files with environment name, so the configuration files for that environment are grouped together when all files are sorted by name.
To make it easier for you to understand, we want you to write a script that can be ran in a pipeline, that’ll convert the files from the original structure that they are stored in the git repository, to the target desired structure, so pipeline can publish them as individual, environment specific configuration artifacts.
Folder Structure in the Git Repository:
configs/
├── dev/
│ ├── service-WEB-configs.txt
│ ├── service-DATABASE-configs.txt
│ ├── service-COMMON-configs.txt
│ ├── service-API-configs.txt
│ └── service-ADMIN-PANEL-configs.txt
├── qa/
├── uat/
└── prod/
should be transformed into, target structure, including the casing:
configs/
├── DEV/
│ ├── DEV_service_web_configs.zip
│ ├── DEV_service_database_configs.zip
│ ├── DEV_service_common_configs.zip
│ ├── DEV_service_api_configs.zip
│ └── DEV_service_admin_panel_configs.zip
├── QA/
├── UAT/
└── PROD/
We can then, instruct our pipeline to publish each environment folder path such as DEV, QA, UAT, PROD as an artifact so they would become an artifact with all configuration files relevant to that environment, and in the format of how this system’s UI expects the upload.
As this script will run inside a CI/CD pipeline, we also need the script to log what operation it’s performing, with a timestamp and if the operation is successful or failed.
What Success Looks like for this Challenge
The main goal is to, create a new folder structure with desired layout and folder and file naming as described in the problem section. Mind you that, some parts of the filenames or folder names are converted into upper or lowercase characters. The casing does matter in the solution as our task is clearly defining that.
The requestor also asked to have proper logging of the operations being performed by the script and validating the successful completion of such operations.
You can create the initial structure of folders and files via this script:
Part-5 Files - Create-ConfigsDirectory.ps1
# PowerShell Script to Create Initial Config File Structure
$envs = 'dev', 'qa', 'uat', 'prod'
$services = 'WEB', 'DATABASE', 'COMMON', 'API', 'ADMIN-PANEL'
foreach ($env in $envs) {
$dirPath = Join-Path -Path "./configs" -ChildPath $env
New-Item -ItemType Directory -Force -Path $dirPath
foreach ($service in $services) {
$fileName = "service-$service-configs.txt"
$filePath = Join-Path -Path $dirPath -ChildPath $fileName
New-Item -ItemType File -Force -Path $filePath
}
}
Assuming you’ve create the initial state of the filesystem with above script, the successfull run will look like below:
Successful Run:
The file structure after the successful run:
Some Pointers For Writing Your Own Solution
Before you attempt this challenge, let’s break down the problem first. We’ll need to perform some tasks.
1- We will need to learn about all the folder to learn about environments and then all the files files present in those folders and loop through these files in a recursive manner. So you need to use a looping mechanism that is available to powershell language.
2- We need to construct the new name for the destination file names for the zip archives we’ll create. In order to achieve this you’ll need to use available string operation function to powershell. This can be done in multiple ways, so there is no prescribed formula for achieving this. I usually use techniques to get the substrings out of the full strings of paths to get what I require and then construct my final string in final step. You’ll understand this better when you see my sample solution script.
3- We need to create .zip archives out of folders. You can achieve this via a pre-built tool like 7zip or else, but as zip is a quite common archive extension, powershell has a native cmdlet that can achieve this for us.
4- Bonus challenge: you can even write another script to validate if end result satisfies the requirements, rather than checking yourself manually.
The Main Learnings From the Challenge
The main learnings from this challenge are:
1- Usage of Get-ChildItem cmdlet which is one of the most common cmdlets while performing file operations.
2- String manipulations functions and operators
3- Usage of Compress-Archive cmdlet for creating archives out of files and folders.
Links:
Replacement operators
Tasks You Need To Perform In Your Script
So to give another breakdown of what you need to do in your script,
1- Make sure the destination folder exists and if not create it.
2- Process the folder and the files within to discover the folders and file to be created.
3- Process the config files and create zip archives with desired name and folder structure as necessary.
4- Print out relevant log messages to terminal that allows you to clearly see the input being evaluated and on failure, what requirement it fails and if satisfies both conditions a relevant success message. (Normally you would either fail or proceed with the pipeline runs according to the result but just printing a message is fine for this mock challenge)
Final Tips
From this point below you will see my sample solution. Depending on your level and learning style, either take a look before or after you gave it your shot.
My Sample Solution
Files for my Sample Solution
Sample-Solution.ps1
[CmdletBinding()]
# Define paths
param(
[string]$rootPath = "./configs",
[string]$targetPath = "./organised-configs"
)
# Function to log messages
function Out-Log {
param([string]$Message)
Write-Host "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss'): $Message"
}
# Create the target directory if it doesn't exist
try {
if (-not (Test-Path $targetPath)) {
New-Item -Path $targetPath -ItemType Directory | Out-Null
Out-Log "Created target directory at $targetPath"
}
} catch {
Out-Log "Error creating target directory: $_"
}
# Process each environment folder
Get-ChildItem -Path $rootPath -Directory | ForEach-Object {
$env = $_.Name.ToUpper()
$newEnvPath = Join-Path -Path $targetPath -ChildPath $env
try {
New-Item -Path $newEnvPath -ItemType Directory -Force | Out-Null
Out-Log "Created new environment folder: $env"
} catch {
Out-Log "Error creating environment folder $($env): $_"
continue
}
# $_ is shorthand form of $PSItem variable, wanted to show you usage of both
Get-ChildItem -Path $PSItem.FullName -File | ForEach-Object {
$serviceName = $_.BaseName -replace 'service-', '' -replace '-configs', ''
$newFileName = "$($env)_service_$($serviceName.ToLower().Replace('-','_'))_configs.zip"
$newFilePath = Join-Path -Path $newEnvPath -ChildPath $newFileName
try {
Compress-Archive -Path $_.FullName -DestinationPath $newFilePath -Force
Out-Log "Archived file: $newFileName"
} catch {
Out-Log "Error archiving file $($_): $_"
}
}
}
Sample Solution Script explained
This PowerShell script organizes and archives configuration files from a specified root directory into a target directory. Here’s a breakdown of the tasks it performs:
Parameter Definition:
- The script defines two parameters, $rootPath and $targetPath, which represent the source and destination directories, respectively. Default values are provided.
Logging Function:
- A function Out-Log is defined to log messages with a timestamp. This function prints messages to the console with the current date and time.
Target Directory Creation:
- The script checks if the target directory ($targetPath) exists. If it doesn’t, the script creates the directory and logs the creation. If an error occurs during this process, it logs the error message.
Processing Environment Folders:
- The script retrieves all subdirectories (representing different environments) from the root directory ($rootPath).
- For each environment directory:
- Converts the environment name to uppercase and constructs a new path within the target directory.
- Creates a new directory for the environment in the target path and logs the creation. If an error occurs, it logs the error and skips to the next environment directory.
Processing Configuration Files:
- For each file in the current environment directory:
- Extracts the service name from the file name by removing certain substrings (service- and -configs).
- Constructs a new file name by combining the environment name, service name (converted to lowercase and with hyphens replaced by underscores), and a _configs.zip suffix.
- Archives (compresses) the file into a ZIP format and saves it in the new environment directory within the target path.
- Logs the successful archiving of the file. If an error occurs during the compression, it logs the error.
- For each file in the current environment directory:
Summary
The script organizes configuration files by environment, compressing each configuration file into a ZIP archive with a standardized naming convention, and places the archived files into corresponding directories in the target path. It includes error handling and logging to track the process and any issues that arise.
Sample Runs
As our script is deterministic, it should always return the same result with the given input.
Conclusion
Working with files and archives will be common requirement for working in a DevOps field in various types of duties. These skills can come handy in CI/CD pipeline applications or any type of automation capability that’d require usage of files. Learning how to manipulate strings and being able to construct a string that is in desired format via the input strings available is also a very common task. It can help you transform any type of log or config file or similar to the string outputs that’ll be useful in another task.
Hopefully by the end of this challenge, you’ll go away better equipped with powershell techniques that’ll allow you to perform similar tasks like this mock challenge. I will see you in the next chapter of this series.
