Background
Working in air-gapped environment means not having the luxury to rely on respective application updater to check for newer version automatically. So I needed to manually download the installers from time to time, which is quite time-consuming and a mundane task.
I usually automate repetitive and mundane work, and I have always wanted to automate this process, but I wasn't able to figure out an effective way to do so because (as far as I know) there is no standardized way of grabbing the download URL from various websites. Let's take a look at some samples.
[
{
"application": "7zip.7zip",
"download_url": "https://www.7-zip.org/a/7z2200-x64.exe"
},
{
"application": "Docker.DockerDesktop",
"download_url": "https://desktop.docker.com/win/main/amd64/81317/Docker%20Desktop%20Installer.exe"
},
{
"application": "Git.Git",
"download_url": "https://github.com/git-for-windows/git/releases/download/v2.37.0.windows.1/Git-2.37.0-64-bit.exe"
},
{
"application": "GitExtensionsTeam.GitExtensions",
"download_url": "https://github.com/gitextensions/gitextensions/releases/download/v3.5.4/GitExtensions-3.5.4.12724-65f01f399.msi"
},
{
"application": "Google.Chrome",
"download_url": "https://dl.google.com/dl/chrome/install/googlechromestandaloneenterprise64.msi"
}
]
There is no fixed templates / format to the URL, and there isn't a way to query for the download URL except for some like Cypress that provides a json file with the version and URL.
Ideas
This thing has always been at the back of my mind for a number of years, and I imagine maintaining the URL myself like such
{
"application": "Google.Chrome",
"version": "103.0.5060.66",
"download_url": "https://dl.google.com/dl/chrome/install/googlechromestandaloneenterprise64.msi"
}
But I would need to manually update this as well, and doing so, would take even longer time than having to visit each website to download the installers myself. Hence, I didn't pursue the option since it didn't quite value-add for me.
Some other idea I had was to write scripts using cypress to navigate through the page to download various applications, but this will break easily if the website gets updated / changed. So the maintenance cost is high as well.
Winget
That is, until I came across winget. winget is a package manager for windows from Microsoft, it allows user to easily manage applications via winget just like you would on Linux using apt-get/yum.
Listing all the installed applications on my machine would look like
╰─ winget list --source winget
Name Id Version Available
-----------------------------------------------------------------------------------------------------------------------
StarUML MKLabs.StarUML 4.1.6
P3X Redis UI PatrikLaszlo.P3XRedisUI 2022.4.116 2022.4.126
Visual Studio Build Tools 2017 2fd33d5a 15.9.28307.1033
An example of a manifest for StarUML would look like
╰─ winget show "MKLabs.StarUML"
Found StarUML [MKLabs.StarUML]
Version: 4.1.6
Publisher: MKLabs Co.,Ltd.
Publisher Support Url: https:
Author: MKLabs Co.,Ltd
Moniker: staruml
Description: A sophisticated software modeler for agile and concise modeling.
Homepage: https:
License: Proprietary
License Url: https:
Copyright: Copyright (c) 2021 MKLabs Co.,Ltd.
Installer:
Type: nullsoft
Download Url: https:
SHA256: dd894019785afc4a49c1b18593ed2e0059ed28b6ce3bf85da0415da59a7a3d6d
While it is easy to list / install / upgrade the application via winget-cli, it is not as easy to parse the content as it is not a native PowerShell object, nor can it be exported to a json format. There are issue and discussion in GitHub tracking this closely, and I hope that these will be supported soon.
Nevertheless, I spent some time recently to write a simple PowerShell script to parse it, and it worked quite nicely.
Solution
I am not well verse in PowerShell script, so while it works, it may not be the best way to write it. I don't really care so much since it does its job as far as I'm concern.
Referring back to the StarUML manifest above. The key is to extract out the URL, and then download it.
This is the command to run to extract the URL
$url=winget show "MKLabs.StarUML" | Select-String "Download Url: " -NoEmphasis | Out-String | % { $_.Trim() } | % { $_.SubString(14) }
Let's take a look at how this command unfolds.
I will be showing the input and output in a single codebox
- Let's fetch the manifest of the application
> winget show "MKLabs.StarUML"
Found StarUML [MKLabs.StarUML]
Version: 4.1.6
Publisher: MKLabs Co.,Ltd.
Publisher Support Url: https:
Author: MKLabs Co.,Ltd
Moniker: staruml
Description: A sophisticated software modeler for agile and concise modeling.
Homepage: https:
License: Proprietary
License Url: https:
Copyright: Copyright (c) 2021 MKLabs Co.,Ltd.
Installer:
Type: nullsoft
Download Url: https:
SHA256: dd894019785afc4a49c1b18593ed2e0059ed28b6ce3bf85da0415da59a7a3d6d
- Use
Select-String to search for the text we want
> winget show "MKLabs.StarUML" | Select-String "Download Url: " -NoEmphasis
Download Url: https:
- Convert output into
String, otherwise, we can't apply String operation later on
> winget show "MKLabs.StarUML" | Select-String "Download Url: " -NoEmphasis | Out-String
Download Url: https:
The output look exactly the same as before, but in String format. If we don't, when we apply String operation, it will throw the following error
InvalidOperation: Method invocation failed because [Microsoft.PowerShell.Commands.MatchInfo] does not contain a method named 'Trim'.
- Apply
Trim operation to remove all whitespaces
> winget show "MKLabs.StarUML" | Select-String "Download Url: " -NoEmphasis | Out-String | % { $_.Trim() }
Download Url: https:
% is a shorthand for ForEach-Object
- Apply
SubString operation to grab only the actual URL
> winget show "MKLabs.StarUML" | Select-String "Download Url: " -NoEmphasis | Out-String | % { $_.Trim() } | % { $_.SubString(14) }
https:
And there we have it!
- To download, we can use
Invoke-WebRequest
Invoke-WebRequest -URI https://staruml.io/download/releases-v4/StarUML%20Setup%204.1.6.exe -OutFile ./staruml.exe
But it is very slow in download, and we have to specify explicit filename. So let's use curl instead
$default_download_dir="./_winget_applications"
$url=https://staruml.io/download/releases-v4/StarUML%20Setup%204.1.6.exe
# -L: follow redirect
# -O -J: we want to retain the remote filename instead of constructing our own
# see https://daniel.haxx.se/blog/2020/09/10/store-the-curl-output-over-there
# --create-dirs: if not exist
# --silent: do not show progress
curl -L $url -O -J --output-dir $default_download_dir --create-dirs --silent
So to recap, what we have done is to
- grab the URL from the manifest
- use curl to download the file
And what we have to do is to repeat the process for a list of applications. That's easy, just put it in a loop, and we're done.
For the complete script, please refer to my GitHub repository.
Conclusion
We looked at the step-by-step process of how I automate the mundane task of downloading application installers using PowerShell script with the help of winget-cli. However, this is not a foolproof solution as there are still a number of applications that have yet to onboard to winget, and so I will not be able to use this method to download those installers automatically.
I also hope to extend this automation to IDE plugins as well, which I have done so in the past for vscode but I now realize that there are much easier way to achieve the same goal.
Source Code
As usual, full source code is available in GitHub.
