Invoke-WebRequest: Perform HTTP Requests, Download Files, Parse Web with PowerShell

The Invoke-WebRequest cmdlet can be used to request HTTP/HTTPS/FTP resources directly from the PowerShell console. You can use this command to send HTTP requests (GET and POST), download files from a website, parse HTML web pages, perform authentication, fill out and submit web forms, etc. In this article, we’ll cover basic examples of using the Invoke-WebRequest cmdlet in PowerShell to interact with web services.

Get Web Page Content with Invoke-WebRequest Cmdlet

The Invoke-WebRequest cmdlet allows you to send the HTTP, HTTPS, or FTP request with the GET method to the specified web page and receive a response from the server. This cmdlet has been available on Windows since version PowerShell 3.0.

Run the following command:

Invoke-WebRequest -Uri "https://woshub.com"

Invoke-WebRequest:  Unable to connect to the remote server
CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

The command loaded the page and displayed its contents in the PowerShell console. The returned response is not just the HTML code of the page. The Invoke-WebRequest cmdlet returns an object of type HtmlWebResponseObject. Such an object is a collection of forms, links, images, and other important elements of an HTML document. Let’s look at all the properties of this object:

$WebResponseObj = Invoke-WebRequest -Uri "https://woshub.com"
$WebResponseObj| Get-Member

To get the raw HTML code of the web page that is contained in the HtmlWebResponseObject object, run:

$WebResponseObj.content

You can list the HTML code along with the HTTP headers returned by the web server:

$WebResponseObj.rawcontent

You can check only the web server HTTP status code and the HTTP headers of the HTML page:

$WebResponseObj.Headers

As you can see, the server has returned a response 200. This means that the request has been successful, and the web server is available and works correctly.

Key               Value
---               -----
Transfer-Encoding chunked
Connection        keep-alive
Vary              Accept-Encoding,Cookie
Cache-Control     max-age=3, must-revalidate
Content-Type      text/html; charset=UTF-8
Date              Wed, 13 Jul 2022 02:28:32 GMT
Server            nginx/1.20.2
X-Powered-By      PHP/5.6.40

To get the last modification time of a web page:

$WebResponseObj.ParsedHtml | Select lastModified

You can specify a User Agent string when connecting to a web resource. PowerShell has a set of built-in User Agent strings:

invoke-webRequest -Uri $uri -userAgent ([Microsoft.PowerShell.Commands.PSUserAgent]::Chrome)

The list of available agents in PowerShell can be displayed like this:

[Microsoft.PowerShell.Commands.PSUserAgent].GetProperties() | Select-Object Name, @{n='UserAgent';e={ [Microsoft.PowerShell.Commands.PSUserAgent]::$($_.Name) }}

Or you can set your own UserAgent string:

Invoke-WebRequest -Uri $uri -UserAgent 'MyApplication/1.1'

Using Invoke-WebRequest with Authentication

Some web resources require authentication to access. You can use various types of authentication with the Invoke-WebRequest cmdlet (Basic, NTLM, Kerberos, OAuth, or certificate authentication).

To perform Basic Authentication (authentication by name and password encoded in base64), you first need to get the username and password:

$cred = Get-Credential
wget -Uri 'https://somesite.com' -Credential $cred

In order to use the current Windows user credentials to perform NTLM or Kerberos authentication, add the -UseDefaultCredentials option:

Invoke-WebRequest 'https://somesite.com' -UseDefaultCredentials

To authenticate with a certificate, you need to specify its thumbprint:

Invoke-WebRequest 'https://somesite.com' -CertificateThumbprint xxxxxxxxxxxxxxxxxxxxxxxxxxxxx

You can use modern Bearer/OAuth token authentication in your PowerShell scripts.

1. First, you need to get an OAuth token from your REST API provider (out of the scope of this post);
2. Convert token with ConvertTo-SecureString cmdlet: $Token = "12345678912345678901234567890" | ConvertTo-SecureString -AsPlainText –Force
3. Now you can perform OAuth authentication:
   $Params = @{
Uri = "https://somesite.com"
Authentication = "Bearer"
Token = $Token }
Invoke-RestMethod @Params

Parse and Scrape HTML a Web Page with PowerShell

The Invoke-WebRequest cmdlet allows you to quickly and conveniently parse the content of any web page. When processing an HTML page, collections of links, web forms, images, scripts, etc., are created.

Let’s look at how to access specific objects on a web page. For example, I want to get a list of all outgoing links (A HREF objects) on the target HTML web page:

$SiteAdress = "https://woshub.com"
$HttpContent = Invoke-WebRequest -URI $SiteAdress
$HttpContent.Links | Foreach {$_.href }

To get the link text itself (contained in the InnerText element), you can use the following command:

$HttpContent.Links | fl innerText, href

You can only select links with a specific CSS class:

$HttpContent.Links | Where-Object {$_.class -eq "page-numbers"} | fl innerText, href

Or specific text in the URL address:

$HttpContent.Links | Where-Object {$_.href -like "*powershell*"} | fl innerText,href

Then display a list of all images on this page:

$Img.Images

Create a collection of full URL paths to these images:

$images = $Img.Images | select src

Initialize a new instance of WebClient class:

$wc = New-Object System.Net.WebClient

And download all the image files from the page (with their original filenames) to the c:\too1s\ folder:

$images | foreach { $wc.DownloadFile( $_.src, ("c:\tools\"+[io.path]::GetFileName($_.src) ) ) }

How to Download File over HTTP/FTP with PowerShell Wget (Invoke-WebRequest)?

Invoke-WebRequest allows you to download files from a web page or FTP site (works like Wget or cURL on Windows). Suppose, you want to download a file from an HTTP using PowerShell. Run the following PowerShell command:

wget "https://download-installer.cdn.mozilla.net/pub/firefox/releases/102.0.1/win64/en-US/Firefox%20Setup%20102.0.1.exe" -outfile “c:\tools\firefox_setup.exe”

This command will download the file from the HTTP site and save it to the specified directory.

You can get the size of a file in MB before downloading it with wget:

$url = "https://download-installer.cdn.mozilla.net/pub/firefox/releases/102.0.1/win64/en-US/Firefox%20Setup%20102.0.1.exe"
(Invoke-WebRequest $url -Method Head).Headers.'Content-Length'/1Mb

Below is an example of a PowerShell script that will find all links to *.pdf files on a target web page and bulk download all found files from the website to your computer (each pdf file is saved under a random name):

$OutDir="C:\docs\download\PDF"
$SiteAdress = "https://sometechdocs.com/pdf"
$HttpContent = Invoke-WebRequest -URI $SiteAdress
$HttpContent.Links | Where-Object {$_.href -like "*.pdf"} | %{Invoke-WebRequest -Uri $_.href -OutFile ($OutDir + $(Get-Random 200000)+".pdf")}

As a result of the script in the target directory, all pdf files from the page will be downloaded. Each file is saved under a random name.

How to Fill and Submit HTML Form with PowerShell?

Many web services require filling various data into HTML forms. With Invoke-WebRequest, you can access any HTML form, fill in the necessary fields, and submit the completed form back to the server. In this example, we’ll show how to sign in to Facebook through its standard web form using PowerShell.

With the following command, we will save the information about connection cookies in a separate session variable:

$fbauth = Invoke-WebRequest https://www.facebook.com/login.php -SessionVariable session

Using the next command, display the list of the fields to fill in the login HTML form (login_form):

$fbauth.Forms["login_form"].Fields

Assign the desired values to all fields:

$fbauth.Forms["login_form"].Fields["email"] = "[email protected]"

$fbauth.Forms["login_form"].Fields["pass"] = "Coo1P$wd"

Etc.

To submit (sent) the filled form to the server, call the action attribute of the HTML form:

$Log = Invoke-WebRequest -method POST -URI ("https://www.facebook.com/login.php" + $fbauth.Forms["login_form"].Action) -Body $fbauth.Forms["login_form"].Fields -WebSession $session

You can also use the JSON format to send data to a web page with the POST method:

$headers = @{
'Content-Type'='application/json'
'apikey'='0987654321'
}
$jsonbody = @{
"siteUrl" ="https://somesite.com"
"email" = "[email protected]"
}
Invoke-WebRequest -Method 'Post' -Uri $url -Body ($jsonbody |ConvertTo-Json) -Headers $headers -ContentType "application/json"

Invoke-WebRequest: Ignore SSL/TLS Certificate Checks

Another issue is that the Invoke-WebRequest cmdlet is closely related to Internet Explorer. For example, in Windows Server Core editions in which IE is not installed (or removed), the Invoke-WebRequest cmdlet cannot be used.

Invoke-WebRequest: The response content cannot be parsed because the Internet Explorer engine is not available, or Internet Explorer’s first-launch configuration is not complete. Specify the UseBasicParsing parameter and try again.

In this case, the WebClient class can be used instead of Invoke-WebRequest. For example, to download a file from a specified URL, use the command.

(New-Object -TypeName 'System.Net.WebClient').DownloadFile($Url, $FileName)

If an invalid SSL certificate is used on an HTTPS site, or PowerShell doesn’t support this type of SSL/TLS protocol, then the Invoke-WebRequest cmdlet drops the connection.

Invoke-WebRequest : The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

Invoke-WebRequest : The request was aborted: Could not create SSL/TLS secure channel.

By default, Windows PowerShell (in early builds of Windows 10, Windows Server 2016, and older versions of Windows) uses the legacy and insecure TLS 1.0 protocol for connections (check the blog post describing the PowerShell module installation error: Install-Module: Unable to download from URI).

If the legacy TLS 1.0 and TLS 1.1 protocols are not disabled in Windows, you must run the following command in order to use TLS 1.2 in PowerShell connections:

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

If a self-signed certificate is used on the HTTPS site, the Invoke-WebRequest cmdlet refuses to receive data from it. To ignore (skip) an invalid SSL certificate, use the following PowerShell code:

add-type @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
}
}
"@
$AllProtocols = [System.Net.SecurityProtocolType]'Ssl3,Tls,Tls11,Tls12'
[System.Net.ServicePointManager]::SecurityProtocol = $AllProtocols
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
$result = Invoke-WebRequest -Uri "https://somesite.com"

Another significant drawback of the Invoke-WebRequest cmdlet is its rather low performance. When downloading a file, the HTTP stream is entirely buffered into memory, and only after the full complete download is completed is saved to disk. Thus, when downloading large files using Invoke-WebReques, you may run out of RAM.