Download HLS streams (PowerShell)

Introduction

Some web player are streaming videos in parts and not as a single file. The common reason is to be able to switch the quality within the movie if necessary. It’s called “Adaptive Bitrate Streaming” (ABS). One of the most famous ABS protocols is “HTTP Live Streaming” (HLS) from Apple.

1. Traffic analysis

If DevTools …

  • isn’t blocked ▼

    Goto the Network tab.

     File            | Request URL
-----------------|------------------------------------------------------
master.m3u8      | https://website.com/path/to/video/master.m3u8
index.m3u8       | https://website.com/path/to/video/high/index.m3u8
mov_part_000.ts  | https://website.com/path/to/video/high/mov_part_000.ts
mov_part_001.ts  | https://website.com/path/to/video/high/mov_part_001.ts
...
  • is blocked ▼

    Use Wireshark instead: Follow this link

1. HLS playlists

  1. Master playlist ▼

    Before the player starts to download/play the movie segments it gets the master playlist.

    master.m3u8

    #EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=1400000
https://website.com/path/to/video/low/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=2800000
https://website.com/path/to/video/mid/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=5000000
https://website.com/path/to/video/high/index.m3u8
  2. Media playlist ▼

    The player then plays a media playlist based on bandwidth / screen size / user selection.

    high/index.m3u8

    #EXTM3U
#EXT-X-VERSION:3
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-TARGETDURATION:10
#EXTINF:10.0,
https://website.com/path/to/video/high/mov_part_000.ts
#EXTINF:10.0,
https://website.com/path/to/video/high/mov_part_001.ts
...
#EXTINF:8.0,
https://website.com/path/to/video/high/mov_part_500.ts
#EXT-X-ENDLIST

3. Media download

If the media playlist …

  • contains file names ▼ 
    function Download-PlaylistFiles()
{
    $ProgressPreference = "SilentlyContinue" # Suppresses progress bar
    Get-Content index.m3u8 | Where-Object {$_ -match "^.*`.ts$"} | ForEach-Object -Parallel {
        $Site = "https://website.com/path/"
        $File = $_

        try {
            Invoke-WebRequest "$Site/$File" -OutFile $File
        } catch {
            Write-Output "$Site/$File"
            Write-Output $_.Exception.Message
        }
    } -ThrottleLimit 10

    [console]::beep(500,1000)
}

Download-PlaylistFiles
  • contains file links ▼ 
    function Download-PlaylistFiles()
{
    $ProgressPreference = "SilentlyContinue" # Suppresses progress bar

    Get-Content index.m3u8 | Where-Object {$_ -match "^.*`.ts$"} | ForEach-Object -Parallel {
        $Link = $_
        $File = $_.Substring($_.LastIndexOf("/") + 1)

        try {
            Invoke-WebRequest $Link -OutFile $File
        } catch {
            Write-Output $Link
            Write-Output $_.Exception.Message
        }
    } -ThrottleLimit 10

    [console]::beep(500,1000)
}

Download-PlaylistFiles
  • wasn’t provided at all ▼ 
    function Download-PlaylistFiles()
{
    $ProgressPreference = "SilentlyContinue" # Suppresses progress bar

    0..500 | ForEach-Object -Parallel {
        $Site = "https://website.com/path/"
        $File = "mov_part_$($_.ToString('000')).ts"

        try {
            Invoke-WebRequest "$Site/$File" -OutFile $File
        } catch {
            Write-Output "$Site/$File"
            Write-Output $_.Exception.Message
        }
    } -ThrottleLimit 10

    Get-ChildItem -File | where length -le 1024 | Remove-Item
}

Download-PlaylistFiles

    If there are files skipped in the playlist sequence, they will still be downloaded and removed afterwards based on their small size (<= 1KB).

3.5 Watch during download

If the media playlist …

  • contains file names ▼

    Just use it.

  • contains file links ▼ 
    function Create-LocalPlaylist([string]$FileName)
{
    $File = $FileName + ".m3u8"

    Get-Content index.m3u8 | ForEach-Object {
        if ($_ -match "^.*`.ts$") {
            $_ = $_.Substring($_.LastIndexOf("/") + 1)
        }
        $Content += "$_`n"
    }

    $Content > $File
}

Create-LocalPlaylist "index_local"
  • wasn’t provided at all ▼ 
    function Create-LocalPlaylist([string]$FileName)
{
    $File = $FileName + ".m3u8"

    $Content = "#EXTM3U`n#EXT-X-VERSION:3`n" +
               "#EXT-X-PLAYLIST-TYPE:VOD`n" +
               "#EXT-X-TARGETDURATION:10`n"

    Get-ChildItem -Name *.ts | Sort {$_ -replace "\D+" -as [int]} | ForEach-Object {
        $Content += "$_`n"
    }

    $Content > $File
}

Create-LocalPlaylist "index"

    To set ‘EXT-X-TARGETDURATION’ properly wait til some segments are downloaded and then use the highest length rounded up.

4. Segment merging

If the media playlist …

  • contains file names ▼ 
    function Concat-PlaylistFiles([string]$FileName)
{
    $File = $FileName + ".mkv"
    $Concat = "concat:"

    Get-Content index.m3u8 | Where-Object {$_ -match "^.*`.ts$"} | ForEach-Object {
        $Concat += "$_|"
    }

    ffmpeg -i $Concat -c copy $File
}

Concat-PlaylistFiles "video"

    Alternative: Operate on the TS file names in the folder directly.

  • contains file links ▼ 
    function Concat-PlaylistFiles([string]$FileName)
{
    $File = $FileName + ".mkv"
    $Concat = "concat:"

    Get-Content index.m3u8 | Where-Object {$_ -match "^.*`.ts$"} | ForEach-Object {
        $_ = $_.Substring($_.LastIndexOf("/") + 1)
        $Concat += "$_|"
    }

    ffmpeg -i $Concat -c copy $File
}

Concat-PlaylistFiles "video"

    Alternative: Operate on the TS file names in the folder directly.

  • wasn’t provided at all ▼ 
    function Concat-PlaylistFiles([string]$FileName)
{
    $File = $FileName + ".mkv"
    $Concat = "concat:"

    Get-ChildItem -Name *.ts | Sort {$_ -replace "\D+" -as [int]} | ForEach-Object {
        $Concat += "$_|"
    }

    ffmpeg -i $Concat -c copy $File
}

Concat-PlaylistFiles "video"
FFmpeg details ▼

The concat protocol (ffmpeg -i “concat:mov_part_000.ts|mov_part_001.ts|…”) gets used and not the concat demuxer (ffmpeg -f concat -i files.txt) to prevent video stutters.

Addendum

  • HLS supports next to on-demand streaming also live streaming.
  • Other famous ABS protocols are:
    – “Dynamic Adaptive Streaming over HTTP” (DASH) from MPEG
    – “HTTP Dynamic streaming” (HDS) from Adobe