Posts Tagged ‘veeam’

Recently trying to upload some SOBR backups using Veeam and randomly all the backups would fail with

15/12/2020 11:52:34 AM :: The TLS version of the connection is not permitted on this storage account.
Shared memory connection was closed.

 

Fix

Add the following registry key : 

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\

Dword Name : AzureConcurrentTaskLimit

Value : 1

GD Star Rating
loading...
GD Star Rating
loading...

If you try and run a backup in Veeam and you get the following error

Using regular backup or replication functionality is restricted for Cloud Connect infrastructure servers to avoid impact on your tenants

It means you have used a Veeam Cloud Connect License on a server to backup Local backups instead of Hosting Cloud Connect Backups

GD Star Rating
loading...
GD Star Rating
loading...

 Report % Rate of change from the last Backup Job Run

Add-PSSnapin -Name VeeamPSSnapIn -ErrorAction SilentlyContinue

Disconnect-VBRServer | out-null

connect-vbrserver -server localhost

$JobsOutput = @()

Foreach ($JobObject in Get-VBRJob | ?{$_.JobType -eq "Backup"})

{

$LastSession = $JobObject.FindLastSession()

$ChangeRate = ($LastSession.Info.Progress.TransferedSize/$LastSession.Info.Progress.TotalUsedSize)*100

$JobOutput = New-Object -TypeName PSObject

$JobOutput | Add-Member -Name "Jobname" -MemberType Noteproperty -Value $JobObject.Name

$JobOutput | Add-Member -Name "Endtime" -MemberType Noteproperty -Value $LastSession.endtime

$JobOutput | Add-Member -Name "TotalUsedSize" -MemberType Noteproperty -Value $LastSession.Info.Progress.TotalUsedSize

$JobOutput | Add-Member -Name "ReadSize" -MemberType Noteproperty -Value $LastSession.Info.Progress.ReadSize

$JobOutput | Add-Member -Name "TransferedSize" -MemberType Noteproperty -Value $LastSession.Info.Progress.TransferedSize

$JobOutput | Add-Member -Name "ChangeRate" -MemberType Noteproperty -Value $ChangeRate

$JobsOutput += $JobOutput

}

$JobsOutput | Out-GridView

Disconnect-VBRServer | out-null

However one result is not a fair estimate of rate of change , and I don’t have a Veeam One license so I decided to Average Stuff

You can run on One VM or all the VM’s in a Backup Job

Add-PSSnapin -Name VeeamPSSnapIn -ErrorAction SilentlyContinue

connect-vbrserver -server localhost

#Query to get all jobs in the time period in hours that have processed data (Data ne 0 ) and have completed

$vbrtasksessions = (Get-VBRBackupSession |

Where-Object {($_.EndTime -ge (Get-Date).addhours(-168) -or $_.CreationTime -ge (Get-Date).AddHours(-168) -or $_.State -eq "Working")}) | Get-VBRTaskSession | Where-Object {$_.Status -notmatch "Idle|InProgress|Pending|Fail"-and $_.Info.Progress.TotalSize -ne "0"}

#Get Backup Job and VM

#$getOne = $vbrtasksessions | ? {$_.JobName -eq "%Veeam Backup or Backup Copy Name%" -and $_.Name -eq "%Name of Server%"}

#Or Get Whole Backup Job and average between all the machines

#$getOne = $vbrtasksessions | ? {$_.JobName -eq "%Veeam Backup or Backup Copy Name%"}

#Cannot Get Total usedSize in Backup

Echo ListofAllFullSizesInBytes $getOne.Info.Progress.TotalSize

#Echo AverageTotalFullBackupSizeInGB $([Math]::Round([Decimal]($getOne.Info.Progress.TotalSize | Measure-Object -Average).Average/1GB, 2))

Echo ListofIntcrementalSizeinBytess $getOne.Info.Progress.TransferedSize

#Echo AverageIntSizeinGB $([Math]::Round([Decimal]($getOne.Info.Progress.TransferedSize | Measure-Object -Average).Average/1GB, 2))

Echo RateofChangein% ((($getOne.Info.Progress.TransferedSize | Measure-Object -Average).Average/($getOne.Info.Progress.TotalSize | Measure-Object -Average).Average)*100)

Disconnect-VBRServer | out-null
GD Star Rating
loading...
GD Star Rating
loading...

Work out how much Space

https://rps.dewin.me/

work out bandwidth

https://rps.dewin.me/bandwidth/

Refs Calculator

http://dewin.me/refs/

 

 

  1. It is not a good idea to target the primary and backup copy job to the same repository as this negates the 321 rule and if an issue was to occur on that repository the primary and secondary copies of data would be lost.
  2. Instead it would be better to use a primary repository configured with ReFS and then have a secondary location configured with Windows Deduplication (Compression should be off instead of auto).
  3. It is not generally considered a good idea to have deduplication for the primary backup chain as this effects performance and increases RTPOs.
  4. The actual space savings depend on the change rate of the source data and the types of chains that are used on the ReFS repository. Also ReFS savings don’t show when viewed through file explorer. To learn more about checking the files savings please take a look at these links;

https://www.virtualtothecore.com/how-much-space-am-i-saving-thanks-to-refs-blockclone/

http://dewin.me/refs/

  1. Some limitation worth mentioning are Windows deduplication is for 4TB files so larger files will not be completely deduped. Also GFS doesn’t show ReFS savings until the point is synthesized as up to this point it will only shows as an incremental and changed blocked cannot be reused by ReFS pointers.

 

 

GD Star Rating
loading...
GD Star Rating
loading...

Recently we had jobs for a server showing up the error : NFC storage connection is unavailable

Image result for NFC storage connection is unavailable

Turns out a colleague had added a server to the Veeam application with a tape drive to the unit which had actually added that server as a Veeam Backup Proxy!

Jobs actually default to “Use Any Available Proxy” so the Jobs were using the tape drive server as a proxy for some of the Jobs hence the NFC Failure

GD Star Rating
loading...
GD Star Rating
loading...

Recently multiple Transform jobs for Veeam on a Synology NAS was maxing out the I/O’s of the NAS , with the errors

Previous full backup chain transformation failed Error: Agent: Failed to process method {Transform.Patch}: An unexpected network error occurred.

applying the below registry keys and spacing out the transform fixed the issue 

 

SessTimeout - Reboot Required

Key: HKLM\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters\
DWORD: SessTimeout
This is a value in seconds. Try a value of 600 decimal (10 minutes).
This increases the amount of time the Windows SMB client will wait for a response from an SMB server before it aborts the connection. The default timeout is one minute.

TcpMaxDataRetransmissions - Reboot Required

Key: HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
DWORD: TcpMaxDataRetransmissions
Try a value of 10.
This increases the number of times the Windows TCP implementation will retransmit a data segment before it aborts the connection. The default number of retries is five.
GD Star Rating
loading...
GD Star Rating
loading...

The first delay is a long cycle of checking every file mentioned in the VSS writers metadata. There are 8 million of FILESTREAM blobs on that server. This must be what causes this cycle to be so long for that volume.

 

Please download the bundle here: https://storage.veeam.com/Fix_125973_aebe4c1eb0.zip

To install, do the following on Veeam server and any Guest Interaction proxies that you have:
1. Locate C:\Program Files (x86)\Veeam\Backup Transport\GuestInteraction\VSS\VeeamGuestHelpers
2. Rename VeeamVssSupport2003_X86.dll to VeeamVssSupport2003_X86.dll_old
3. Rename VeeamVssSupport2008R2_X64.dll to VeeamVssSupport2008R2_X64.dll_old
4. Rename VeeamVssSupportXP_X86.dll to VeeamVssSupportXP_X86.dll_old
5. Extract new versions of these 3 DLL files from the archive.

GD Star Rating
loading...
GD Star Rating
loading...

A Server 2012 R2 started getting VSS Writer Errors randomly in Veeam backup job. The server is running AADconnect which is set to autoupdate

A restart of the VSS SQL Writer which usually fixes this did not resolve the issue.

A look in the event log shows

A VSS writer has rejected an event with error 0x800423f4, The writer experienced a non-transient error. If the backup process is retried,
the error is likely to reoccur.
. Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer.

Operation:
PrepareForSnapshot Event

Context:
Execution Context: Writer
Writer Class Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
Writer Name: SqlServerWriter
Writer Instance Name: SQL Server Code-Named ‘Denali’ CTP2:SQLWriter
Writer Instance ID: {6c73bfe9-f82f-4854-bec4-4382c314a583}
Command Line: “C:\Program Files\Microsoft SQL Server\90\Shared\sqlwriter.exe”
Process ID: 4292

With some

SQLVDI: Loc=SignalAbort. Desc=Client initiates abort. ErrorCode=(0). Process=4292. Thread=3940. Client. Instance=LOCALDB#SHDA41B2. VD=Global\{9C831400-DE4D-4364-BA22-A8299CF545FC}4_SQLVDIMemoryName_0.

Looks like you need to download the SQL Fix from here : https://support.microsoft.com/en-us/help/2983175/cumulative-update-package-2-for-sql-server-2012-sp2

Mirror Here : https://mega.nz/#!3AZ3gK7I!kDPhnAlur4XtslKxGXwmbnLitJiwN9R6rK-z4Rh0N-s

 

**Update , fix does not seem to work

A repair on the SQL database and then run the below script

Looks like this might be an issue with AADcoonect

https://github.com/pariswells/pariswells.com/blob/master/fixaadconnectvsswriter.ps1

 

**Update

 ADSync launches an SQL Server Local DB under it’s own user account
– The User Profile Service thinks ADSync is no longer logged on, and unloads the registry
– SQL Server though still has handles to the registry, but they’re invalid now

Detailed explanation:
https://support.microsoft.com/en-us/help/2287297/a-com-application-may-stop-working-on-windows-server-2008-when-a-user

In short: Computer Configuration->Administrative Templates->System->User Profiles->Do not forcefully unload the user registry at user logoff

GD Star Rating
loading...
GD Star Rating
loading...

 Error: Unfreeze error: [Backup job failed.
Cannot create a shadow copy of the volumes containing writer’s data.
A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{60e3e5dd-73fc-46d6-ab6b-6b8723df09e3}]. Writer’s state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]

I restarted the SQL Server VSS Writer ( Can be done during production as only used for backups ) and a retry still errored.

In the end I needed to reboot the server

GD Star Rating
loading...
GD Star Rating
loading...