End of SSH-RSA support for Azure Repos

Bohdan Janousek

February 15th, 202433 3

Azure Repos provides two methods for users to access a git repository in Azure Repos – HTTPS and SSH. To use SSH, you need to create a key pair using one of the supported encryption methods. In the past we’ve been supporting only SSH-RSA and we’ve asked users to enable the SSH-RSA here. This is not required to be done anymore as in 2022 we’ve added support for the RSA-SHA2-256 and RSA-SHA2-512 to Azure DevOps Service. Later that year, the same support was also added to Azure DevOps Server 2022 and in August 2023 to Azure DevOps Server 2020 and 2019. The relevant release notes are linked here:

We are now announcing the deprecation of SSH-RSA as a supported encryption method for connecting to Azure Repos using SSH.

The SSH-RSA is a weak encryption method. It is also already deprecated by OpenSSH and cannot be used unless enabled explicitly.

This change impacts you immediately if you are using Azure DevOps Service and are using SSH-RSA keys to connect to repos through SSH.

If you use Azure DevOps Server, then this change does not currently impact you. However, this change will impact you when you upgrade to the next version of Azure DevOps Server 2022.3 (expected to be released towards the end of 2024), since that version will not support SSH-RSA keys. If you use any of the following versions of Azure DevOps Server, you are strongly encouraged to move from SSH-RSA keys to more secure RSA-SHA2-256 or RSA-SHA2-512 keys:

  • Azure DevOps Server 2019 Update 1.2 Patch 4 and later
  • Azure DevOps Server 2020 Update 1.2 Patch 7 and later
  • Azure DevOps Server 2022

Migrating to more secure ciphers as supported by current versions of Azure DevOps Server will prevent issues in the future when you upgrade to newer versions of the server.

The deprecation of SSH-RSA ciphers in Azure DevOps Service will be done in four phases as explained below.

Phase I – User opt-in

Any user of Azure DevOps Service can migrate from SSH-RSA to more secure ciphers supported by Azure Repos. These are RSA-SHA2-256 or RSA-SHA2-512. To do so, users can follow these steps:

  1. Generate new public private key either by running ssh-keygen -t rsa-sha2-256 or ssh-keygen -t rsa-sha2-512.
  2. Add the key generated in point 1 to the SSH agent. This can be done by running command ssh-add <PathToYourPrivateKey>.
  3. Change the local SSH configuration so the key generated in point 1 is listed before the SSH-RSA key. This is to ensure more secure algorithms will be used instead of SSH-RSA.
  4. Upload the public part of the key generated in point 1 to Azure DevOps. See this how to do so.

Phase II – Throttling/delaying

Starting early March 2024, we will start to delay any SSH operation where the SSH-RSA was used to secure the SSH channel. There will be a warning shown in the command line output stating:

“ssh-rsa is about to be deprecated and your request has been throttled. Please use rsa-sha2-256 or rsa-sha2-512 instead. Your session will continue automatically. For more details see https://devblogs.microsoft.com/devops/ssh-rsa-deprecation.”

Phase III – Brown out

In April 2024, we will start to fail the executions of any operation where the SSH-RSA was used to secure the channel. We will execute the failures in a couple of stages each with different number of failure intervals and different length of the individual interval. The intervals will start at random times during the day. Each stage will last for about a week.

Stage Interval length Count of intervals in a day Total failure time in a day
1 30 minutes 1 30 minutes
2 1 hour 3 3 hours
3 2 hours 4 8 hours
4 1 hour 12 12 hours

To ensure it is clear to user why the SSH operation failed we will add an error message to command line output. The error message will be:

“You’re using ssh-rsa that is about to be deprecated and your request has been blocked intentionally. Any SSH session using SSH-RSA is subject to brown out (failure during random time periods). Please use rsa-sha2-256 or rsa-sha2-512 instead. For more details see https://devblogs.microsoft.com/devops/ssh-rsa-deprecation.”

Phase IV – SSH-RSA removal

Late in Q2 2024, we will start failing the execution of any operation where the SSH-RSA was used to secure the SSH channel. To ensure it is clear to user why the SSH operation failed we will add to command line output the following error message:

“You’re using ssh-rsa that is unsupported. Please use rsa-sha2-256 or rsa-sha2-512 instead. For more details see https://devblogs.microsoft.com/devops/ssh-rsa-deprecation.”

FAQ

Q: I am running the Azure DevOps Server. Can I use SSH-RSA going forward?
A: Yes, you can for now. Nonetheless, when you are on any version of Azure DevOps Server supporting new ciphers, then you should consider moving to these more secure ones. See the Phase I – User opt-in chapter for more details how to do so.

Q: I am running Azure DevOps Server version without support for the new ciphers, but I do want to use a more secure cipher than SSH-RSA. Can I do that?
A: Yes, but you need to upgrade to any version of Azure DevOps Server supporting these more secure ciphers.

Q: Are there plans to remove support for the SSH-RSA also from the Azure DevOps Server?
A: Yes. In a future version we will remove the support for SSH-RSA from Azure DevOps Server. So, please consider moving to a more secure cipher. See the Phase I – User opt-in chapter for more details how to do so.

Q: Will I be able to use the SSH-RSA to connect to Azure DevOps Server after you remove support for SSH-RSA?
A: Yes, but this will require your organization administrators or IT specialists to take special action. Without that, you will be able to use only the supported ciphers.

Bohdan Janousek Senior Program Manager, Azure DevOps

Read next

33 comments

Leave a comment

  • Alexandre Bonfitto 9

    When will Azure DevOps support Ed25519 keys?

    • Bohdan JanousekMicrosoft employee 0

      Thanks for the question Alexandre, we do not have immediate plans to add the support for the elliptical keys. Nonetheless we are considering to add these in the future.

        • Bohdan JanousekMicrosoft employee 0

          That developer forum ticket is known. As said – we are considering adding the support for the elliptical keys in the future.

          • Michael Mairegger 6

            As you can see from the linked forum ticket the developer community felt abandoned by Microsoft not responding to this ticket for years.

            I am using a YubiKey with elliptical keys for ssh auth on GitHub and every other ssh auth. Only for Azure Devops a ssh-key is needed which is very annoying.

            But thanks for stating that the issue is on your radar. Hopefully this issue will be addressed with high priority

          • Jeff Sharp 4

            The lack of Microsoft response to that ticket is highly disappointing. That’s not how a developer community is supposed to work.

            Any response would have been better than no response. No response leaves the community free to make its own conclusions, which become more negative over time.

      • Jason White 5

        You have got to be joking. This is insanity that it is 2024 and ED25519, the de-facto standard SSH key algorithm, is STILL NOT SUPPORTED! I am having to keep legacy RSA keys hanging around because Azure DevOps is so archaic and out of date. ED25519 has existed for at least 10 years now, what’s the hold up? Why do you even restrict key algo’s in the first place? Just support everything that OpenSSH supports natively! There is absolutely no good reason not to!

        Utter, utter, madness.

        • Bohdan JanousekMicrosoft employee 0

          I do appreciate the fact that not supporting the elliptical keys feels odd in 2024. I do not want to jump the gun now to confirm the support will be added soon and, in the end, delaying that. That is why it was said to be considered added in the future.

      • Floimair Florian 4

        Well, it took you forever to retire what is already considered insecure since 2018. It took you the exact same amount of time to even react to the pending request that Michael Mairegger thankfully referenced here.

        Don’t get me wrong, I am thankful that there is finally a statement about this as the request up to now was basically ignored even though there were lots of comments on it, so definitely not a niche request.

        In terms of security this is a bummer that ssh-rsa even was still active up to now. Supporting elliptic keys is a thing possible on almost any platform except Azure DevOps. Also to me this looks to be a low-hanging fruit that could quickly be realized.

        Please consider adding elliptic keys in the near future.

        Thank you!

        • Bohdan JanousekMicrosoft employee 0

          This blog post was done to drive the attention of the users who are still using SSH-RSA to connect to Azure DevOps Service although RSA-SHA2-256 or RSA-SHA2-512 were added some time ago. We want user to stop using SSH-RSA now It’s first step. When that first step is completed, we will plan adding support for other algorithms and for sure the elliptical one(s) will be on the plate.

          We want to do this slower rather than big-bang approach to make sure all works OK before next step is taken.

      • Jeff Sharp 1

        Lack of ed25519 is one reason why my clients’ teams never completed adoption of Azure Repos. We used RSA keys thinking that eventually ed25519 support would be added. The industry was moving in that direction, so it seemed a fairly safe bet. As time passed and no ed25519 support arrived, our adoption slowed, stopped, and ultimately reversed.

        • Bohdan JanousekMicrosoft employee 0

          Jeff, thanks for the insight from your as well as your clients’view. Can you provide more details about two things?
          1) What is the ratio between SSH and HTTPS use while accessing the Azure Repos?
          2) What is the main driver for using SSH and not HTTPS?

      • Timber Schroeder 1

        Perhaps consider fast tracking offering modern keys. This was one of a litany of reasons that I have largely abandoned Azure DevOps in my workplace, and continued friction like this from extremely basic feature requests like EC support that have had highly upvoted requests for over six years now is insane.

  • Eli Elgaev 1

    What is the real reason Microsoft is trying not to accept ED25519?
    Ed25519 are now default since OpenSSH 9.5, what is the reason to delay it?

  • Eli Elgaev 0

    Generating a key with:

    $ ssh-keygen -t rsa-sha2-512

    and updating my public key from:

    $ cat ~/.ssh/id_rsa.pub

    still prompts me:

    git@ssh.dev.azure.com's password:
  • Antony JULIEN 0

    What about people using old platforms that don’t support SHA2 ? Will they have to abandon Azure ? Or will there be a mean to continue using RSA ?

    • Bohdan JanousekMicrosoft employee 0

      Can you confirm what legacy system you’re using and how you were affected so far? The main intention is to improve the security and the first step is to remove the ciphers that are weak.

  • Tomasz Bondarewicz 2

    Hi,
    The way this change is being introduced unfortunately, in my opinion, complicates error analysis. Because when my software, which authenticates to the Azure DevOps repository, encounters errors, I don’t know how to pinpoint the exact cause of the problem or how to test fixes.
    Errors occur randomly. Time windows can start and end at any moment, and during their duration, not all connection attempts result in an error.
    If it starts to work – I cannot be sure if it’s because of my fixes or just because the “error window” has ended.
    I think, there should be some separate “endpoint” available, perhaps under a specific DNS address, that already has permanently disabled support for ssh-rsa. It could be used for verification to ensure that the systems I’m using are correctly configured.
    The advice in the “Phase I – User opt-in” section is, in my knowledge, misleading.
    There is no such thing as “generating rsa-sha2-256 or rsa-sha2-512 keys with ssh-keygen -t parameter.” What matters is whether the system from which we connect supports a given cipher and whether rsa-sha2-256 / rsa-sha2-512 are among the supported ciphers. The key itself does not imply a specific cipher but it’s negotiated during the handshaking between the client and the server.
    If my system (specifically, I use Flux on Azure Kubernetes Services with Ubuntu 22.04 worker nodes) started receiving errors when attempting to connect to repositories on Azure DevOps, it seems to me that I don’t need to generate new keys. Additionally, I am unable to disable the ssh-rsa cipher from the list supported by AKS worker nodes because they are provisioned by Azure.
    Please let me know if I’m wrong in the above interpretation.
    E.g. I have regenerated my keys yesterday’s morning and the eerors stopped. But then they reappeared after ~8 hours for ~1,5h. It’s been “quiet” since then through the night. But I have no idea if it’s OK now just because there is no “failure interval” for me right now.

    • Peter Ernst 3

      We have a similar thing happening with ArgoCD. Debugging this is extremely difficult since there is no way to check if our changes really fixed the issue, or if the Brown-out-Window has closed. Additionally we have been unable to restrict the used Algorithms through standard Linux methods inside of the container.
      I really like the idea of a endpoint, where the ssh-rsa HostKeyAlgorithm is disabled, or some other way such that you can reproduce this issue.

      • Alex Movergan 2

        Same nightmare with FluxCD. I’ve regenerated keys like 100 times. Same story. The error won’t go away.

        UPD: I tried to set the flux source controller to use --ssh-kex-algos=rsa-sha2-512, and now I can’t checkout with a new error:

        no common algorithm for key exchange; client offered: [ext-info-c kex-strict-c-v00@openssh.com], server offered: [diffie-hellman-group1-sha1 diffie-hellman-group14-sha1 diffie-hellman-group-exchange-sha256]

        UPD2: fixed it by setting

          set {
    name  = "sourceController.container.additionalArgs[0]"
    value = "--ssh-hostkey-algos=rsa-sha2-512"
  }
        • Tomasz Bondarewicz 0

          Thanks @Alex Movergan !

          @Bohdan Janousek I still think that you are not doing this deprecation right :-/ Info about key regeneration is misleading in the first place. Second are the random brown-out windows.

    • Bohdan JanousekMicrosoft employee 0

      ssh-keygen support (of course) to generate the desired signature type when signing certificates using an RSA CA key. The available RSA signature variants are “ssh-rsa” SHA1 signatures, not recommended), “rsa-sha2-256”, and “rsa-sha2-512” (the default). See https://www.man7.org/linux/man-pages/man1/ssh-keygen.1.html for more details.

      Step 3 of the opt-in phase explicitly asked to “Change the local SSH configuration so the key generated in point 1 is listed before the SSH-RSA key. This is to ensure more secure algorithms will be used instead of SSH-RSA.”

      Guidance how to deal with common errors is provided here https://learn.microsoft.com/en-us/azure/devops/repos/git/use-ssh-keys-to-authenticate?view=azure-devops#q-ssh-cannot-establish-a-connection-what-should-i-do (that is linked in the blog post too).

      • Rouke Broersma 1

        @bohdan The link you sent specifically says this about the -t option:

           -t dsa | ecdsa | ecdsa-sk | ed25519 | ed25519-sk | rsa
           Specifies the type of key to create.  The possible values
           are “dsa”, “ecdsa”, “ecdsa-sk”, “ed25519”, “ed25519-sk”,
           or “rsa”.

                   This flag may also be used to specify the desired
           signature type when signing certificates using an RSA CA
           key.  The available RSA signature variants are “ssh-rsa”
           (SHA1 signatures, not recommended), “rsa-sha2-256”, <strong>and
           “rsa-sha2-512” (the default).</strong>

        This indicates to me that this is not the most relevant step in the guide. The more secure option is after all already the default so most people will have keys using the secure signature algorithm.

        To me this makes Step 3 the most important step in the guide, yet this is the step with the least information. It would be great if you could modify the blogpost and write down exactly what should be changed in ~/.ssh/config in order to work with azure devops. Right now you’re leaving this as an exercise to the reader, instead of explicitly listing the required configuration. This is unhelpful when coupled with random brownouts. We have no way of confirming if configuration is correct or not, until it is too late.

        • Pablo Patruno 0

          Hi, did you get any luck ? I was struggling with this all the day . Even though I generate the key with the 256 or 512 signature I am still getting the same error .

      • Tomasz Bondarewicz 0

        What about environments where there is restricted (or none) access to local SSH configuration, like AKS worker nodes ?
        Someone can also assume, that the “local SSH configuration” regards the OS-level ssh config, while it can be as well the ssh implementation in installed software – like (as I suppose) ssh go library in FluxCD.

        Random, undocumented brown-out windows make it nightmare even more.

  • Panos Sdonas 0

    Anyone knows when is the next brown out period? We can’t tell if our changes worked or not at the moment.

  • Mcneal, Evan B 6

    For ArgoCD users that are experiencing pain during the recent random “brownouts”, it seems that the ArgoCD Team is aware of an issue as outlined here: https://github.com/argoproj/argo-cd/issues/17634

    Using ecdsa or ED25519 would be ideal but Azure DevOps only supports rsa2-512 and 256, so we are waiting on either a fix from ArgoCD to explicitly support those or Microsoft to actually support new standard ssh tokens.

    @microsoft please consider supporting more than rsa in the future as this is a clearly huge problem for many people. The brownouts were NOT a good method to allow devs to test changes to this and this whole migration experience has been terrible.

  • Eric Newton 1

    Really having a devil of a time trying to get jenkins on ubuntu 14 to work. It doesnt work and then it works and then it doesnt work again.

    Thanks for the nondeterministic testing ability. Couldn’t even give a number of seconds the window would be denying requests??? Had you reported the number of seconds the window would be denying requests, then AT LEAST i’d have something tangible to work with.

    I do things to try to fix, it starts “working”, and so I think my last change makes it work. And then a f**king hour later its back to not working again.

    This is an absolutely atrocious way to conduct a phase out. The brownout period shouldnt last more than 10-30 minutes, and should BE DECLARED IN THE ERROR MESSAGE WHEN IT ENDS.

    Yet, its 6:00pm eastern time 25-Apr-2024 and now I cant work for one entire hour. Thanks y’all. REALLY APPRECIATED.

  • Eric Newton 0

    Now again at 8:00 pm i’m not able to continue working for a f**king hour. THANKS GUYS.

Feedback usabilla icon