Resolved -
Root Cause
The cloud portal sends email using Microsoft Exchange. To prepare for an upcoming deprecation of Exchange Web Services, Octave's IT department modified permissions applied to connected applications. Equivalent permissions using new application role assignments were in place, so a seamless transition was expected. However, it turned out that the new role-based access control model did not provide parity with the previous level of access required. This caused the issue and prevented emails such as password reset links from being sent out by the cloud portal.
Resolution
Once the impact was realized, Octave IT rolled back the permission changes, and this restored email capability.
Prevention
Short-term
Octave IT department missed providing advance notice to cloud portal which uses legacy permission model to allow them to carry out testing. They will do so for any future changes. This would have prevented the issue experienced on Sep 14.
Long-term
Cloud portal team will update the integration with Microsoft Exchange using new role-based access controls before existing functionality is deprecated and becomes unavailable to use. Cloud portal team has backlog items open, and they will be scheduled to address this to avoid any future impact.
Sep 16, 12:47 UTC
Update -
Root Cause
The cloud portal sends email using Microsoft Exchange. To prepare for an upcoming deprecation of Exchange Web Services, Octave's IT department modified permissions applied to connected applications. Equivalent permissions using new application role assignments were in place, so a seamless transition was expected. However, it turned out that the new role-based access control model did not provide parity with the previous level of access required. This caused the issue and prevented emails such as password reset links from being sent out by the cloud portal.
Resolution
Once the impact was realized, Octave IT rolled back the permission changes, and this restored email capability.
Prevention
Short-term
Octave IT department missed providing advance notice to cloud portal which uses legacy permission model to allow them to carry out testing. They will do so for any future changes. This would have prevented the issue experienced on Sep 14.
Long-term
Cloud portal team will update the integration with Microsoft Exchange using new role-based access controls before existing functionality is deprecated and becomes unavailable to use. Cloud portal team has backlog items open, and they will be scheduled to address this to avoid any future impact.
Sep 15, 14:20 UTC
Monitoring -
The issue is resolved now and users should be receiving emails from the portal. We will continue to monitor the situation. Thank you for your patience.
Sep 15, 01:02 UTC
Investigating -
We have observed and received reports that currently, users are not receiving emails from the Octave Cloud portal for password resets, NUL requests etc. We are aware of this issue and our teams are currently investigating this. Thank you for your patience while we work to resolve the issue.
Sep 14, 20:39 UTC