Solano Labs, Inc
solano-labs
Homepage: https://www.solanolabs.com
Status Page: http://status.tddium.com/
Open Incidents
No open incidents
Scheduled Incidents
No scheduled incidents
Previous Incidents
Server Reboots
2017-01-31 06:57:08| Action | Date | Description |
|---|---|---|
| Completed | 2017-01-31 06:57:00 | The scheduled maintenance has been completed. |
| Scheduled | 2017-01-31 04:59:00 | Due to the partial service interruption earlier today, we will reboot and redeploy several pieces of core infrastructure as a precautionary measure to ensure that services are running properly.During the maintenance window, builds will be paused and the app may be in maintenance mode for up to 30 minutes. |
Server Failure
2017-01-09 20:38:30| Action | Date | Description |
|---|---|---|
| Resolved | 2017-01-09 21:34:00 | This incident has been resolved. |
| Monitoring | 2017-01-09 21:00:00 | A fix has been implemented and we are monitoring the results. |
| Identified | 2017-01-09 20:44:00 | We've identified the issue as an issue with an upstream provider. We're working to confirm this is as root cause and then recover. |
| Investigating | 2017-01-09 20:38:00 | We're experiencing a sudden issue with one of our API servers, causing the application to error. We've set the app to Maintenance Mode while we investigate the cause of this issue and work to address it. Our apologies for this inconvenience. |
Server Reboots
2016-12-30 18:36:22| Action | Date | Description |
|---|---|---|
| Scheduled | 2017-12-30 18:36:00 | We will perform routine maintenance for our core infrastructure. During the maintenance window, the app may be offline or in maintenance mode for up to 90 minutes. |
| Completed | 2017-01-01 20:36:00 | The scheduled maintenance has been completed. |
| Verifying | 2017-01-01 19:57:00 | Verification is currently underway for the maintenance items. |
| Scheduled | 2017-01-01 18:38:00 | We will perform routine maintenance for our core infrastructure. During the maintenance window, the app may be offline or in maintenance mode for up to 90 minutes. |
| Scheduled | 2016-12-30 18:36:00 | We will perform routine maintenance for our core infrastructure. During the maintenance window, the app may be offline or in maintenance mode for up to 90 minutes. |
Setup Hook failures due to npm timeouts
2016-12-15 17:30:17| Action | Date | Description |
|---|---|---|
| Resolved | 2016-12-15 23:57:00 | This incident has been resolved. |
| Monitoring | 2016-12-15 20:03:00 | The issues with npm appears to have passed. We're continuing to monitor its networking and the networking of other upstream providers, e.g. rubygems.org, that may affect build performance. |
| Investigating | 2016-12-15 17:30:00 | We're receiving notices of setup hook issues. Our investigations thus far are pointing towards issues with npm timeouts (See latency spike here: http://status.npmjs.org/). We'll continue to update here as we have more information. |
Server Reboots
2016-12-09 23:19:18| Action | Date | Description |
|---|---|---|
| Completed | 2016-12-12 17:50:00 | The scheduled maintenance has been completed. |
| Verifying | 2016-12-11 20:32:00 | Maintenance work is done. We are verifying the system and will resume builds shortly. |
| Scheduled | 2016-12-11 19:11:00 | We will perform routine maintenance. During the window, the app may be offline or in maintenance-mode for 90 minutes. |
| Scheduled | 2016-12-09 23:19:00 | We will perform routine maintenance. During the window, the app may be offline or in maintenance-mode for 90 minutes. |
Worker VM Issues
2016-11-25 22:06:44| Action | Date | Description |
|---|---|---|
| Resolved | 2016-11-25 22:06:00 | This incident has been resolved. |
| Monitoring | 2016-11-25 21:00:00 | A fix has been implemented and we are monitoring the results. |
| Identified | 2016-11-25 20:36:00 | We've identified the issue affecting the VMs and are replacing them with the corrected version. |
| Investigating | 2016-11-25 20:05:00 | We're currently investigating an issue affecting some of our worker VMs that are preventing the system from assigning them tasks. We'll continue to provide information as this investigation continues. |
Worker VM Issues
2016-11-12 03:48:35| Action | Date | Description |
|---|---|---|
| Resolved | 2016-11-12 03:48:00 | This incident has been resolved. |
| Monitoring | 2016-11-11 16:59:00 | A fix has been implemented and we are monitoring the results. |
| Identified | 2016-11-11 15:56:00 | The issue has been identified and a fix is being implemented. |
| Investigating | 2016-11-11 15:24:00 | We are currently investigating this issue. |
Build Restarts
2016-11-09 03:24:35| Action | Date | Description |
|---|---|---|
| Resolved | 2016-11-09 03:24:00 | This incident has been resolved. |
| Identified | 2016-11-08 21:52:00 | The issue has been identified and a fix is being implemented. |
| Investigating | 2016-11-08 21:02:00 | We are currently investigating the cause of numerous build restarts. These restarts are affecting queue throughput and build result turnaround. We will provide further updates as our investigation continues. |
Server Reboots & Worker Networking Changes Part 2
2016-11-06 20:35:50| Action | Date | Description |
|---|---|---|
| Completed | 2016-11-06 20:35:00 | The scheduled maintenance has been completed. |
| Verifying | 2016-11-06 19:57:00 | Verification is currently underway for the maintenance items. |
| Scheduled | 2016-11-06 18:00:00 | We will be undergoing scheduled maintenance during this time. |
Server Reboots & Worker Networking Changes
2016-11-05 21:19:55| Action | Date | Description |
|---|---|---|
| Completed | 2016-11-05 22:19:00 | The scheduled maintenance has been completed. |
| Verifying | 2016-11-05 21:50:00 | Verification is currently underway for the maintenance items. |
| Scheduled | 2016-11-02 17:48:00 | In the course of emergency maintenance yesterday, we discovered a few less urgent issues that will require additional maintenance. We will conduct this maintenance over the weekend in two phases.During the maintenance window, builds will be paused and the app may be in maintenance mode for up to 90 minutes. |