Widgetbox Status

Intermittent Slowness Right Now

Widgetbox.com and your widgets are sometimes being served slowly right now. In some cases you may see a hung page.

We are working full speed on resolving the issue and will report when we have more information.

Update 11:00 PST - Our hosting provider made a storage configuration change that didn't work with our system. They backed out the change and systems resumed full speed immediately. We apologize for the slowness.

Move To New Datacenter Went Smoothly

Widgetbox moved all operations to a new datacenter this weekend. There was no downtime and no slowness.

In the process we upgraded all our hardware and our network capacity: you should see a noticeable speed improvement.

The new datacenter will allow us to grow better as the size of the widgesphere grows. You guys are keeping us on our toes!

Widget Serving Is Down

Many of your widgets are not currently being served right now. This is a networking issue that we are all actively working on. We will let you know as more information is available.

Update: all widgets are being served as of 1pm PST. The networking issues were again within our hosting provider. We have active and immediate plans to move to a more reliable host.

Widgetbox Connection Issues

We're experiencing intermittant delays and dropped connections serving your widgets and browing the gallery.

We're all working on it and will update this as soon as more information is available.

Update: All systems are fine now. There was a networking issue that we are continuing to investigate both within our hosting provider (Rackspace) and in our own systems.

Developer metrics will be late

We’ve done some major upgrades over the last few days and as a consequence, your developer metrics will be a day late. There’s an outside chance that it will be two days, depending on how some jobs run tonight.

No metrics are lost, we just had to delay processing.

We’re very sorry for the inconvenience.

Slowness right now

We're experiencing slowness in widget serving and in browing our gallery right now. Will update as more details are available.

Update 12:00 noon: no more slowness. The issue was networking hardware at our ISP. Will update as more details are available.

Update: Our provider, Rackspace, installed the wrong license on our networking hardware. The resolution for this, and for the many other Rackspace-related issues, is to move to a different provider and to multiple datacenters. This set of projects is in progress.

All Systems Go

All systems have been operating very smoothly since the network upgrade two days ago.

Widget Serving and Gallery Issues

We've been having intermittent issues serving your widgets and our widget gallery today. This has affected a large portion of your widgets, on an intermittent basis.

The whole team is working on it. 

We are in the process of upgrading our network hardware, which should fix the problems.

We sincerely apologize and will give you more details as they are available.

Update 3:15pm PST:  our hosting provider says the network upgrade could be complete in an hour.  We'll give you an update then.

Update 4:15pm PST: we are serving all your widgets now.  Late this evening (Pacific PST time) all widgets will be down for up to 30 minutes while we roll out another network upgrade.  We'll give another update as we know more details.

Update 7:15pm PST: we are still serving all your widgets.  At 9:30pm PST we will be down somewhere between 10 and 30 minutes for a final network hardware upgrade.

Update 11:15pm PST:  there was 3 minute of downtime during the upgrade.  The network hardware now has 5 times more capacity.  All systems look good.  We are monitoring the systems intensively.  We will be quick to report if any issues arise as volume ramps up tomorrow morning.

Again, we sincerely apologize for the intermittently down service today for many of your widgets.  It was due to our failure to keep up with our continuing record growth.  Beyond the network upgrade and other things done today, we learned a lot about planning for growth and will be making many changes that we will be rolling out in the coming days and weeks.

Please let us know if you have any questions or comments.

Service Slowdown

Hi everyone,

We had a slowdown of the widget gallery and widget serving this morning, PST.

The trouble started due to record traffic (which is good of course) causing us to move traffic to a backup database. This would be fine but for some (as yet unknown) reason an employee at our hosting provider, Rackspace, restarted that database. This caused traffic to back up and we exceeded the capacity of our network hardware. The result was traffic slowed down and lots of your web pages with widgets were affected.

We are taking the following actions:

1. Quadrupling the capacity of our network hardware.
2. Moving synchronously loaded dynamic content to the edge of our deployment, to reduce mid-server latency that causes traffic to back up.
3. Investigate the issue further with Rackspace.
4. Begin an investigation of using other hosting providers.

We're very sorry for this -- we aim for a much higher standard of service.  We know many of you live and die by your web pages and the functionality your widgets bring.

Please let us know if you have any questions or comments.

Service Outage

We had a service outage early this morning, Pacific Time (PST).

1. Our hosting provider, Rackspace, lost power in their Dallas Fort Worth data center for 3 hours.
2. All of our servers were down for that time.
3. For one hour after the power outage, our gallery -- but not the widgets on your web pages -- was showing errors until the data center and our servers were fully back online.
   

We are fully operational now.  We and Rackspace continue to research the problem.

We're working on a topology that will prevent a single-data center power outage from causing such a problem in the future.

We deeply apologize for this interruption in service.  If you have any additional issues, questions or concerns, please contact us.