Suddenly several 502 on prod

Developer corner
1

We get some 502 error on our deployed production app for some days now, is this something encountered by everyone on weweb ?

Thanks you

2 Likes
2

Hi @Alextr :wave:

Ah yesss, I’ve been seeing the same issue on one of my personal apps. It turns out it’s a false positive in the sense that when Better Stack tells me it’s down, the app is actually live (I am able to access it and it works fine).

The team is looking it but so far it looks like it can happen when:

  • the homepage is a private page,
  • an unauthenticated user tries to access it,
  • that user is redirected to a public page as a result (e.g. a login page).

At the moment, when an unauthenticated user tries to access a private page, there’s a call made to a weweb server. When the app interacts with that server, it seems like there’s a 502 error happening for 1-2 seconds at times.

We’re not sure what’s causing that error, we’re looking into this BUT… there are two things to note:

  1. monitoring tools might report 2-3 minutes downtime because they recheck the status of the app every 2-3 minutes but if you try accessing your app during those 2-3 minutes of “downtime”, you should find it works fine. If not, please record a video refreshing the page while looking at the console and network tabs in the browser inspector and open a bug ticket here.

  2. in a few weeks time, the way private pages work will be different (there will no longer be calls to a server) so this problem should disappear completely. It will also improve private page loading speed no matter where you’re based in the world (because currently there’s a delay when the app interacts with our server in the US).

Does that help?

If you think you might not be experiencing the same issue, could monitor the URL of a public page on the same app and create a bug report here with a video refreshing the page while looking at the console and network tabs in the browser inspector.

3

I am having a lot of 502 as well! It happens since last week (27th of may) and it’s totally driving me nuts!

I’m using Xano so I’m looking on Xano’s end if there’s anything going on, but there’s clearly nothing out of the ordinary in my view, so I’m suspecting it could be more of a WeWeb Issue.

It seems also to be happening frequently at specific time ranges of the day (around 13:00, 16:00 and 17:25~30 GMT-0).

I did issue a bug report. I’m still awaiting any answer regarding this problem.

1 Like
4

Hi @matt_b :wave:

Thanks for creating the support ticket. Can you send me the link of the ticket in a private message so I can take a look?

Three follow up questions to help us investigate:

  1. have you checked if users are able to access the app during the downtime?
  2. are you using Better Stack for monitoring?
  3. are you monitoring a private page or a public page?

From our tests so far, the downtimes reported by Better Stack are “false positives” in the sense that they seem like false alarms because users can access the live apps during the supposed downtime.

I’ve been in touch with Better Stack’s support and they confirmed: “based on the reply from the server, the incidents seem to be genuine on our side. That doesn’t necessarily mean that the incident is happening to the real users as well, though, that’s a response that’s returned to us by the target server.”

Since the end-users of the live apps are apparently able to access the apps at all times, and the other monitoring tools we have are not reporting any server errors, we have stopped our investigations for now but we can reopen for sure if you tell us your end-users are affected.

5

Hi Joyce ! :slight_smile:

  1. Users have been reporting either facing the 502 message written in text in their browser, that pages are not fully loading or functions not responding, and this, several times in the day.
  2. I am currently not using any monitoring tools for WeWeb.
  3. Point 2.

It’s currently under investigation though (as I was informed today), so I’ll wait for a response.

1 Like
6

Ah ok, definitely a priority issue then! I’ll let the team follow up with you through the support app to get to the bottom of it.

7

502 is a gateway error that I associate with the backend more than the static assets served on the front-end.

502s are particularly disconcerting because usually there is no record on the backend of that call ever happening, because the gateway never forwarded it.

If you have a Xano back-end, you might look at its load in terms of number of requests, how long those requests take to process, and what kind of plan you have with them. Xano doesn’t auto-scale, so when there are too many request-seconds at once, this 502 thing can happen.

If this is Xano, the solution is one of three things:

  1. cut the number of requests (usually making your front-end from weweb less chatty),
  2. make the requests faster (e.g. fixing up the function stack in question) or
  3. pay Xano money for a larger instance to handle the excess load.

This kind of question comes up frequently and we work it as part of our focus on the hardest 5% in State Change daily office hours.

1 Like
8

Thanks for your input raydeck.

I did all of those things. Without turning this into a Xano related discussion too much, I’ve investigated my API history and it is pretty much clean and nothing suggest heavy API traffic. We do have a constant API traffic, but nothing heavy. I did encounter 502 because of heavy traffic in the past, but this time, It does not fit the heavy traffic symptom as in all API calls are snappy and very short in general with a API call runtime that equals the processing runtime (vs. API runtime was 5+ seconds greater than the actual runtime). Plus, it seems more of a time-of-day issue than a traffic issue.

Right now it’s stable, but according to the observed behaviour, it’ll become unstable in 15-20 minutes.

I’ll continue to investigate.

9

Based on the description, it’s most likely WeWeb’s servers causing the issue. We had this many times on the projects. There is no way of Xano or any other API causing to throw the browser the 502 in a way the OP describes imo, unless the pages are prerendered maybe?

2 Likes
10

I’m didn’t know about that! In which case one might co spider self hosting the static assets. A number of state change members have done that with limited work and good results.

1 Like
11

Exactly,

When it’s about API overload, there’s a significant slowdown of response time before completely not responding, but it will never show the 502 error on screen since it’s more about backend communication. It will either slow, hang, or unexpectedly disconnect users. In this case, 95% of users stays connected. It just fails while processing something midway or at the very beginning of a page load.

There seems to be some improvement yesterday. None of our users reported anything.

EDIT: Someone got a 502 around 12:30 AM GMT+0

12

I’ve also been experiencing the 502 Bad Gateway issues in production. This has occurred for logged-in users being navigated from one private page to another, as well as for logged-out users opening a public page.

1 Like
13

Hi, we still have those issues from time to time, is there any update on that on weweb’s side?

@Joyce

14

Hi @ArthurC :wave:

Yes, we think this should be fixed with the June 26th release :slight_smile:

15

Hi Joyce, thank you for the update, have a great day!

1 Like
16

Hi everyone :wave:

This should be fixed with the new release. Make sure to republish your app though.

If you still experience issues, please let me know asap so I can reopen the ticket internally. For now it’s marked as something to keep an eye on but there’s no new active investigation because we think the new way of changing page navigation will solve the issue.

1 Like
17

Thank you for your message!

1 Like